From python-checkins at python.org  Thu Sep  1 04:16:42 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Thu, 01 Sep 2011 04:16:42 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_accept_bytes_fo?=
 =?utf8?q?r_the_AST_=27string=27_type?=
Message-ID: <E1Qywpe-0007km-Ea@dinsdale.python.org>

http://hg.python.org/cpython/rev/c8b6d4417274
changeset:   72154:c8b6d4417274
branch:      3.2
parent:      72151:1e01543c3d0a
user:        Benjamin Peterson <benjamin at python.org>
date:        Wed Aug 31 22:13:03 2011 -0400
summary:
  accept bytes for the AST 'string' type

This is a temporary kludge and all is well in 3.3.

files:
  Misc/NEWS           |  3 +++
  Parser/asdl_c.py    |  2 +-
  Python/Python-ast.c |  2 +-
  3 files changed, 5 insertions(+), 2 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Accept bytes for the AST string type. This is temporary until a proper fix in
+  3.3.
+
 - Issue #9200: The str.is* methods now work with strings that contain non-BMP
   characters even in narrow Unicode builds.
 
diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py
--- a/Parser/asdl_c.py
+++ b/Parser/asdl_c.py
@@ -805,7 +805,7 @@
 
 static int obj2ast_string(PyObject* obj, PyObject** out, PyArena* arena)
 {
-    if (!PyUnicode_CheckExact(obj)) {
+    if (!PyUnicode_CheckExact(obj) && !PyBytes_CheckExact(obj)) {
         PyErr_SetString(PyExc_TypeError, "AST string must be of type str");
         return 1;
     }
diff --git a/Python/Python-ast.c b/Python/Python-ast.c
--- a/Python/Python-ast.c
+++ b/Python/Python-ast.c
@@ -611,7 +611,7 @@
 
 static int obj2ast_string(PyObject* obj, PyObject** out, PyArena* arena)
 {
-    if (!PyUnicode_CheckExact(obj)) {
+    if (!PyUnicode_CheckExact(obj) && !PyBytes_CheckExact(obj)) {
         PyErr_SetString(PyExc_TypeError, "AST string must be of type str");
         return 1;
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep  1 04:16:43 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Thu, 01 Sep 2011 04:16:43 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_expose_linux_extended_file_?=
 =?utf8?q?system_attributes_=28closes_=2312720=29?=
Message-ID: <E1Qywpf-0007kq-KJ@dinsdale.python.org>

http://hg.python.org/cpython/rev/4eb0b1819bda
changeset:   72155:4eb0b1819bda
parent:      72152:0968acf0e6db
user:        Benjamin Peterson <benjamin at python.org>
date:        Wed Aug 31 22:15:17 2011 -0400
summary:
  expose linux extended file system attributes (closes #12720)

files:
  Doc/library/os.rst    |  148 +++++++++++
  Lib/test/test_os.py   |   94 +++++++
  Modules/posixmodule.c |  402 ++++++++++++++++++++++++++++++
  configure             |    2 +-
  configure.in          |    2 +-
  pyconfig.h.in         |    3 +
  6 files changed, 649 insertions(+), 2 deletions(-)


diff --git a/Doc/library/os.rst b/Doc/library/os.rst
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -751,6 +751,26 @@
       This function is not available on MacOS.
 
 
+.. function:: fgetxattr(fd, attr)
+
+   This works exactly like :func:`getxattr` but operates on a file descriptor,
+   *fd*, instead of a path.
+
+   Availability: Linux
+
+   .. versionadded:: 3.3
+
+
+.. function:: flistxattr(fd)
+
+   This is exactly like :func:`listxattr` but operates on a file descriptor,
+   *fd*, instead of a path.
+
+   Availability: Linux
+
+   .. versionadded:: 3.3
+
+
 .. function:: fdlistdir(fd)
 
    Like :func:`listdir`, but uses a file descriptor instead and always returns
@@ -836,6 +856,27 @@
    Availability: Unix.
 
 
+.. function:: fremovexattr(fd, attr)
+
+   This works exactly like :func:`removexattr` but operates on a file
+   descriptor, *fd*, instead of a path.
+
+   Availability: Linux
+
+   .. versionadded:: 3.3
+
+
+.. function:: fsetxattr(fd, attr, value, flags=0)
+
+   This works exactly like :func:`setxattr` but on a file descriptor, *fd*,
+   instead of a path.
+
+
+   Availability: Linux
+
+   .. versionadded:: 3.3
+
+
 .. function:: futimesat(dirfd, path, (atime, mtime))
               futimesat(dirfd, path, None)
 
@@ -1536,6 +1577,17 @@
    Availability: Unix.
 
 
+.. function:: getxattr(path, attr)
+
+   Return the value of the extended filesystem attribute *attr* for
+   *path*. *attr* can be bytes or str. If it is str, it is encoded with the
+   filesystem encoding.
+
+   Availability: Linux
+
+   .. versionadded:: 3.3
+
+
 .. function:: lchflags(path, flags)
 
    Set the flags of *path* to the numeric *flags*, like :func:`chflags`, but do not
@@ -1561,6 +1613,15 @@
    Availability: Unix.
 
 
+.. function:: lgetxattr(path, attr)
+
+   This works exactly like :func:`getxattr` but doesn't follow symlinks.
+
+   Availability: Linux
+
+   .. versionadded:: 3.3
+
+
 .. function:: link(source, link_name)
 
    Create a hard link pointing to *source* named *link_name*.
@@ -1585,6 +1646,44 @@
    .. versionchanged:: 3.2
       The *path* parameter became optional.
 
+
+.. function:: listxattr(path)
+
+   Return a list of the extended filesystem attributes on *path*. Attributes are
+   returned as string decoded with the filesystem encoding.
+
+   Availability: Linux
+
+   .. versionadded:: 3.3
+
+
+.. function:: llistxattr(path)
+
+   This works exactly like :func:`listxattr` but doesn't follow symlinks.
+
+   Availability: Linux
+
+   .. versionadded:: 3.3
+
+
+.. function:: lremoveattr(path, attr)
+
+   This works exactly like :func:`removeattr` but doesn't follow symlinks.
+
+   Availability: Linux
+
+   .. versionadded:: 3.3
+
+
+.. function:: lsetxattr(path, attr, value, flags=0)
+
+   This works exactly like :func:`setxattr` but doesn't follow symlinks.
+
+   Availability: Linux
+
+   .. versionadded:: 3.3
+
+
 .. function:: lstat(path)
 
    Perform the equivalent of an :c:func:`lstat` system call on the given path.
@@ -1758,6 +1857,17 @@
    successfully removed.
 
 
+.. function:: removexattr(path, attr)
+
+   Removes the extended filesystem attribute *attr* from *path*. *attr* should
+   be bytes or str. If it is a string, it is encoded with the filesystem
+   encoding.
+
+   Availability: Linux
+
+   .. versionadded:: 3.3
+
+
 .. function:: rename(src, dst)
 
    Rename the file or directory *src* to *dst*.  If *dst* is a directory,
@@ -1794,6 +1904,44 @@
    Availability: Unix, Windows.
 
 
+.. data:: XATTR_SIZE_MAX
+
+   The maximum size the value of an extended attribute can be. Currently, this
+   is 64 kilobytes on Linux.
+
+
+.. data:: XATTR_CREATE
+
+   This is a possible value for the flags argument in :func:`setxattr`. It
+   indicates the operation must create an attribute.
+
+
+.. data:: XATTR_REPLACE
+
+   This is a possible value for the flags argument in :func:`setxattr`. It
+   indicates the operation must replace an existing attribute.
+
+
+.. function:: setxattr(path, attr, value, flags=0)
+
+   Set the extended filesystem attribute *attr* on *path* to *value*. *attr*
+   must be a bytes or str with no embedded NULs. If it is str, it is encoded
+   with the filesystem encoding. *flags* may be :data:`XATTR_REPLACE` or
+   :data:`XATTR_CREATE`. If :data:`XATTR_REPLACE` is given and the attribute
+   does not exist, ``EEXISTS`` will be raised. If :data:`XATTR_CREATE` is given
+   and the attribute already exists, the attribute will not be created and
+   ``ENODATA`` will be raised.
+
+   Availability: Linux
+
+   .. note::
+
+      A bug in Linux kernel versions less than 2.6.39 caused the flags argument
+      to be ignored on some filesystems.
+
+   .. versionadded:: 3.3
+
+
 .. function:: stat(path)
 
    Perform the equivalent of a :c:func:`stat` system call on the given path.
diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -14,6 +14,8 @@
 from test import support
 import contextlib
 import mmap
+import platform
+import re
 import uuid
 import asyncore
 import asynchat
@@ -1506,6 +1508,97 @@
                         raise
 
 
+def supports_extended_attributes():
+    if not hasattr(os, "setxattr"):
+        return False
+    try:
+        with open(support.TESTFN, "wb") as fp:
+            try:
+                os.fsetxattr(fp.fileno(), b"user.test", b"")
+            except OSError as e:
+                if e.errno != errno.ENOTSUP:
+                    raise
+                return False
+    finally:
+        support.unlink(support.TESTFN)
+    # Kernels < 2.6.39 don't respect setxattr flags.
+    kernel_version = platform.release()
+    m = re.match("2.6.(\d{1,2})", kernel_version)
+    return m is None or int(m.group(1)) >= 39
+
+
+ at unittest.skipUnless(supports_extended_attributes(),
+                     "no non-broken extended attribute support")
+class ExtendedAttributeTests(unittest.TestCase):
+
+    def tearDown(self):
+        support.unlink(support.TESTFN)
+
+    def _check_xattrs_str(self, s, getxattr, setxattr, removexattr, listxattr):
+        fn = support.TESTFN
+        open(fn, "wb").close()
+        with self.assertRaises(OSError) as cm:
+            getxattr(fn, s("user.test"))
+        self.assertEqual(cm.exception.errno, errno.ENODATA)
+        self.assertEqual(listxattr(fn), [])
+        setxattr(fn, s("user.test"), b"")
+        self.assertEqual(listxattr(fn), ["user.test"])
+        self.assertEqual(getxattr(fn, b"user.test"), b"")
+        setxattr(fn, s("user.test"), b"hello", os.XATTR_REPLACE)
+        self.assertEqual(getxattr(fn, b"user.test"), b"hello")
+        with self.assertRaises(OSError) as cm:
+            setxattr(fn, s("user.test"), b"bye", os.XATTR_CREATE)
+        self.assertEqual(cm.exception.errno, errno.EEXIST)
+        with self.assertRaises(OSError) as cm:
+            setxattr(fn, s("user.test2"), b"bye", os.XATTR_REPLACE)
+        self.assertEqual(cm.exception.errno, errno.ENODATA)
+        setxattr(fn, s("user.test2"), b"foo", os.XATTR_CREATE)
+        self.assertEqual(sorted(listxattr(fn)), ["user.test", "user.test2"])
+        removexattr(fn, s("user.test"))
+        with self.assertRaises(OSError) as cm:
+            getxattr(fn, s("user.test"))
+        self.assertEqual(cm.exception.errno, errno.ENODATA)
+        self.assertEqual(listxattr(fn), ["user.test2"])
+        self.assertEqual(getxattr(fn, s("user.test2")), b"foo")
+        setxattr(fn, s("user.test"), b"a"*1024)
+        self.assertEqual(getxattr(fn, s("user.test")), b"a"*1024)
+        removexattr(fn, s("user.test"))
+        many = sorted("user.test{}".format(i) for i in range(100))
+        for thing in many:
+            setxattr(fn, thing, b"x")
+        self.assertEqual(sorted(listxattr(fn)), many)
+
+    def _check_xattrs(self, *args):
+        def make_bytes(s):
+            return bytes(s, "ascii")
+        self._check_xattrs_str(str, *args)
+        support.unlink(support.TESTFN)
+        self._check_xattrs_str(make_bytes, *args)
+
+    def test_simple(self):
+        self._check_xattrs(os.getxattr, os.setxattr, os.removexattr,
+                           os.listxattr)
+
+    def test_lpath(self):
+        self._check_xattrs(os.lgetxattr, os.lsetxattr, os.lremovexattr,
+                           os.llistxattr)
+
+    def test_fds(self):
+        def getxattr(path, *args):
+            with open(path, "rb") as fp:
+                return os.fgetxattr(fp.fileno(), *args)
+        def setxattr(path, *args):
+            with open(path, "wb") as fp:
+                os.fsetxattr(fp.fileno(), *args)
+        def removexattr(path, *args):
+            with open(path, "wb") as fp:
+                os.fremovexattr(fp.fileno(), *args)
+        def listxattr(path, *args):
+            with open(path, "rb") as fp:
+                return os.flistxattr(fp.fileno(), *args)
+        self._check_xattrs(getxattr, setxattr, removexattr, listxattr)
+
+
 @support.reap_threads
 def test_main():
     support.run_unittest(
@@ -1529,6 +1622,7 @@
         LinkTests,
         TestSendfile,
         ProgramPriorityTests,
+        ExtendedAttributeTests,
     )
 
 if __name__ == "__main__":
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -107,6 +107,10 @@
 #include <sched.h>
 #endif
 
+#ifdef HAVE_ATTR_XATTR_H
+#include <attr/xattr.h>
+#endif
+
 #if defined(__FreeBSD__) || defined(__DragonFly__) || defined(__APPLE__)
 #ifdef HAVE_SYS_SOCKET_H
 #include <sys/socket.h>
@@ -9950,6 +9954,384 @@
 }
 #endif
 
+#ifdef HAVE_ATTR_XATTR_H
+
+static int
+try_getxattr(const char *path, const char *name,
+             ssize_t (*get)(const char *, const char *, void *, size_t),
+             Py_ssize_t buf_size, PyObject **res)
+{
+    PyObject *value;
+    Py_ssize_t len;
+
+    assert(buf_size <= XATTR_SIZE_MAX);
+    value = PyBytes_FromStringAndSize(NULL, buf_size);
+    if (!value)
+        return 0;
+    Py_BEGIN_ALLOW_THREADS;
+    len = get(path, name, PyBytes_AS_STRING(value), buf_size);
+    Py_END_ALLOW_THREADS;
+    if (len < 0) {
+        Py_DECREF(value);
+        if (errno == ERANGE) {
+            value = NULL;
+        }
+        else {
+            posix_error();
+            return 0;
+        }
+    }
+    else if (len != buf_size) {
+        /* Can only shrink. */
+        _PyBytes_Resize(&value, len);
+    }
+    *res = value;
+    return 1;
+}
+
+static PyObject *
+getxattr_common(const char *path, PyObject *name_obj,
+                ssize_t (*get)(const char *, const char *, void *, size_t))
+{
+    PyObject *value;
+    const char *name = PyBytes_AS_STRING(name_obj);
+
+    /* Try a small value first. */
+    if (!try_getxattr(path, name, get, 128, &value))
+        return NULL;
+    if (value)
+        return value;
+    /* Now the maximum possible one. */
+    if (!try_getxattr(path, name, get, XATTR_SIZE_MAX, &value))
+        return NULL;
+    assert(value);
+    return value;
+}
+
+PyDoc_STRVAR(posix_getxattr__doc__,
+"getxattr(path, attr) -> value\n\n\
+Return the value of extended attribute *name* on *path*.");
+
+static PyObject *
+posix_getxattr(PyObject *self, PyObject *args)
+{
+    PyObject *path, *res, *name;
+
+    if (!PyArg_ParseTuple(args, "O&O&:getxattr", PyUnicode_FSConverter, &path,
+                          PyUnicode_FSConverter, &name))
+        return NULL;
+    res = getxattr_common(PyBytes_AS_STRING(path), name, getxattr);
+    Py_DECREF(path);
+    Py_DECREF(name);
+    return res;
+}
+
+PyDoc_STRVAR(posix_lgetxattr__doc__,
+"lgetxattr(path, attr) -> value\n\n\
+Like getxattr but don't follow symlinks.");
+
+static PyObject *
+posix_lgetxattr(PyObject *self, PyObject *args)
+{
+    PyObject *path, *res, *name;
+
+    if (!PyArg_ParseTuple(args, "O&O&:lgetxattr", PyUnicode_FSConverter, &path,
+                          PyUnicode_FSConverter, &name))
+        return NULL;
+    res = getxattr_common(PyBytes_AS_STRING(path), name, lgetxattr);
+    Py_DECREF(path);
+    Py_DECREF(name);
+    return res;
+}
+
+static ssize_t
+wrap_fgetxattr(const char *path, const char *name, void *value, size_t size)
+{
+    /* Hack to share code. */
+    return fgetxattr((int)(Py_uintptr_t)path, name, value, size);
+}
+
+PyDoc_STRVAR(posix_fgetxattr__doc__,
+"fgetxattr(fd, attr) -> value\n\n\
+Like getxattr but operate on a fd instead of a path.");
+
+static PyObject *
+posix_fgetxattr(PyObject *self, PyObject *args)
+{
+    PyObject *res, *name;
+    int fd;
+
+    if (!PyArg_ParseTuple(args, "iO&:fgetxattr", &fd, PyUnicode_FSConverter, &name))
+        return NULL;
+    res = getxattr_common((const char *)(Py_uintptr_t)fd, name, wrap_fgetxattr);
+    Py_DECREF(name);
+    return res;
+}
+
+PyDoc_STRVAR(posix_setxattr__doc__,
+"setxattr(path, attr, value, flags=0)\n\n\
+Set extended attribute *attr* on *path* to *value*.");
+
+static PyObject *
+posix_setxattr(PyObject *self, PyObject *args)
+{
+    PyObject *path, *name;
+    Py_buffer data;
+    int flags = 0, err;
+
+    if (!PyArg_ParseTuple(args, "O&O&y*|i:setxattr", PyUnicode_FSConverter,
+                          &path, PyUnicode_FSConverter, &name, &data, &flags))
+        return NULL;
+    Py_BEGIN_ALLOW_THREADS;
+    err = setxattr(PyBytes_AS_STRING(path), PyBytes_AS_STRING(name),
+                   data.buf, data.len, flags);
+    Py_END_ALLOW_THREADS;
+    Py_DECREF(path);
+    Py_DECREF(name);
+    PyBuffer_Release(&data);
+    if (err)
+        return posix_error();
+    Py_RETURN_NONE;
+}
+
+PyDoc_STRVAR(posix_lsetxattr__doc__,
+"lsetxattr(path, attr, value, flags=0)\n\n\
+Like setxattr but don't follow symlinks.");
+
+static PyObject *
+posix_lsetxattr(PyObject *self, PyObject *args)
+{
+    PyObject *path, *name;
+    Py_buffer data;
+    int flags = 0, err;
+
+    if (!PyArg_ParseTuple(args, "O&O&y*|i:lsetxattr", PyUnicode_FSConverter,
+                          &path, PyUnicode_FSConverter, &name, &data, &flags))
+        return NULL;
+    Py_BEGIN_ALLOW_THREADS;
+    err = lsetxattr(PyBytes_AS_STRING(path), PyBytes_AS_STRING(name),
+                    data.buf, data.len, flags);
+    Py_END_ALLOW_THREADS;
+    Py_DECREF(path);
+    Py_DECREF(name);
+    PyBuffer_Release(&data);
+    if (err)
+        return posix_error();
+    Py_RETURN_NONE;
+}
+
+PyDoc_STRVAR(posix_fsetxattr__doc__,
+"fsetxattr(fd, attr, value, flags=0)\n\n\
+Like setxattr but operates on *fd* instead of a path.");
+
+static PyObject *
+posix_fsetxattr(PyObject *self, PyObject *args)
+{
+    Py_buffer data;
+    const char *name;
+    int fd, flags = 0, err;
+
+    if (!PyArg_ParseTuple(args, "iO&y*|i:fsetxattr", &fd, PyUnicode_FSConverter,
+                          &name, &data, &flags))
+        return NULL;
+    Py_BEGIN_ALLOW_THREADS;
+    err = fsetxattr(fd, PyBytes_AS_STRING(name), data.buf, data.len, flags);
+    Py_END_ALLOW_THREADS;
+    Py_DECREF(name);
+    PyBuffer_Release(&data);
+    if (err)
+        return posix_error();
+    Py_RETURN_NONE;
+}
+
+PyDoc_STRVAR(posix_removexattr__doc__,
+"removexattr(path, attr)\n\n\
+Remove extended attribute *attr* on *path*.");
+
+static PyObject *
+posix_removexattr(PyObject *self, PyObject *args)
+{
+    PyObject *path, *name;
+    int err;
+
+    if (!PyArg_ParseTuple(args, "O&O&:removexattr", PyUnicode_FSConverter, &path,
+                          PyUnicode_FSConverter, &name))
+        return NULL;
+    Py_BEGIN_ALLOW_THREADS;
+    err = removexattr(PyBytes_AS_STRING(path), PyBytes_AS_STRING(name));
+    Py_END_ALLOW_THREADS;
+    Py_DECREF(path);
+    Py_DECREF(name);
+    if (err)
+        return posix_error();
+    Py_RETURN_NONE;
+}
+
+PyDoc_STRVAR(posix_lremovexattr__doc__,
+"lremovexattr(path, attr)\n\n\
+Like removexattr but don't follow symlinks.");
+
+static PyObject *
+posix_lremovexattr(PyObject *self, PyObject *args)
+{
+    PyObject *path, *name;
+    int err;
+
+    if (!PyArg_ParseTuple(args, "O&O&:lremovexattr", PyUnicode_FSConverter, &path,
+                          PyUnicode_FSConverter, &name))
+        return NULL;
+    Py_BEGIN_ALLOW_THREADS;
+    err = lremovexattr(PyBytes_AS_STRING(path), PyBytes_AS_STRING(name));
+    Py_END_ALLOW_THREADS;
+    Py_DECREF(path);
+    Py_DECREF(name);
+    if (err)
+        return posix_error();
+    Py_RETURN_NONE;
+}
+
+PyDoc_STRVAR(posix_fremovexattr__doc__,
+"fremovexattr(fd, attr)\n\n\
+Like removexattr but operates on a file descriptor.");
+
+static PyObject *
+posix_fremovexattr(PyObject *self, PyObject *args)
+{
+    PyObject *name;
+    int fd, err;
+
+    if (!PyArg_ParseTuple(args, "iO&:fremovexattr", &fd,
+                          PyUnicode_FSConverter, &name))
+        return NULL;
+    Py_BEGIN_ALLOW_THREADS;
+    err = fremovexattr(fd, PyBytes_AS_STRING(name));
+    Py_END_ALLOW_THREADS;
+    Py_DECREF(name);
+    if (err)
+        return posix_error();
+    Py_RETURN_NONE;
+}
+
+static Py_ssize_t
+try_listxattr(const char *path, ssize_t (*list)(const char *, char *, size_t),
+              Py_ssize_t buf_size, char **buf)
+{
+    Py_ssize_t len;
+
+    *buf = PyMem_MALLOC(buf_size);
+    if (!*buf) {
+        PyErr_NoMemory();
+        return -1;
+    }
+    Py_BEGIN_ALLOW_THREADS;
+    len = list(path, *buf, buf_size);
+    Py_END_ALLOW_THREADS;
+    if (len < 0) {
+        PyMem_FREE(*buf);
+        if (errno != ERANGE)
+            posix_error();
+        return -1;
+    }
+    return len;
+}
+
+static PyObject *
+listxattr_common(const char *path, ssize_t (*list)(const char *, char *, size_t))
+{
+    PyObject *res, *attr;
+    Py_ssize_t len, err, start, i;
+    char *buf;
+
+    len = try_listxattr(path, list, 256, &buf);
+    if (len < 0) {
+        if (PyErr_Occurred())
+            return NULL;
+        len = try_listxattr(path, list, XATTR_LIST_MAX, &buf);
+        if (len < 0)
+            return NULL;
+    }
+    res = PyList_New(0);
+    if (!res) {
+        PyMem_FREE(buf);
+        return NULL;
+    }
+    for (start = i = 0; i < len; i++) {
+        if (!buf[i]) {
+            attr = PyUnicode_DecodeFSDefaultAndSize(&buf[start], i - start);
+            if (!attr) {
+                Py_DECREF(res);
+                PyMem_FREE(buf);
+                return NULL;
+            }
+            err = PyList_Append(res, attr);
+            Py_DECREF(attr);
+            if (err) {
+                Py_DECREF(res);
+                PyMem_FREE(buf);
+                return NULL;
+            }
+            start = i + 1;
+        }
+    }
+    PyMem_FREE(buf);
+    return res;
+}
+
+PyDoc_STRVAR(posix_listxattr__doc__,
+"listxattr(path)\n\n\
+Return a list of extended attributes on *path*.");
+
+static PyObject *
+posix_listxattr(PyObject *self, PyObject *args)
+{
+    PyObject *path, *res;
+
+    if (!PyArg_ParseTuple(args, "O&:listxattr", PyUnicode_FSConverter, &path))
+        return NULL;
+    res = listxattr_common(PyBytes_AS_STRING(path), listxattr);
+    Py_DECREF(path);
+    return res;
+}
+
+PyDoc_STRVAR(posix_llistxattr__doc__,
+"llistxattr(path)\n\n\
+Like listxattr but don't follow symlinks..");
+
+static PyObject *
+posix_llistxattr(PyObject *self, PyObject *args)
+{
+    PyObject *path, *res;
+
+    if (!PyArg_ParseTuple(args, "O&:llistxattr", PyUnicode_FSConverter, &path))
+        return NULL;
+    res = listxattr_common(PyBytes_AS_STRING(path), llistxattr);
+    Py_DECREF(path);
+    return res;
+}
+
+static ssize_t
+wrap_flistxattr(const char *path, char *buf, size_t len)
+{
+    /* Hack to share code. */
+    return flistxattr((int)(Py_uintptr_t)path, buf, len);
+}
+
+PyDoc_STRVAR(posix_flistxattr__doc__,
+"flistxattr(path)\n\n\
+Like flistxattr but operates on a file descriptor.");
+
+static PyObject *
+posix_flistxattr(PyObject *self, PyObject *args)
+{
+    long fd;
+
+    if (!PyArg_ParseTuple(args, "i:flistxattr", &fd))
+        return NULL;
+    return listxattr_common((const char *)(Py_uintptr_t)fd, wrap_flistxattr);
+}
+
+#endif /* HAVE_ATTR_XATTR_H */
+
 static PyMethodDef posix_methods[] = {
     {"access",          posix_access, METH_VARARGS, posix_access__doc__},
 #ifdef HAVE_TTYNAME
@@ -10399,6 +10781,20 @@
 #ifdef HAVE_MKFIFOAT
     {"mkfifoat",        posix_mkfifoat, METH_VARARGS, posix_mkfifoat__doc__},
 #endif
+#ifdef HAVE_ATTR_XATTR_H
+    {"setxattr", posix_setxattr, METH_VARARGS, posix_setxattr__doc__},
+    {"lsetxattr", posix_lsetxattr, METH_VARARGS, posix_lsetxattr__doc__},
+    {"fsetxattr", posix_fsetxattr, METH_VARARGS, posix_fsetxattr__doc__},
+    {"getxattr", posix_getxattr, METH_VARARGS, posix_getxattr__doc__},
+    {"lgetxattr", posix_lgetxattr, METH_VARARGS, posix_lgetxattr__doc__},
+    {"fgetxattr", posix_fgetxattr, METH_VARARGS, posix_fgetxattr__doc__},
+    {"removexattr", posix_removexattr, METH_VARARGS, posix_removexattr__doc__},
+    {"lremovexattr", posix_lremovexattr, METH_VARARGS, posix_lremovexattr__doc__},
+    {"fremovexattr", posix_fremovexattr, METH_VARARGS, posix_fremovexattr__doc__},
+    {"listxattr", posix_listxattr, METH_VARARGS, posix_listxattr__doc__},
+    {"llistxattr", posix_llistxattr, METH_VARARGS, posix_llistxattr__doc__},
+    {"flistxattr", posix_flistxattr, METH_VARARGS, posix_flistxattr__doc__},
+#endif
     {NULL,              NULL}            /* Sentinel */
 };
 
@@ -10848,6 +11244,12 @@
 #endif
 #endif
 
+#ifdef HAVE_ATTR_XATTR_H
+    if (ins(d, "XATTR_CREATE", (long)XATTR_CREATE)) return -1;
+    if (ins(d, "XATTR_REPLACE", (long)XATTR_REPLACE)) return -1;
+    if (ins(d, "XATTR_SIZE_MAX", (long)XATTR_SIZE_MAX)) return -1;
+#endif
+
 #if defined(PYOS_OS2)
     if (insertvalues(d)) return -1;
 #endif
diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -6090,7 +6090,7 @@
 
 fi
 
-for ac_header in asm/types.h conio.h curses.h direct.h dlfcn.h errno.h \
+for ac_header in asm/types.h attr/xattr.h conio.h curses.h direct.h dlfcn.h errno.h \
 fcntl.h grp.h \
 ieeefp.h io.h langinfo.h libintl.h ncurses.h poll.h process.h pthread.h \
 sched.h shadow.h signal.h stdint.h stropts.h termios.h \
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -1299,7 +1299,7 @@
 
 # checks for header files
 AC_HEADER_STDC
-AC_CHECK_HEADERS(asm/types.h conio.h curses.h direct.h dlfcn.h errno.h \
+AC_CHECK_HEADERS(asm/types.h attr/xattr.h conio.h curses.h direct.h dlfcn.h errno.h \
 fcntl.h grp.h \
 ieeefp.h io.h langinfo.h libintl.h ncurses.h poll.h process.h pthread.h \
 sched.h shadow.h signal.h stdint.h stropts.h termios.h \
diff --git a/pyconfig.h.in b/pyconfig.h.in
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -64,6 +64,9 @@
 /* Define if GCC supports __attribute__((format(PyArg_ParseTuple, 2, 3))) */
 #undef HAVE_ATTRIBUTE_FORMAT_PARSETUPLE
 
+/* Define to 1 if you have the <attr/xattr.h> header file. */
+#undef HAVE_ATTR_XATTR_H
+
 /* Define to 1 if you have the `bind_textdomain_codeset' function. */
 #undef HAVE_BIND_TEXTDOMAIN_CODESET
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep  1 04:16:44 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Thu, 01 Sep 2011 04:16:44 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?b?OiBtZXJnZSAzLjIgKG51bGwp?=
Message-ID: <E1Qywpg-0007ku-GQ@dinsdale.python.org>

http://hg.python.org/cpython/rev/cfccc755c457
changeset:   72156:cfccc755c457
parent:      72155:4eb0b1819bda
parent:      72154:c8b6d4417274
user:        Benjamin Peterson <benjamin at python.org>
date:        Wed Aug 31 22:16:34 2011 -0400
summary:
  merge 3.2 (null)

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep  1 04:21:52 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Thu, 01 Sep 2011 04:21:52 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_news_note_on_xattrs?=
Message-ID: <E1Qywue-00083S-S1@dinsdale.python.org>

http://hg.python.org/cpython/rev/74671651540d
changeset:   72157:74671651540d
user:        Benjamin Peterson <benjamin at python.org>
date:        Wed Aug 31 22:21:44 2011 -0400
summary:
  news note on xattrs

files:
  Misc/NEWS |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -268,6 +268,8 @@
 Library
 -------
 
+- Issue #12720: Expose low-level Linux extended file attribute functions in os.
+
 - Issue #10946: The distutils commands bdist_dumb, bdist_wininst and bdist_msi
   now respect a --skip-build option given to bdist.  The packaging commands
   were fixed too.

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Thu Sep  1 05:17:21 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Thu, 01 Sep 2011 05:17:21 +0200
Subject: [Python-checkins] Daily reference leaks (0968acf0e6db): sum=0
Message-ID: <E1QyxmL-000456-Rf@ap.vmr.nerim.net>

results for 0968acf0e6db on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogjZSCdh', '-x']

From python-checkins at python.org  Thu Sep  1 07:03:31 2011
From: python-checkins at python.org (ross.lagerwall)
Date: Thu, 01 Sep 2011 07:03:31 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Remove_duplicat?=
 =?utf8?q?e_text_in_os_documentation=2E?=
Message-ID: <E1QyzR5-0003uR-4h@dinsdale.python.org>

http://hg.python.org/cpython/rev/65bd22be72dc
changeset:   72158:65bd22be72dc
branch:      3.2
parent:      72154:c8b6d4417274
user:        Ross Lagerwall <rosslagerwall at gmail.com>
date:        Thu Sep 01 06:58:52 2011 +0200
summary:
  Remove duplicate text in os documentation.

files:
  Doc/library/os.rst |  5 -----
  1 files changed, 0 insertions(+), 5 deletions(-)


diff --git a/Doc/library/os.rst b/Doc/library/os.rst
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -29,11 +29,6 @@
   objects, and result in an object of the same type, if a path or file name is
   returned.
 
-.. note::
-
-   If not separately noted, all functions that claim "Availability: Unix" are
-   supported on Mac OS X, which builds on a Unix core.
-
 * An "Availability: Unix" note means that this function is commonly found on
   Unix systems.  It does not make any claims about its existence on a specific
   operating system.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep  1 07:03:31 2011
From: python-checkins at python.org (ross.lagerwall)
Date: Thu, 01 Sep 2011 07:03:31 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge_with_3=2E2=2E?=
Message-ID: <E1QyzR5-0003ub-Sz@dinsdale.python.org>

http://hg.python.org/cpython/rev/e5c2d623c20b
changeset:   72159:e5c2d623c20b
parent:      72157:74671651540d
parent:      72158:65bd22be72dc
user:        Ross Lagerwall <rosslagerwall at gmail.com>
date:        Thu Sep 01 07:02:31 2011 +0200
summary:
  Merge with 3.2.

files:
  Doc/library/os.rst |  5 -----
  1 files changed, 0 insertions(+), 5 deletions(-)


diff --git a/Doc/library/os.rst b/Doc/library/os.rst
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -29,11 +29,6 @@
   objects, and result in an object of the same type, if a path or file name is
   returned.
 
-.. note::
-
-   If not separately noted, all functions that claim "Availability: Unix" are
-   supported on Mac OS X, which builds on a Unix core.
-
 * An "Availability: Unix" note means that this function is commonly found on
   Unix systems.  It does not make any claims about its existence on a specific
   operating system.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep  1 07:19:23 2011
From: python-checkins at python.org (ezio.melotti)
Date: Thu, 01 Sep 2011 07:19:23 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_From_RFC_3629_5?=
 =?utf8?q?-_and_6-bytes_UTF-8_sequences_are_invalid=2C_so_remove_them_from?=
Message-ID: <E1QyzgR-0004um-U6@dinsdale.python.org>

http://hg.python.org/cpython/rev/106c57d52b47
changeset:   72160:106c57d52b47
branch:      3.2
parent:      72158:65bd22be72dc
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Sep 01 08:11:28 2011 +0300
summary:
  From RFC 3629 5- and 6-bytes UTF-8 sequences are invalid, so remove them from the doc.

files:
  Doc/library/codecs.rst |  9 ++-------
  1 files changed, 2 insertions(+), 7 deletions(-)


diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst
--- a/Doc/library/codecs.rst
+++ b/Doc/library/codecs.rst
@@ -839,7 +839,7 @@
 characters: UTF-8. UTF-8 is an 8-bit encoding, which means there are no issues
 with byte order in UTF-8. Each byte in a UTF-8 byte sequence consists of two
 parts: Marker bits (the most significant bits) and payload bits. The marker bits
-are a sequence of zero to six 1 bits followed by a 0 bit. Unicode characters are
+are a sequence of zero to four ``1`` bits followed by a ``0`` bit. Unicode characters are
 encoded like this (with x being payload bits, which when concatenated give the
 Unicode character):
 
@@ -852,12 +852,7 @@
 +-----------------------------------+----------------------------------------------+
 | ``U-00000800`` ... ``U-0000FFFF`` | 1110xxxx 10xxxxxx 10xxxxxx                   |
 +-----------------------------------+----------------------------------------------+
-| ``U-00010000`` ... ``U-001FFFFF`` | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx          |
-+-----------------------------------+----------------------------------------------+
-| ``U-00200000`` ... ``U-03FFFFFF`` | 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx |
-+-----------------------------------+----------------------------------------------+
-| ``U-04000000`` ... ``U-7FFFFFFF`` | 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx |
-|                                   | 10xxxxxx                                     |
+| ``U-00010000`` ... ``U-0010FFFF`` | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx          |
 +-----------------------------------+----------------------------------------------+
 
 The least significant bit of the Unicode character is the rightmost x bit.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep  1 07:19:24 2011
From: python-checkins at python.org (ezio.melotti)
Date: Thu, 01 Sep 2011 07:19:24 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_doc_fix_with_3=2E2=2E?=
Message-ID: <E1QyzgS-0004uw-Li@dinsdale.python.org>

http://hg.python.org/cpython/rev/53e532f5e698
changeset:   72161:53e532f5e698
parent:      72159:e5c2d623c20b
parent:      72160:106c57d52b47
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Sep 01 08:13:46 2011 +0300
summary:
  Merge doc fix with 3.2.

files:
  Doc/library/codecs.rst |  9 ++-------
  1 files changed, 2 insertions(+), 7 deletions(-)


diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst
--- a/Doc/library/codecs.rst
+++ b/Doc/library/codecs.rst
@@ -840,7 +840,7 @@
 characters: UTF-8. UTF-8 is an 8-bit encoding, which means there are no issues
 with byte order in UTF-8. Each byte in a UTF-8 byte sequence consists of two
 parts: Marker bits (the most significant bits) and payload bits. The marker bits
-are a sequence of zero to six 1 bits followed by a 0 bit. Unicode characters are
+are a sequence of zero to four ``1`` bits followed by a ``0`` bit. Unicode characters are
 encoded like this (with x being payload bits, which when concatenated give the
 Unicode character):
 
@@ -853,12 +853,7 @@
 +-----------------------------------+----------------------------------------------+
 | ``U-00000800`` ... ``U-0000FFFF`` | 1110xxxx 10xxxxxx 10xxxxxx                   |
 +-----------------------------------+----------------------------------------------+
-| ``U-00010000`` ... ``U-001FFFFF`` | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx          |
-+-----------------------------------+----------------------------------------------+
-| ``U-00200000`` ... ``U-03FFFFFF`` | 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx |
-+-----------------------------------+----------------------------------------------+
-| ``U-04000000`` ... ``U-7FFFFFFF`` | 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx |
-|                                   | 10xxxxxx                                     |
+| ``U-00010000`` ... ``U-0010FFFF`` | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx          |
 +-----------------------------------+----------------------------------------------+
 
 The least significant bit of the Unicode character is the rightmost x bit.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep  1 07:19:25 2011
From: python-checkins at python.org (ezio.melotti)
Date: Thu, 01 Sep 2011 07:19:25 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_From_RFC_3629_5?=
 =?utf8?q?-_and_6-bytes_UTF-8_sequences_are_invalid=2C_so_remove_them_from?=
Message-ID: <E1QyzgT-0004v0-DW@dinsdale.python.org>

http://hg.python.org/cpython/rev/4d584ebbfa77
changeset:   72162:4d584ebbfa77
branch:      2.7
parent:      72153:4dcbae65df3f
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Sep 01 08:19:01 2011 +0300
summary:
  From RFC 3629 5- and 6-bytes UTF-8 sequences are invalid, so remove them from the doc.

files:
  Doc/library/codecs.rst |  9 ++-------
  1 files changed, 2 insertions(+), 7 deletions(-)


diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst
--- a/Doc/library/codecs.rst
+++ b/Doc/library/codecs.rst
@@ -811,7 +811,7 @@
 characters: UTF-8. UTF-8 is an 8-bit encoding, which means there are no issues
 with byte order in UTF-8. Each byte in a UTF-8 byte sequence consists of two
 parts: Marker bits (the most significant bits) and payload bits. The marker bits
-are a sequence of zero to six 1 bits followed by a 0 bit. Unicode characters are
+are a sequence of zero to four ``1`` bits followed by a ``0`` bit. Unicode characters are
 encoded like this (with x being payload bits, which when concatenated give the
 Unicode character):
 
@@ -824,12 +824,7 @@
 +-----------------------------------+----------------------------------------------+
 | ``U-00000800`` ... ``U-0000FFFF`` | 1110xxxx 10xxxxxx 10xxxxxx                   |
 +-----------------------------------+----------------------------------------------+
-| ``U-00010000`` ... ``U-001FFFFF`` | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx          |
-+-----------------------------------+----------------------------------------------+
-| ``U-00200000`` ... ``U-03FFFFFF`` | 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx |
-+-----------------------------------+----------------------------------------------+
-| ``U-04000000`` ... ``U-7FFFFFFF`` | 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx |
-|                                   | 10xxxxxx                                     |
+| ``U-00010000`` ... ``U-0010FFFF`` | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx          |
 +-----------------------------------+----------------------------------------------+
 
 The least significant bit of the Unicode character is the rightmost x bit.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep  1 21:41:13 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 01 Sep 2011 21:41:13 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyODAy?=
 =?utf8?q?=3A_the_Windows_error_ERROR=5FDIRECTORY_=28numbered_267=29_is_no?=
 =?utf8?q?w?=
Message-ID: <E1QzD8T-0004zU-29@dinsdale.python.org>

http://hg.python.org/cpython/rev/385c2ec78f16
changeset:   72163:385c2ec78f16
branch:      3.2
parent:      72160:106c57d52b47
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu Sep 01 21:37:43 2011 +0200
summary:
  Issue #12802: the Windows error ERROR_DIRECTORY (numbered 267) is now
mapped to POSIX errno ENOTDIR (previously EINVAL).

files:
  Lib/test/test_exceptions.py |   8 ++++++++
  Misc/NEWS                   |   3 +++
  PC/errmap.h                 |   1 +
  PC/generrmap.c              |  21 +++++++++++++++------
  4 files changed, 27 insertions(+), 6 deletions(-)


diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py
--- a/Lib/test/test_exceptions.py
+++ b/Lib/test/test_exceptions.py
@@ -5,6 +5,7 @@
 import unittest
 import pickle
 import weakref
+import errno
 
 from test.support import (TESTFN, unlink, run_unittest, captured_output,
                           gc_collect, cpython_only)
@@ -849,6 +850,13 @@
             self.fail("RuntimeError not raised")
         self.assertEqual(wr(), None)
 
+    def test_errno_ENOTDIR(self):
+        # Issue #12802: "not a directory" errors are ENOTDIR even on Windows
+        with self.assertRaises(OSError) as cm:
+            os.listdir(__file__)
+        self.assertEqual(cm.exception.errno, errno.ENOTDIR, cm.exception)
+
+
 def test_main():
     run_unittest(ExceptionTests)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #12802: the Windows error ERROR_DIRECTORY (numbered 267) is now
+  mapped to POSIX errno ENOTDIR (previously EINVAL).
+
 - Accept bytes for the AST string type. This is temporary until a proper fix in
   3.3.
 
diff --git a/PC/errmap.h b/PC/errmap.h
--- a/PC/errmap.h
+++ b/PC/errmap.h
@@ -72,6 +72,7 @@
         case 202: return 8;
         case 206: return 2;
         case 215: return 11;
+        case 267: return 20;
         case 1816: return 12;
         default: return EINVAL;
     }
diff --git a/PC/generrmap.c b/PC/generrmap.c
--- a/PC/generrmap.c
+++ b/PC/generrmap.c
@@ -1,3 +1,6 @@
+#include <windows.h>
+#include <fcntl.h>
+#include <io.h>
 #include <stdio.h>
 #include <errno.h>
 
@@ -6,15 +9,21 @@
 int main()
 {
     int i;
+    _setmode(fileno(stdout), O_BINARY);
     printf("/* Generated file. Do not edit. */\n");
     printf("int winerror_to_errno(int winerror)\n");
-    printf("{\n\tswitch(winerror) {\n");
+    printf("{\n    switch(winerror) {\n");
     for(i=1; i < 65000; i++) {
         _dosmaperr(i);
-        if (errno == EINVAL)
-            continue;
-        printf("\t\tcase %d: return %d;\n", i, errno);
+        if (errno == EINVAL) {
+            /* Issue #12802 */
+            if (i == ERROR_DIRECTORY)
+                errno = ENOTDIR;
+            else
+                continue;
+        }
+        printf("        case %d: return %d;\n", i, errno);
     }
-    printf("\t\tdefault: return EINVAL;\n");
-    printf("\t}\n}\n");
+    printf("        default: return EINVAL;\n");
+    printf("    }\n}\n");
 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep  1 21:41:13 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 01 Sep 2011 21:41:13 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2312802=3A_the_Windows_error_ERROR=5FDIRECTORY_=28num?=
 =?utf8?q?bered_267=29_is_now?=
Message-ID: <E1QzD8T-0004zY-QY@dinsdale.python.org>

http://hg.python.org/cpython/rev/d72d5c942232
changeset:   72164:d72d5c942232
parent:      72161:53e532f5e698
parent:      72163:385c2ec78f16
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu Sep 01 21:38:37 2011 +0200
summary:
  Issue #12802: the Windows error ERROR_DIRECTORY (numbered 267) is now
mapped to POSIX errno ENOTDIR (previously EINVAL).

files:
  Lib/test/test_exceptions.py |   8 ++++++++
  Misc/NEWS                   |   3 +++
  PC/errmap.h                 |   1 +
  PC/generrmap.c              |  21 +++++++++++++++------
  4 files changed, 27 insertions(+), 6 deletions(-)


diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py
--- a/Lib/test/test_exceptions.py
+++ b/Lib/test/test_exceptions.py
@@ -5,6 +5,7 @@
 import unittest
 import pickle
 import weakref
+import errno
 
 from test.support import (TESTFN, unlink, run_unittest, captured_output,
                           gc_collect, cpython_only, no_tracing)
@@ -852,6 +853,13 @@
             self.fail("RuntimeError not raised")
         self.assertEqual(wr(), None)
 
+    def test_errno_ENOTDIR(self):
+        # Issue #12802: "not a directory" errors are ENOTDIR even on Windows
+        with self.assertRaises(OSError) as cm:
+            os.listdir(__file__)
+        self.assertEqual(cm.exception.errno, errno.ENOTDIR, cm.exception)
+
+
 def test_main():
     run_unittest(ExceptionTests)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #12802: the Windows error ERROR_DIRECTORY (numbered 267) is now
+  mapped to POSIX errno ENOTDIR (previously EINVAL).
+
 - Issue #9200: The str.is* methods now work with strings that contain non-BMP
   characters even in narrow Unicode builds.
 
diff --git a/PC/errmap.h b/PC/errmap.h
--- a/PC/errmap.h
+++ b/PC/errmap.h
@@ -72,6 +72,7 @@
         case 202: return 8;
         case 206: return 2;
         case 215: return 11;
+        case 267: return 20;
         case 1816: return 12;
         default: return EINVAL;
     }
diff --git a/PC/generrmap.c b/PC/generrmap.c
--- a/PC/generrmap.c
+++ b/PC/generrmap.c
@@ -1,3 +1,6 @@
+#include <windows.h>
+#include <fcntl.h>
+#include <io.h>
 #include <stdio.h>
 #include <errno.h>
 
@@ -6,15 +9,21 @@
 int main()
 {
     int i;
+    _setmode(fileno(stdout), O_BINARY);
     printf("/* Generated file. Do not edit. */\n");
     printf("int winerror_to_errno(int winerror)\n");
-    printf("{\n\tswitch(winerror) {\n");
+    printf("{\n    switch(winerror) {\n");
     for(i=1; i < 65000; i++) {
         _dosmaperr(i);
-        if (errno == EINVAL)
-            continue;
-        printf("\t\tcase %d: return %d;\n", i, errno);
+        if (errno == EINVAL) {
+            /* Issue #12802 */
+            if (i == ERROR_DIRECTORY)
+                errno = ENOTDIR;
+            else
+                continue;
+        }
+        printf("        case %d: return %d;\n", i, errno);
     }
-    printf("\t\tdefault: return EINVAL;\n");
-    printf("\t}\n}\n");
+    printf("        default: return EINVAL;\n");
+    printf("    }\n}\n");
 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep  1 22:42:37 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Thu, 01 Sep 2011 22:42:37 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_make_sure_to_in?=
 =?utf8?q?itialize_the_method_wrapper_type?=
Message-ID: <E1QzE5t-00029J-Au@dinsdale.python.org>

http://hg.python.org/cpython/rev/51e27f42beda
changeset:   72165:51e27f42beda
branch:      3.2
parent:      72163:385c2ec78f16
user:        Benjamin Peterson <benjamin at python.org>
date:        Thu Sep 01 16:32:31 2011 -0400
summary:
  make sure to initialize the method wrapper type

files:
  Include/descrobject.h |  1 +
  Objects/descrobject.c |  9 +++------
  Objects/object.c      |  3 +++
  3 files changed, 7 insertions(+), 6 deletions(-)


diff --git a/Include/descrobject.h b/Include/descrobject.h
--- a/Include/descrobject.h
+++ b/Include/descrobject.h
@@ -77,6 +77,7 @@
 PyAPI_DATA(PyTypeObject) PyMethodDescr_Type;
 PyAPI_DATA(PyTypeObject) PyWrapperDescr_Type;
 PyAPI_DATA(PyTypeObject) PyDictProxy_Type;
+PyAPI_DATA(PyTypeObject) _PyMethodWrapper_Type;
 
 PyAPI_FUNC(PyObject *) PyDescr_NewMethod(PyTypeObject *, PyMethodDef *);
 PyAPI_FUNC(PyObject *) PyDescr_NewClassMethod(PyTypeObject *, PyMethodDef *);
diff --git a/Objects/descrobject.c b/Objects/descrobject.c
--- a/Objects/descrobject.c
+++ b/Objects/descrobject.c
@@ -846,16 +846,13 @@
 /* This has no reason to be in this file except that adding new files is a
    bit of a pain */
 
-/* forward */
-static PyTypeObject wrappertype;
-
 typedef struct {
     PyObject_HEAD
     PyWrapperDescrObject *descr;
     PyObject *self;
 } wrapperobject;
 
-#define Wrapper_Check(v) (Py_TYPE(v) == &wrappertype)
+#define Wrapper_Check(v) (Py_TYPE(v) == &_PyMethodWrapper_Type)
 
 static void
 wrapper_dealloc(wrapperobject *wp)
@@ -1021,7 +1018,7 @@
     return 0;
 }
 
-static PyTypeObject wrappertype = {
+PyTypeObject _PyMethodWrapper_Type = {
     PyVarObject_HEAD_INIT(&PyType_Type, 0)
     "method-wrapper",                           /* tp_name */
     sizeof(wrapperobject),                      /* tp_basicsize */
@@ -1070,7 +1067,7 @@
     assert(_PyObject_RealIsSubclass((PyObject *)Py_TYPE(self),
                                     (PyObject *)PyDescr_TYPE(descr)));
 
-    wp = PyObject_GC_New(wrapperobject, &wrappertype);
+    wp = PyObject_GC_New(wrapperobject, &_PyMethodWrapper_Type);
     if (wp != NULL) {
         Py_INCREF(descr);
         wp->descr = descr;
diff --git a/Objects/object.c b/Objects/object.c
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -1625,6 +1625,9 @@
     if (PyType_Ready(&PyWrapperDescr_Type) < 0)
         Py_FatalError("Can't initialize wrapper type");
 
+    if (PyType_Ready(&_PyMethodWrapper_Type) < 0)
+        Py_FatalError("Can't initialize method wrapper type");
+
     if (PyType_Ready(&PyEllipsis_Type) < 0)
         Py_FatalError("Can't initialize ellipsis type");
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep  1 22:42:38 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Thu, 01 Sep 2011 22:42:38 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_merge_3=2E2?=
Message-ID: <E1QzE5u-00029U-1m@dinsdale.python.org>

http://hg.python.org/cpython/rev/0212858cae57
changeset:   72166:0212858cae57
parent:      72164:d72d5c942232
parent:      72165:51e27f42beda
user:        Benjamin Peterson <benjamin at python.org>
date:        Thu Sep 01 16:33:56 2011 -0400
summary:
  merge 3.2

files:
  Include/descrobject.h |  1 +
  Objects/descrobject.c |  9 +++------
  Objects/object.c      |  3 +++
  3 files changed, 7 insertions(+), 6 deletions(-)


diff --git a/Include/descrobject.h b/Include/descrobject.h
--- a/Include/descrobject.h
+++ b/Include/descrobject.h
@@ -77,6 +77,7 @@
 PyAPI_DATA(PyTypeObject) PyMethodDescr_Type;
 PyAPI_DATA(PyTypeObject) PyWrapperDescr_Type;
 PyAPI_DATA(PyTypeObject) PyDictProxy_Type;
+PyAPI_DATA(PyTypeObject) _PyMethodWrapper_Type;
 
 PyAPI_FUNC(PyObject *) PyDescr_NewMethod(PyTypeObject *, PyMethodDef *);
 PyAPI_FUNC(PyObject *) PyDescr_NewClassMethod(PyTypeObject *, PyMethodDef *);
diff --git a/Objects/descrobject.c b/Objects/descrobject.c
--- a/Objects/descrobject.c
+++ b/Objects/descrobject.c
@@ -846,16 +846,13 @@
 /* This has no reason to be in this file except that adding new files is a
    bit of a pain */
 
-/* forward */
-static PyTypeObject wrappertype;
-
 typedef struct {
     PyObject_HEAD
     PyWrapperDescrObject *descr;
     PyObject *self;
 } wrapperobject;
 
-#define Wrapper_Check(v) (Py_TYPE(v) == &wrappertype)
+#define Wrapper_Check(v) (Py_TYPE(v) == &_PyMethodWrapper_Type)
 
 static void
 wrapper_dealloc(wrapperobject *wp)
@@ -1021,7 +1018,7 @@
     return 0;
 }
 
-static PyTypeObject wrappertype = {
+PyTypeObject _PyMethodWrapper_Type = {
     PyVarObject_HEAD_INIT(&PyType_Type, 0)
     "method-wrapper",                           /* tp_name */
     sizeof(wrapperobject),                      /* tp_basicsize */
@@ -1070,7 +1067,7 @@
     assert(_PyObject_RealIsSubclass((PyObject *)Py_TYPE(self),
                                     (PyObject *)PyDescr_TYPE(descr)));
 
-    wp = PyObject_GC_New(wrapperobject, &wrappertype);
+    wp = PyObject_GC_New(wrapperobject, &_PyMethodWrapper_Type);
     if (wp != NULL) {
         Py_INCREF(descr);
         wp->descr = descr;
diff --git a/Objects/object.c b/Objects/object.c
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -1558,6 +1558,9 @@
     if (PyType_Ready(&PyWrapperDescr_Type) < 0)
         Py_FatalError("Can't initialize wrapper type");
 
+    if (PyType_Ready(&_PyMethodWrapper_Type) < 0)
+        Py_FatalError("Can't initialize method wrapper type");
+
     if (PyType_Ready(&PyEllipsis_Type) < 0)
         Py_FatalError("Can't initialize ellipsis type");
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep  1 23:07:26 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Thu, 01 Sep 2011 23:07:26 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312868=3A_Skip_test?=
 =?utf8?q?=5Ffaulthandler=2Etest=5Fstack=5Foverflow=28=29_on_OpenBSD=3A?=
Message-ID: <E1QzETu-0004QE-5E@dinsdale.python.org>

http://hg.python.org/cpython/rev/a29b72950795
changeset:   72167:a29b72950795
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Thu Sep 01 23:08:21 2011 +0200
summary:
  Issue #12868: Skip test_faulthandler.test_stack_overflow() on OpenBSD:
sigaltstack(2) doesn't work when linked with pthread.

files:
  Lib/test/test_faulthandler.py |  3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_faulthandler.py b/Lib/test/test_faulthandler.py
--- a/Lib/test/test_faulthandler.py
+++ b/Lib/test/test_faulthandler.py
@@ -174,6 +174,9 @@
             2,
             'xyz')
 
+    @unittest.skipIf(sys.platform.startswith('openbsd') and HAVE_THREADS,
+                     "Issue #12868: sigaltstack() doesn't work on "
+                     "OpenBSD if Python is compiled with pthread")
     @unittest.skipIf(not hasattr(faulthandler, '_stack_overflow'),
                      'need faulthandler._stack_overflow()')
     def test_stack_overflow(self):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep  1 23:54:23 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 01 Sep 2011 23:54:23 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312494=3A_Close_pip?=
 =?utf8?q?es_and_kill_process_on_error_in_subprocess_functions?=
Message-ID: <E1QzFDL-0008Ld-2v@dinsdale.python.org>

http://hg.python.org/cpython/rev/86b7f14485c9
changeset:   72168:86b7f14485c9
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 01 23:45:04 2011 +0200
summary:
  Issue #12494: Close pipes and kill process on error in subprocess functions

On error, call(), check_call(), check_output() and getstatusoutput() functions
of the subprocess module now kill the process, read its status (to avoid
zombis) and close pipes.

files:
  Lib/subprocess.py |  56 +++++++++++++++++++++-------------
  Misc/NEWS         |   4 ++
  2 files changed, 38 insertions(+), 22 deletions(-)


diff --git a/Lib/subprocess.py b/Lib/subprocess.py
--- a/Lib/subprocess.py
+++ b/Lib/subprocess.py
@@ -464,13 +464,13 @@
 
     retcode = call(["ls", "-l"])
     """
-    p = Popen(*popenargs, **kwargs)
-    try:
-        return p.wait(timeout=timeout)
-    except TimeoutExpired:
-        p.kill()
-        p.wait()
-        raise
+    with Popen(*popenargs, **kwargs) as p:
+        try:
+            return p.wait(timeout=timeout)
+        except:
+            p.kill()
+            p.wait()
+            raise
 
 
 def check_call(*popenargs, **kwargs):
@@ -514,16 +514,20 @@
     """
     if 'stdout' in kwargs:
         raise ValueError('stdout argument not allowed, it will be overridden.')
-    process = Popen(*popenargs, stdout=PIPE, **kwargs)
-    try:
-        output, unused_err = process.communicate(timeout=timeout)
-    except TimeoutExpired:
-        process.kill()
-        output, unused_err = process.communicate()
-        raise TimeoutExpired(process.args, timeout, output=output)
-    retcode = process.poll()
-    if retcode:
-        raise CalledProcessError(retcode, process.args, output=output)
+    with Popen(*popenargs, stdout=PIPE, **kwargs) as process:
+        try:
+            output, unused_err = process.communicate(timeout=timeout)
+        except TimeoutExpired:
+            process.kill()
+            output, unused_err = process.communicate()
+            raise TimeoutExpired(process.args, timeout, output=output)
+        except:
+            process.kill()
+            process.wait()
+            raise
+        retcode = process.poll()
+        if retcode:
+            raise CalledProcessError(retcode, process.args, output=output)
     return output
 
 
@@ -618,11 +622,19 @@
     >>> subprocess.getstatusoutput('/bin/junk')
     (256, 'sh: /bin/junk: not found')
     """
-    pipe = os.popen('{ ' + cmd + '; } 2>&1', 'r')
-    text = pipe.read()
-    sts = pipe.close()
-    if sts is None: sts = 0
-    if text[-1:] == '\n': text = text[:-1]
+    with os.popen('{ ' + cmd + '; } 2>&1', 'r') as pipe:
+        try:
+            text = pipe.read()
+            sts = pipe.close()
+        except:
+            process = pipe._proc
+            process.kill()
+            process.wait()
+            raise
+    if sts is None:
+        sts = 0
+    if text[-1:] == '\n':
+        text = text[:-1]
     return sts, text
 
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -271,6 +271,10 @@
 Library
 -------
 
+- Issue #12494: On error, call(), check_call(), check_output() and
+  getstatusoutput() functions of the subprocess module now kill the process,
+  read its status (to avoid zombis) and close pipes.
+
 - Issue #12720: Expose low-level Linux extended file attribute functions in os.
 
 - Issue #10946: The distutils commands bdist_dumb, bdist_wininst and bdist_msi

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 00:22:26 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 02 Sep 2011 00:22:26 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Remove_unused_v?=
 =?utf8?q?ariable_if_Python_is_build_without_threads?=
Message-ID: <E1QzFeU-0002LL-PO@dinsdale.python.org>

http://hg.python.org/cpython/rev/cd1978dbac08
changeset:   72169:cd1978dbac08
branch:      3.2
parent:      72165:51e27f42beda
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 02 00:11:43 2011 +0200
summary:
  Remove unused variable if Python is build without threads

files:
  Python/import.c |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -2349,7 +2349,9 @@
 {
     PyObject *result;
     PyObject *modules;
+#ifdef WITH_THREAD
     long me;
+#endif
 
     /* Try to get the module from sys.modules[name] */
     modules = PyImport_GetModuleDict();

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 00:22:27 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 02 Sep 2011 00:22:27 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_3=2E2=3A_Remove_unused_variable_if_Python_is_build_wit?=
 =?utf8?q?hout_threads?=
Message-ID: <E1QzFeV-0002LW-I6@dinsdale.python.org>

http://hg.python.org/cpython/rev/aaab27d4c99c
changeset:   72170:aaab27d4c99c
parent:      72168:86b7f14485c9
parent:      72169:cd1978dbac08
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 02 00:13:16 2011 +0200
summary:
  Merge 3.2: Remove unused variable if Python is build without threads

files:
  Python/import.c |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -2653,7 +2653,9 @@
 PyImport_ImportModuleNoBlock(const char *name)
 {
     PyObject *nameobj, *modules, *result;
+#ifdef WITH_THREAD
     long me;
+#endif
 
     /* Try to get the module from sys.modules[name] */
     modules = PyImport_GetModuleDict();

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 00:22:28 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 02 Sep 2011 00:22:28 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Remove_unused_v?=
 =?utf8?q?ariable_if_Python_is_build_without_threads?=
Message-ID: <E1QzFeW-0002La-CP@dinsdale.python.org>

http://hg.python.org/cpython/rev/85a12278de69
changeset:   72171:85a12278de69
branch:      2.7
parent:      72162:4d584ebbfa77
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 02 00:21:36 2011 +0200
summary:
  Remove unused variable if Python is build without threads

files:
  Python/import.c |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -2064,7 +2064,9 @@
 {
     PyObject *result;
     PyObject *modules;
+#ifdef WITH_THREAD
     long me;
+#endif
 
     /* Try to get the module from sys.modules[name] */
     modules = PyImport_GetModuleDict();

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 01:05:21 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 02 Sep 2011 01:05:21 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_IDLE=3A_fix_som?=
 =?utf8?q?e_RessourceWarning=2C_reuse_tokenize=2Eopen=28=29?=
Message-ID: <E1QzGK1-0006zC-VW@dinsdale.python.org>

http://hg.python.org/cpython/rev/d1d2dfbca812
changeset:   72172:d1d2dfbca812
branch:      3.2
parent:      72169:cd1978dbac08
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 02 00:57:04 2011 +0200
summary:
  IDLE: fix some RessourceWarning, reuse tokenize.open()

files:
  Lib/idlelib/PyShell.py       |  26 ++++++++++--------
  Lib/idlelib/ScriptBinding.py |  33 ++++++++++-------------
  2 files changed, 28 insertions(+), 31 deletions(-)


diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py
--- a/Lib/idlelib/PyShell.py
+++ b/Lib/idlelib/PyShell.py
@@ -201,18 +201,18 @@
         breaks = self.breakpoints
         filename = self.io.filename
         try:
-            lines = open(self.breakpointPath,"r").readlines()
+            with open(self.breakpointPath, "r") as fp:
+                lines = fp.readlines()
         except IOError:
             lines = []
-        new_file = open(self.breakpointPath,"w")
-        for line in lines:
-            if not line.startswith(filename + '='):
-                new_file.write(line)
-        self.update_breakpoints()
-        breaks = self.breakpoints
-        if breaks:
-            new_file.write(filename + '=' + str(breaks) + '\n')
-        new_file.close()
+        with open(self.breakpointPath, "w") as new_file:
+            for line in lines:
+                if not line.startswith(filename + '='):
+                    new_file.write(line)
+            self.update_breakpoints()
+            breaks = self.breakpoints
+            if breaks:
+                new_file.write(filename + '=' + str(breaks) + '\n')
 
     def restore_file_breaks(self):
         self.text.update()   # this enables setting "BREAK" tags to be visible
@@ -220,7 +220,8 @@
         if filename is None:
             return
         if os.path.isfile(self.breakpointPath):
-            lines = open(self.breakpointPath,"r").readlines()
+            with open(self.breakpointPath, "r") as fp:
+                lines = fp.readlines()
             for line in lines:
                 if line.startswith(filename + '='):
                     breakpoint_linenumbers = eval(line[len(filename)+1:])
@@ -571,7 +572,8 @@
     def execfile(self, filename, source=None):
         "Execute an existing file"
         if source is None:
-            source = open(filename, "r").read()
+            with open(filename, "r") as fp:
+                source = fp.read()
         try:
             code = compile(source, filename, "exec")
         except (OverflowError, SyntaxError):
diff --git a/Lib/idlelib/ScriptBinding.py b/Lib/idlelib/ScriptBinding.py
--- a/Lib/idlelib/ScriptBinding.py
+++ b/Lib/idlelib/ScriptBinding.py
@@ -67,25 +67,20 @@
 
     def tabnanny(self, filename):
         # XXX: tabnanny should work on binary files as well
-        with open(filename, 'r', encoding='iso-8859-1') as f:
-            two_lines = f.readline() + f.readline()
-        encoding = IOBinding.coding_spec(two_lines)
-        if not encoding:
-            encoding = 'utf-8'
-        f = open(filename, 'r', encoding=encoding)
-        try:
-            tabnanny.process_tokens(tokenize.generate_tokens(f.readline))
-        except tokenize.TokenError as msg:
-            msgtxt, (lineno, start) = msg
-            self.editwin.gotoline(lineno)
-            self.errorbox("Tabnanny Tokenizing Error",
-                          "Token Error: %s" % msgtxt)
-            return False
-        except tabnanny.NannyNag as nag:
-            # The error messages from tabnanny are too confusing...
-            self.editwin.gotoline(nag.get_lineno())
-            self.errorbox("Tab/space error", indent_message)
-            return False
+        with tokenize.open(filename) as f:
+            try:
+                tabnanny.process_tokens(tokenize.generate_tokens(f.readline))
+            except tokenize.TokenError as msg:
+                msgtxt, (lineno, start) = msg
+                self.editwin.gotoline(lineno)
+                self.errorbox("Tabnanny Tokenizing Error",
+                              "Token Error: %s" % msgtxt)
+                return False
+            except tabnanny.NannyNag as nag:
+                # The error messages from tabnanny are too confusing...
+                self.editwin.gotoline(nag.get_lineno())
+                self.errorbox("Tab/space error", indent_message)
+                return False
         return True
 
     def checksyntax(self, filename):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 01:05:22 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 02 Sep 2011 01:05:22 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyNjM2?=
 =?utf8?q?=3A_IDLE_reads_the_coding_cookie_when_executing_a_Python_script?=
 =?utf8?q?=2E?=
Message-ID: <E1QzGK2-0006zG-Ma@dinsdale.python.org>

http://hg.python.org/cpython/rev/fe7777f1ed14
changeset:   72173:fe7777f1ed14
branch:      3.2
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 02 01:00:40 2011 +0200
summary:
  Issue #12636: IDLE reads the coding cookie when executing a Python script.

files:
  Lib/idlelib/PyShell.py |  11 ++++++-----
  Misc/NEWS              |   2 ++
  2 files changed, 8 insertions(+), 5 deletions(-)


diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py
--- a/Lib/idlelib/PyShell.py
+++ b/Lib/idlelib/PyShell.py
@@ -1,16 +1,17 @@
 #! /usr/bin/env python3
 
+import getopt
 import os
 import os.path
-import sys
-import getopt
 import re
 import socket
+import subprocess
+import sys
+import threading
 import time
-import threading
+import tokenize
 import traceback
 import types
-import subprocess
 
 import linecache
 from code import InteractiveInterpreter
@@ -572,7 +573,7 @@
     def execfile(self, filename, source=None):
         "Execute an existing file"
         if source is None:
-            with open(filename, "r") as fp:
+            with tokenize.open(filename) as fp:
                 source = fp.read()
         try:
             code = compile(source, filename, "exec")
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -28,6 +28,8 @@
 Library
 -------
 
+- Issue #12636: IDLE reads the coding cookie when executing a Python script.
+
 - Issue #10946: The distutils commands bdist_dumb, bdist_wininst and bdist_msi
   now respect a --skip-build option given to bdist.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 01:05:23 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 02 Sep 2011 01:05:23 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_3=2E2=3A_Issue_=2312636=3A_IDLE_reads_the_coding_cooki?=
 =?utf8?q?e_when_executing_a_Python?=
Message-ID: <E1QzGK3-0006zK-Os@dinsdale.python.org>

http://hg.python.org/cpython/rev/a8748022504f
changeset:   72174:a8748022504f
parent:      72170:aaab27d4c99c
parent:      72173:fe7777f1ed14
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 02 01:02:23 2011 +0200
summary:
  Merge 3.2: Issue #12636: IDLE reads the coding cookie when executing a Python script.

And "IDLE: fix some RessourceWarning, reuse tokenize.open()"

files:
  Lib/idlelib/PyShell.py       |  35 +++++++++++++----------
  Lib/idlelib/ScriptBinding.py |  33 +++++++++-------------
  Misc/NEWS                    |   2 +
  3 files changed, 35 insertions(+), 35 deletions(-)


diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py
--- a/Lib/idlelib/PyShell.py
+++ b/Lib/idlelib/PyShell.py
@@ -1,16 +1,17 @@
 #! /usr/bin/env python3
 
+import getopt
 import os
 import os.path
-import sys
-import getopt
 import re
 import socket
+import subprocess
+import sys
+import threading
 import time
-import threading
+import tokenize
 import traceback
 import types
-import subprocess
 
 import linecache
 from code import InteractiveInterpreter
@@ -201,18 +202,18 @@
         breaks = self.breakpoints
         filename = self.io.filename
         try:
-            lines = open(self.breakpointPath,"r").readlines()
+            with open(self.breakpointPath, "r") as fp:
+                lines = fp.readlines()
         except IOError:
             lines = []
-        new_file = open(self.breakpointPath,"w")
-        for line in lines:
-            if not line.startswith(filename + '='):
-                new_file.write(line)
-        self.update_breakpoints()
-        breaks = self.breakpoints
-        if breaks:
-            new_file.write(filename + '=' + str(breaks) + '\n')
-        new_file.close()
+        with open(self.breakpointPath, "w") as new_file:
+            for line in lines:
+                if not line.startswith(filename + '='):
+                    new_file.write(line)
+            self.update_breakpoints()
+            breaks = self.breakpoints
+            if breaks:
+                new_file.write(filename + '=' + str(breaks) + '\n')
 
     def restore_file_breaks(self):
         self.text.update()   # this enables setting "BREAK" tags to be visible
@@ -220,7 +221,8 @@
         if filename is None:
             return
         if os.path.isfile(self.breakpointPath):
-            lines = open(self.breakpointPath,"r").readlines()
+            with open(self.breakpointPath, "r") as fp:
+                lines = fp.readlines()
             for line in lines:
                 if line.startswith(filename + '='):
                     breakpoint_linenumbers = eval(line[len(filename)+1:])
@@ -571,7 +573,8 @@
     def execfile(self, filename, source=None):
         "Execute an existing file"
         if source is None:
-            source = open(filename, "r").read()
+            with tokenize.open(filename) as fp:
+                source = fp.read()
         try:
             code = compile(source, filename, "exec")
         except (OverflowError, SyntaxError):
diff --git a/Lib/idlelib/ScriptBinding.py b/Lib/idlelib/ScriptBinding.py
--- a/Lib/idlelib/ScriptBinding.py
+++ b/Lib/idlelib/ScriptBinding.py
@@ -67,25 +67,20 @@
 
     def tabnanny(self, filename):
         # XXX: tabnanny should work on binary files as well
-        with open(filename, 'r', encoding='iso-8859-1') as f:
-            two_lines = f.readline() + f.readline()
-        encoding = IOBinding.coding_spec(two_lines)
-        if not encoding:
-            encoding = 'utf-8'
-        f = open(filename, 'r', encoding=encoding)
-        try:
-            tabnanny.process_tokens(tokenize.generate_tokens(f.readline))
-        except tokenize.TokenError as msg:
-            msgtxt, (lineno, start) = msg
-            self.editwin.gotoline(lineno)
-            self.errorbox("Tabnanny Tokenizing Error",
-                          "Token Error: %s" % msgtxt)
-            return False
-        except tabnanny.NannyNag as nag:
-            # The error messages from tabnanny are too confusing...
-            self.editwin.gotoline(nag.get_lineno())
-            self.errorbox("Tab/space error", indent_message)
-            return False
+        with tokenize.open(filename) as f:
+            try:
+                tabnanny.process_tokens(tokenize.generate_tokens(f.readline))
+            except tokenize.TokenError as msg:
+                msgtxt, (lineno, start) = msg
+                self.editwin.gotoline(lineno)
+                self.errorbox("Tabnanny Tokenizing Error",
+                              "Token Error: %s" % msgtxt)
+                return False
+            except tabnanny.NannyNag as nag:
+                # The error messages from tabnanny are too confusing...
+                self.editwin.gotoline(nag.get_lineno())
+                self.errorbox("Tab/space error", indent_message)
+                return False
         return True
 
     def checksyntax(self, filename):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -271,6 +271,8 @@
 Library
 -------
 
+- Issue #12636: IDLE reads the coding cookie when executing a Python script.
+
 - Issue #12494: On error, call(), check_call(), check_output() and
   getstatusoutput() functions of the subprocess module now kill the process,
   read its status (to avoid zombis) and close pipes.

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Fri Sep  2 05:22:10 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Fri, 02 Sep 2011 05:22:10 +0200
Subject: [Python-checkins] Daily reference leaks (a8748022504f): sum=0
Message-ID: <E1QzKKY-0002yd-E6@ap.vmr.nerim.net>

results for a8748022504f on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogwn8mpj', '-x']

From python-checkins at python.org  Fri Sep  2 17:45:00 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:00 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Avoid_using_the?=
 =?utf8?q?_default_reST_role=2E__Makes_Doc/tools/rstlint=2Epy_happy=2E?=
Message-ID: <E1QzVvQ-00030C-Ip@dinsdale.python.org>

http://hg.python.org/cpython/rev/d09a44fed132
changeset:   72175:d09a44fed132
branch:      3.2
parent:      72151:1e01543c3d0a
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 02:47:34 2011 +0200
summary:
  Avoid using the default reST role.  Makes Doc/tools/rstlint.py happy.

files:
  Doc/library/base64.rst       |   4 +-
  Doc/library/configparser.rst |   6 ++--
  Doc/library/email.header.rst |   4 +-
  Doc/library/inspect.rst      |   8 +++---
  Doc/library/unittest.rst     |  28 ++++++++++++------------
  5 files changed, 25 insertions(+), 25 deletions(-)


diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst
--- a/Doc/library/base64.rst
+++ b/Doc/library/base64.rst
@@ -45,8 +45,8 @@
    at least length 2 (additional characters are ignored) which specifies the
    alternative alphabet used instead of the ``+`` and ``/`` characters.
 
-   The decoded string is returned.  A `binascii.Error` is raised if *s* is
-   incorrectly padded.
+   The decoded string is returned.  A :exc:`binascii.Error` exception is raised
+   if *s* is incorrectly padded.
 
    If *validate* is ``False`` (the default), non-base64-alphabet characters are
    discarded prior to the padding check.  If *validate* is ``True``,
diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst
--- a/Doc/library/configparser.rst
+++ b/Doc/library/configparser.rst
@@ -806,17 +806,17 @@
    cfg = configparser.ConfigParser()
    cfg.read('example.cfg')
 
-   # Set the optional `raw` argument of get() to True if you wish to disable
+   # Set the optional *raw* argument of get() to True if you wish to disable
    # interpolation in a single get operation.
    print(cfg.get('Section1', 'foo', raw=False)) # -> "Python is fun!"
    print(cfg.get('Section1', 'foo', raw=True))  # -> "%(bar)s is %(baz)s!"
 
-   # The optional `vars` argument is a dict with members that will take
+   # The optional *vars* argument is a dict with members that will take
    # precedence in interpolation.
    print(cfg.get('Section1', 'foo', vars={'bar': 'Documentation',
                                              'baz': 'evil'}))
 
-   # The optional `fallback` argument can be used to provide a fallback value
+   # The optional *fallback* argument can be used to provide a fallback value
    print(cfg.get('Section1', 'foo'))
          # -> "Python is fun!"
 
diff --git a/Doc/library/email.header.rst b/Doc/library/email.header.rst
--- a/Doc/library/email.header.rst
+++ b/Doc/library/email.header.rst
@@ -141,11 +141,11 @@
       Returns an approximation of the :class:`Header` as a string, using an
       unlimited line length.  All pieces are converted to unicode using the
       specified encoding and joined together appropriately.  Any pieces with a
-      charset of `unknown-8bit` are decoded as `ASCII` using the `replace`
+      charset of ``'unknown-8bit'`` are decoded as ASCII using the ``'replace'``
       error handler.
 
       .. versionchanged:: 3.2
-         Added handling for the `unknown-8bit` charset.
+         Added handling for the ``'unknown-8bit'`` charset.
 
 
    .. method:: __eq__(other)
diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst
--- a/Doc/library/inspect.rst
+++ b/Doc/library/inspect.rst
@@ -575,13 +575,13 @@
 may be called.
 
 For cases where you want passive introspection, like documentation tools, this
-can be inconvenient. `getattr_static` has the same signature as :func:`getattr`
+can be inconvenient. :func:`getattr_static` has the same signature as :func:`getattr`
 but avoids executing code when it fetches attributes.
 
 .. function:: getattr_static(obj, attr, default=None)
 
    Retrieve attributes without triggering dynamic lookup via the
-   descriptor protocol, `__getattr__` or `__getattribute__`.
+   descriptor protocol, :meth:`__getattr__` or :meth:`__getattribute__`.
 
    Note: this function may not be able to retrieve all attributes
    that getattr can fetch (like dynamically created attributes)
@@ -589,12 +589,12 @@
    that raise AttributeError). It can also return descriptors objects
    instead of instance members.
 
-   If the instance `__dict__` is shadowed by another member (for example a
+   If the instance :attr:`__dict__` is shadowed by another member (for example a
    property) then this function will be unable to find instance members.
 
    .. versionadded:: 3.2
 
-`getattr_static` does not resolve descriptors, for example slot descriptors or
+:func:`getattr_static` does not resolve descriptors, for example slot descriptors or
 getset descriptors on objects implemented in C. The descriptor object
 is returned instead of the underlying attribute.
 
diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -293,7 +293,7 @@
 
    As a shortcut, ``python -m unittest`` is the equivalent of
    ``python -m unittest discover``. If you want to pass arguments to test
-   discovery the `discover` sub-command must be used explicitly.
+   discovery the ``discover`` sub-command must be used explicitly.
 
 The ``discover`` sub-command has the following options:
 
@@ -305,11 +305,11 @@
 
 .. cmdoption:: -s directory
 
-   Directory to start discovery ('.' default)
+   Directory to start discovery (``.`` default)
 
 .. cmdoption:: -p pattern
 
-   Pattern to match test files ('test*.py' default)
+   Pattern to match test files (``test*.py`` default)
 
 .. cmdoption:: -t directory
 
@@ -724,8 +724,8 @@
    single test.
 
    .. versionchanged::
-      `TestCase` can be instantiated successfully without providing a method
-      name. This makes it easier to experiment with `TestCase` from the
+      :class:`TestCase` can be instantiated successfully without providing a method
+      name. This makes it easier to experiment with :class:`TestCase` from the
       interactive interpreter.
 
    *methodName* defaults to :meth:`runTest`.
@@ -940,17 +940,17 @@
    +---------------------------------------------------------+--------------------------------------+------------+
    | Method                                                  | Checks that                          | New in     |
    +=========================================================+======================================+============+
-   | :meth:`assertRaises(exc, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises `exc`  |            |
+   | :meth:`assertRaises(exc, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises *exc*  |            |
    | <TestCase.assertRaises>`                                |                                      |            |
    +---------------------------------------------------------+--------------------------------------+------------+
-   | :meth:`assertRaisesRegex(exc, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises `exc`  | 3.1        |
-   | <TestCase.assertRaisesRegex>`                           | and the message matches `re`         |            |
+   | :meth:`assertRaisesRegex(exc, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises *exc*  | 3.1        |
+   | <TestCase.assertRaisesRegex>`                           | and the message matches *re*         |            |
    +---------------------------------------------------------+--------------------------------------+------------+
-   | :meth:`assertWarns(warn, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises `warn` | 3.2        |
+   | :meth:`assertWarns(warn, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises *warn* | 3.2        |
    | <TestCase.assertWarns>`                                 |                                      |            |
    +---------------------------------------------------------+--------------------------------------+------------+
-   | :meth:`assertWarnsRegex(warn, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises `warn` | 3.2        |
-   | <TestCase.assertWarnsRegex>`                            | and the message matches `re`         |            |
+   | :meth:`assertWarnsRegex(warn, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises *warn* | 3.2        |
+   | <TestCase.assertWarnsRegex>`                            | and the message matches *re*         |            |
    +---------------------------------------------------------+--------------------------------------+------------+
 
    .. method:: assertRaises(exception, callable, *args, **kwds)
@@ -1092,7 +1092,7 @@
    | :meth:`assertNotRegex(s, re)          | ``not regex.search(s)``        | 3.2          |
    | <TestCase.assertNotRegex>`            |                                |              |
    +---------------------------------------+--------------------------------+--------------+
-   | :meth:`assertCountEqual(a, b)         | `a` and `b` have the same      | 3.2          |
+   | :meth:`assertCountEqual(a, b)         | *a* and *b* have the same      | 3.2          |
    | <TestCase.assertCountEqual>`          | elements in the same number,   |              |
    |                                       | regardless of their order      |              |
    +---------------------------------------+--------------------------------+--------------+
@@ -1887,7 +1887,7 @@
 .. class:: TextTestRunner(stream=None, descriptions=True, verbosity=1, runnerclass=None, warnings=None)
 
    A basic test runner implementation that outputs results to a stream. If *stream*
-   is `None`, the default, `sys.stderr` is used as the output stream. This class
+   is ``None``, the default, :data:`sys.stderr` is used as the output stream. This class
    has a few configurable parameters, but is essentially very simple.  Graphical
    applications which run test suites should provide alternate implementations.
 
@@ -1904,7 +1904,7 @@
       Added the ``warnings`` argument.
 
    .. versionchanged:: 3.2
-      The default stream is set to `sys.stderr` at instantiation time rather
+      The default stream is set to :data:`sys.stderr` at instantiation time rather
       than import time.
 
    .. method:: _makeResult()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:01 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:01 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_some_misuse?=
 =?utf8?q?s_of_Sphinx_roles_and_one_typo?=
Message-ID: <E1QzVvR-00030T-Bf@dinsdale.python.org>

http://hg.python.org/cpython/rev/67a57c50a3e9
changeset:   72176:67a57c50a3e9
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 03:19:30 2011 +0200
summary:
  Fix some misuses of Sphinx roles and one typo

files:
  Doc/c-api/init.rst       |  4 ++--
  Doc/faq/design.rst       |  2 +-
  Doc/faq/windows.rst      |  2 +-
  Doc/howto/logging.rst    |  6 +++---
  Doc/library/argparse.rst |  2 +-
  5 files changed, 8 insertions(+), 8 deletions(-)


diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -122,7 +122,7 @@
    program name is ``'/usr/local/bin/python'``, the prefix is ``'/usr/local'``. The
    returned string points into static storage; the caller should not modify its
    value.  This corresponds to the :makevar:`prefix` variable in the top-level
-   :file:`Makefile` and the :option:`--prefix` argument to the :program:`configure`
+   :file:`Makefile` and the ``--prefix`` argument to the :program:`configure`
    script at build time.  The value is available to Python code as ``sys.prefix``.
    It is only useful on Unix.  See also the next function.
 
@@ -135,7 +135,7 @@
    program name is ``'/usr/local/bin/python'``, the exec-prefix is
    ``'/usr/local'``.  The returned string points into static storage; the caller
    should not modify its value.  This corresponds to the :makevar:`exec_prefix`
-   variable in the top-level :file:`Makefile` and the :option:`--exec-prefix`
+   variable in the top-level :file:`Makefile` and the ``--exec-prefix``
    argument to the :program:`configure` script at build  time.  The value is
    available to Python code as ``sys.exec_prefix``.  It is only useful on Unix.
 
diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst
--- a/Doc/faq/design.rst
+++ b/Doc/faq/design.rst
@@ -667,7 +667,7 @@
 Python 2.6 adds an :mod:`abc` module that lets you define Abstract Base Classes
 (ABCs).  You can then use :func:`isinstance` and :func:`issubclass` to check
 whether an instance or a class implements a particular ABC.  The
-:mod:`collections` modules defines a set of useful ABCs such as
+:mod:`collections` module defines a set of useful ABCs such as
 :class:`Iterable`, :class:`Container`, and :class:`MutableMapping`.
 
 For Python, many of the advantages of interface specifications can be obtained
diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst
--- a/Doc/faq/windows.rst
+++ b/Doc/faq/windows.rst
@@ -546,7 +546,7 @@
 :func:`execfile` with the name of your file as argument.
 
 Also note that you can not mix-and-match Debug and Release versions.  If you
-wish to use the Debug Multithreaded DLL, then your module *must* have an "_d"
+wish to use the Debug Multithreaded DLL, then your module *must* have ``_d``
 appended to the base name.
 
 
diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst
--- a/Doc/howto/logging.rst
+++ b/Doc/howto/logging.rst
@@ -412,10 +412,10 @@
   :meth:`Logger.error`, and :meth:`Logger.critical` all create log records with
   a message and a level that corresponds to their respective method names. The
   message is actually a format string, which may contain the standard string
-  substitution syntax of :const:`%s`, :const:`%d`, :const:`%f`, and so on.  The
+  substitution syntax of ``%s``, ``%d``, ``%f``, and so on.  The
   rest of their arguments is a list of objects that correspond with the
-  substitution fields in the message.  With regard to :const:`**kwargs`, the
-  logging methods care only about a keyword of :const:`exc_info` and use it to
+  substitution fields in the message.  With regard to ``**kwargs``, the
+  logging methods care only about a keyword of ``exc_info`` and use it to
   determine whether to log exception information.
 
 * :meth:`Logger.exception` creates a log message similar to
diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst
--- a/Doc/library/argparse.rst
+++ b/Doc/library/argparse.rst
@@ -155,7 +155,7 @@
      conflicting optionals.
 
    * prog_ - The name of the program (default:
-     :data:`sys.argv[0]`)
+     ``sys.argv[0]``)
 
    * usage_ - The string describing the program usage (default: generated)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:02 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:02 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Adapt/remove_me?=
 =?utf8?q?ntions_of_functions_gone_in_3=2Ex?=
Message-ID: <E1QzVvS-00032B-4I@dinsdale.python.org>

http://hg.python.org/cpython/rev/fcb037cf261b
changeset:   72177:fcb037cf261b
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 03:20:13 2011 +0200
summary:
  Adapt/remove mentions of functions gone in 3.x

files:
  Doc/faq/programming.rst |  9 ---------
  Doc/faq/windows.rst     |  2 +-
  Doc/glossary.rst        |  2 +-
  3 files changed, 2 insertions(+), 11 deletions(-)


diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst
--- a/Doc/faq/programming.rst
+++ b/Doc/faq/programming.rst
@@ -473,15 +473,6 @@
        ...
        g(x, *args, **kwargs)
 
-In the unlikely case that you care about Python versions older than 2.0, use
-:func:`apply`::
-
-   def f(x, *args, **kwargs):
-       ...
-       kwargs['width'] = '14.3c'
-       ...
-       apply(g, (x,)+args, kwargs)
-
 
 How do I write a function with output parameters (call by reference)?
 ---------------------------------------------------------------------
diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst
--- a/Doc/faq/windows.rst
+++ b/Doc/faq/windows.rst
@@ -543,7 +543,7 @@
 
 If you can't change compilers or flags, try using :c:func:`Py_RunSimpleString`.
 A trick to get it to run an arbitrary file is to construct a call to
-:func:`execfile` with the name of your file as argument.
+:func:`exec` and :func:`open` with the name of your file as argument.
 
 Also note that you can not mix-and-match Debug and Release versions.  If you
 wish to use the Debug Multithreaded DLL, then your module *must* have ``_d``
diff --git a/Doc/glossary.rst b/Doc/glossary.rst
--- a/Doc/glossary.rst
+++ b/Doc/glossary.rst
@@ -492,7 +492,7 @@
       :func:`builtins.open` and :func:`os.open` are distinguished by their
       namespaces.  Namespaces also aid readability and maintainability by making
       it clear which module implements a function.  For instance, writing
-      :func:`random.seed` or :func:`itertools.izip` makes it clear that those
+      :func:`random.seed` or :func:`itertools.islice` makes it clear that those
       functions are implemented by the :mod:`random` and :mod:`itertools`
       modules, respectively.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:02 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:02 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Clean_up_packaging=2Eutil?=
 =?utf8?q?=3A_add_=5F=5Fall=5F=5F=2C_remove_some_unused_functions=2E?=
Message-ID: <E1QzVvS-00033C-Vk@dinsdale.python.org>

http://hg.python.org/cpython/rev/28e4cd8fd864
changeset:   72178:28e4cd8fd864
parent:      72152:0968acf0e6db
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 05:11:29 2011 +0200
summary:
  Clean up packaging.util: add __all__, remove some unused functions.

This huge module is the heir of six distutils modules, and contains
a number of miscellaneous functions.  I have attempted to help readers
of the source code with an annoted __all__.  Removed or deprecated
functions have been removed from the documentation; I?m working on
another patch to document the remaining public functions.

For the curious:

The unzip_file and untar_file were used by (or intended to be used by)
?pysetup install path/to/archive.tar.gz?, but the code presently used
shutil.unpack_archive and an helper function, so I just deleted them.
They?re still in the repository if we need them in the future.

The find_packages function is not used anymore but I want to discuss
module and package auto-discovery (in ?pysetup create?) again before
removing it.

subst_vars now lives in sysconfig; rfc822_escape is inlined in
packaging.metadata.  Other functions are for internal use only, or
deprecated; I have left them out of __all__ and sprinkled TODO notes
for future cleanups.

files:
  Doc/library/packaging.util.rst       |   37 ---
  Lib/packaging/command/build_py.py    |    2 +-
  Lib/packaging/command/install_lib.py |    2 +-
  Lib/packaging/command/register.py    |    1 -
  Lib/packaging/command/upload.py      |    1 -
  Lib/packaging/config.py              |    2 +-
  Lib/packaging/create.py              |    6 +-
  Lib/packaging/tests/test_util.py     |    9 +-
  Lib/packaging/util.py                |  163 +++-----------
  9 files changed, 42 insertions(+), 181 deletions(-)


diff --git a/Doc/library/packaging.util.rst b/Doc/library/packaging.util.rst
--- a/Doc/library/packaging.util.rst
+++ b/Doc/library/packaging.util.rst
@@ -90,34 +90,6 @@
    Search the path for a given executable name.
 
 
-.. function:: subst_vars(s, local_vars)
-
-   Perform shell/Perl-style variable substitution on *s*.  Every occurrence of
-   ``$`` followed by a name is considered a variable, and variable is
-   substituted by the value found in the *local_vars* dictionary, or in
-   ``os.environ`` if it's not in *local_vars*. *os.environ* is first
-   checked/augmented to guarantee that it contains certain values: see
-   :func:`check_environ`.  Raise :exc:`ValueError` for any variables not found
-   in either *local_vars* or ``os.environ``.
-
-   Note that this is not a fully-fledged string interpolation function. A valid
-   ``$variable`` can consist only of upper and lower case letters, numbers and
-   an underscore. No { } or ( ) style quoting is available.
-
-
-.. function:: split_quoted(s)
-
-   Split a string up according to Unix shell-like rules for quotes and
-   backslashes. In short: words are delimited by spaces, as long as those spaces
-   are not escaped by a backslash, or inside a quoted string. Single and double
-   quotes are equivalent, and the quote characters can be backslash-escaped.
-   The backslash is stripped from any two-character escape sequence, leaving
-   only the escaped character.  The quote characters are stripped from any
-   quoted string.  Returns a list of words.
-
-   .. TODO Should probably be moved into the standard library.
-
-
 .. function:: execute(func, args[, msg=None, verbose=0, dry_run=0])
 
    Perform some action that affects the outside world (for instance, writing to
@@ -175,12 +147,3 @@
    figure out to use direct compilation or not (see the source for details).
    The *direct* flag is used by the script generated in indirect mode; unless
    you know what you're doing, leave it set to ``None``.
-
-
-.. function:: rfc822_escape(header)
-
-   Return a version of *header* escaped for inclusion in an :rfc:`822` header, by
-   ensuring there are 8 spaces space after each newline.  Note that it does no
-   other modification of the string.
-
-   .. TODO this _can_ be replaced
diff --git a/Lib/packaging/command/build_py.py b/Lib/packaging/command/build_py.py
--- a/Lib/packaging/command/build_py.py
+++ b/Lib/packaging/command/build_py.py
@@ -393,7 +393,7 @@
                            self.get_command_name())
             return
 
-        from packaging.util import byte_compile
+        from packaging.util import byte_compile  # FIXME use compileall
         prefix = self.build_lib
         if prefix[-1] != os.sep:
             prefix = prefix + os.sep
diff --git a/Lib/packaging/command/install_lib.py b/Lib/packaging/command/install_lib.py
--- a/Lib/packaging/command/install_lib.py
+++ b/Lib/packaging/command/install_lib.py
@@ -122,7 +122,7 @@
                            self.get_command_name())
             return
 
-        from packaging.util import byte_compile
+        from packaging.util import byte_compile  # FIXME use compileall
 
         # Get the "--root" directory supplied to the "install_dist" command,
         # and use it as a prefix to strip off the purported filename
diff --git a/Lib/packaging/command/register.py b/Lib/packaging/command/register.py
--- a/Lib/packaging/command/register.py
+++ b/Lib/packaging/command/register.py
@@ -2,7 +2,6 @@
 
 # Contributed by Richard Jones
 
-import io
 import getpass
 import urllib.error
 import urllib.parse
diff --git a/Lib/packaging/command/upload.py b/Lib/packaging/command/upload.py
--- a/Lib/packaging/command/upload.py
+++ b/Lib/packaging/command/upload.py
@@ -5,7 +5,6 @@
 import logging
 import platform
 import urllib.parse
-from io import BytesIO
 from base64 import standard_b64encode
 from hashlib import md5
 from urllib.error import HTTPError
diff --git a/Lib/packaging/config.py b/Lib/packaging/config.py
--- a/Lib/packaging/config.py
+++ b/Lib/packaging/config.py
@@ -216,7 +216,7 @@
             for data in files.get('package_data', []):
                 data = data.split('=')
                 if len(data) != 2:
-                    continue  # XXX error should never pass silently
+                    continue  # FIXME errors should never pass silently
                 key, value = data
                 self.dist.package_data[key.strip()] = value.strip()
 
diff --git a/Lib/packaging/create.py b/Lib/packaging/create.py
--- a/Lib/packaging/create.py
+++ b/Lib/packaging/create.py
@@ -36,7 +36,7 @@
 from packaging.version import is_valid_version
 
 _FILENAME = 'setup.cfg'
-_DEFAULT_CFG = '.pypkgcreate'
+_DEFAULT_CFG = '.pypkgcreate'  # FIXME use a section in user .pydistutils.cfg
 
 _helptext = {
     'name': '''
@@ -127,6 +127,10 @@
         print('\nERROR: You must select "Y" or "N".\n')
 
 
+# XXX use util.ask
+# FIXME: if prompt ends with '?', don't add ':'
+
+
 def ask(question, default=None, helptext=None, required=True,
         lengthy=False, multiline=False):
     prompt = '%s: ' % (question,)
diff --git a/Lib/packaging/tests/test_util.py b/Lib/packaging/tests/test_util.py
--- a/Lib/packaging/tests/test_util.py
+++ b/Lib/packaging/tests/test_util.py
@@ -15,7 +15,7 @@
 from packaging import util
 from packaging.dist import Distribution
 from packaging.util import (
-    convert_path, change_root, split_quoted, strtobool, rfc822_escape,
+    convert_path, change_root, split_quoted, strtobool,
     get_compiler_versions, _MAC_OS_X_LD_VERSION, byte_compile, find_packages,
     spawn, get_pypirc_path, generate_pypirc, read_pypirc, resolve_name, iglob,
     RICH_GLOB, egginfo_to_distinfo, is_setuptools, is_distutils, is_packaging,
@@ -255,13 +255,6 @@
         for n in no:
             self.assertFalse(strtobool(n))
 
-    def test_rfc822_escape(self):
-        header = 'I am a\npoor\nlonesome\nheader\n'
-        res = rfc822_escape(header)
-        wanted = ('I am a%(8s)spoor%(8s)slonesome%(8s)s'
-                  'header%(8s)s') % {'8s': '\n' + 8 * ' '}
-        self.assertEqual(res, wanted)
-
     def test_find_exe_version(self):
         # the ld version scheme under MAC OS is:
         #   ^@(#)PROGRAM:ld  PROJECT:ld64-VERSION
diff --git a/Lib/packaging/util.py b/Lib/packaging/util.py
--- a/Lib/packaging/util.py
+++ b/Lib/packaging/util.py
@@ -8,8 +8,6 @@
 import shutil
 import string
 import hashlib
-import tarfile
-import zipfile
 import posixpath
 import subprocess
 import sysconfig
@@ -23,6 +21,30 @@
                               PackagingByteCompileError, PackagingExecError,
                               InstallationException, PackagingInternalError)
 
+__all__ = [
+    # file dependencies
+    'newer', 'newer_group',
+    # helpers for commands (dry-run system)
+    'execute', 'write_file',
+    # spawning programs
+    'find_executable', 'spawn',
+    # path manipulation
+    'convert_path', 'change_root',
+    # 2to3 conversion
+    'Mixin2to3', 'run_2to3',
+    # packaging compatibility helpers
+    'cfg_to_args', 'generate_setup_py',
+    'egginfo_to_distinfo',
+    'get_install_method',
+    # misc
+    'ask', 'check_environ', 'encode_multipart', 'resolve_name',
+    # querying for information  TODO move to sysconfig
+    'get_compiler_versions', 'get_platform', 'set_platform',
+    # configuration  TODO move to packaging.config
+    'get_pypirc_path', 'read_pypirc', 'generate_pypirc',
+    'strtobool', 'split_multiline',
+]
+
 _PLATFORM = None
 _DEFAULT_INSTALLER = 'packaging'
 
@@ -152,31 +174,6 @@
     _environ_checked = True
 
 
-def subst_vars(s, local_vars):
-    """Perform shell/Perl-style variable substitution on 'string'.
-
-    Every occurrence of '$' followed by a name is considered a variable, and
-    variable is substituted by the value found in the 'local_vars'
-    dictionary, or in 'os.environ' if it's not in 'local_vars'.
-    'os.environ' is first checked/augmented to guarantee that it contains
-    certain values: see 'check_environ()'.  Raise ValueError for any
-    variables not found in either 'local_vars' or 'os.environ'.
-    """
-    check_environ()
-
-    def _subst(match, local_vars=local_vars):
-        var_name = match.group(1)
-        if var_name in local_vars:
-            return str(local_vars[var_name])
-        else:
-            return os.environ[var_name]
-
-    try:
-        return re.sub(r'\$([a-zA-Z_][a-zA-Z_0-9]*)', _subst, s)
-    except KeyError as var:
-        raise ValueError("invalid variable '$%s'" % var)
-
-
 # Needed by 'split_quoted()'
 _wordchars_re = _squote_re = _dquote_re = None
 
@@ -188,6 +185,8 @@
     _dquote_re = re.compile(r'"(?:[^"\\]|\\.)*"')
 
 
+# TODO replace with shlex.split after testing
+
 def split_quoted(s):
     """Split a string up according to Unix shell-like rules for quotes and
     backslashes.
@@ -435,15 +434,6 @@
                               file, cfile_base)
 
 
-def rfc822_escape(header):
-    """Return a form of *header* suitable for inclusion in an RFC 822-header.
-
-    This function ensures there are 8 spaces after each newline.
-    """
-    lines = header.split('\n')
-    sep = '\n' + 8 * ' '
-    return sep.join(lines)
-
 _RE_VERSION = re.compile('(\d+\.\d+(\.\d+)*)')
 _MAC_OS_X_LD_VERSION = re.compile('^@\(#\)PROGRAM:ld  '
                                   'PROJECT:ld64-((\d+)(\.\d+)*)')
@@ -543,6 +533,10 @@
     """Create *filename* and write *contents* to it.
 
     *contents* is a sequence of strings without line terminators.
+
+    This functions is not intended to replace the usual with open + write
+    idiom in all cases, only with Command.execute, which runs depending on
+    the dry_run argument and also logs its arguments).
     """
     with open(filename, "w") as f:
         for line in contents:
@@ -562,6 +556,7 @@
 
 
 def _under(path, root):
+    # XXX use os.path
     path = path.split(os.sep)
     root = root.split(os.sep)
     if len(root) > len(path):
@@ -664,103 +659,11 @@
     return base, ext
 
 
-def unzip_file(filename, location, flatten=True):
-    """Unzip the file *filename* into the *location* directory."""
-    if not os.path.exists(location):
-        os.makedirs(location)
-    with open(filename, 'rb') as zipfp:
-        zip = zipfile.ZipFile(zipfp)
-        leading = has_leading_dir(zip.namelist()) and flatten
-        for name in zip.namelist():
-            data = zip.read(name)
-            fn = name
-            if leading:
-                fn = split_leading_dir(name)[1]
-            fn = os.path.join(location, fn)
-            dir = os.path.dirname(fn)
-            if not os.path.exists(dir):
-                os.makedirs(dir)
-            if fn.endswith('/') or fn.endswith('\\'):
-                # A directory
-                if not os.path.exists(fn):
-                    os.makedirs(fn)
-            else:
-                with open(fn, 'wb') as fp:
-                    fp.write(data)
-
-
-def untar_file(filename, location):
-    """Untar the file *filename* into the *location* directory."""
-    if not os.path.exists(location):
-        os.makedirs(location)
-    if filename.lower().endswith('.gz') or filename.lower().endswith('.tgz'):
-        mode = 'r:gz'
-    elif (filename.lower().endswith('.bz2')
-          or filename.lower().endswith('.tbz')):
-        mode = 'r:bz2'
-    elif filename.lower().endswith('.tar'):
-        mode = 'r'
-    else:
-        mode = 'r:*'
-    with tarfile.open(filename, mode) as tar:
-        leading = has_leading_dir(member.name for member in tar.getmembers())
-        for member in tar.getmembers():
-            fn = member.name
-            if leading:
-                fn = split_leading_dir(fn)[1]
-            path = os.path.join(location, fn)
-            if member.isdir():
-                if not os.path.exists(path):
-                    os.makedirs(path)
-            else:
-                try:
-                    fp = tar.extractfile(member)
-                except (KeyError, AttributeError):
-                    # Some corrupt tar files seem to produce this
-                    # (specifically bad symlinks)
-                    continue
-                try:
-                    if not os.path.exists(os.path.dirname(path)):
-                        os.makedirs(os.path.dirname(path))
-                        with open(path, 'wb') as destfp:
-                            shutil.copyfileobj(fp, destfp)
-                finally:
-                    fp.close()
-
-
-def has_leading_dir(paths):
-    """Return true if all the paths have the same leading path name.
-
-    In other words, check that everything is in one subdirectory in an
-    archive.
-    """
-    common_prefix = None
-    for path in paths:
-        prefix, rest = split_leading_dir(path)
-        if not prefix:
-            return False
-        elif common_prefix is None:
-            common_prefix = prefix
-        elif prefix != common_prefix:
-            return False
-    return True
-
-
-def split_leading_dir(path):
-    path = str(path)
-    path = path.lstrip('/').lstrip('\\')
-    if '/' in path and (('\\' in path and path.find('/') < path.find('\\'))
-                        or '\\' not in path):
-        return path.split('/', 1)
-    elif '\\' in path:
-        return path.split('\\', 1)
-    else:
-        return path, ''
-
 if sys.platform == 'darwin':
     _cfg_target = None
     _cfg_target_split = None
 
+
 def spawn(cmd, search_path=True, verbose=0, dry_run=False, env=None):
     """Run another program specified as a command list 'cmd' in a new process.
 
@@ -1510,7 +1413,7 @@
     for key, values in fields:
         # handle multiple entries for the same name
         if not isinstance(values, (tuple, list)):
-            values=[values]
+            values = [values]
 
         for value in values:
             l.extend((

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:03 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:03 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_doc_changes_from_3=2E2=2E?=
Message-ID: <E1QzVvT-00034F-OT@dinsdale.python.org>

http://hg.python.org/cpython/rev/89d1a3719234
changeset:   72179:89d1a3719234
parent:      72178:28e4cd8fd864
parent:      72177:fcb037cf261b
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 05:57:12 2011 +0200
summary:
  Merge doc changes from 3.2.

rstlint complains about packaging docs but I?m working on those in
another patch.

files:
  Doc/c-api/init.rst           |   4 +-
  Doc/faq/design.rst           |   2 +-
  Doc/faq/programming.rst      |   9 -------
  Doc/faq/windows.rst          |   4 +-
  Doc/glossary.rst             |   6 ++--
  Doc/howto/logging.rst        |   6 ++--
  Doc/library/argparse.rst     |   2 +-
  Doc/library/base64.rst       |   4 +-
  Doc/library/configparser.rst |   6 ++--
  Doc/library/email.header.rst |   4 +-
  Doc/library/inspect.rst      |   8 +++---
  Doc/library/unittest.rst     |  28 ++++++++++++------------
  12 files changed, 37 insertions(+), 46 deletions(-)


diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -122,7 +122,7 @@
    program name is ``'/usr/local/bin/python'``, the prefix is ``'/usr/local'``. The
    returned string points into static storage; the caller should not modify its
    value.  This corresponds to the :makevar:`prefix` variable in the top-level
-   :file:`Makefile` and the :option:`--prefix` argument to the :program:`configure`
+   :file:`Makefile` and the ``--prefix`` argument to the :program:`configure`
    script at build time.  The value is available to Python code as ``sys.prefix``.
    It is only useful on Unix.  See also the next function.
 
@@ -135,7 +135,7 @@
    program name is ``'/usr/local/bin/python'``, the exec-prefix is
    ``'/usr/local'``.  The returned string points into static storage; the caller
    should not modify its value.  This corresponds to the :makevar:`exec_prefix`
-   variable in the top-level :file:`Makefile` and the :option:`--exec-prefix`
+   variable in the top-level :file:`Makefile` and the ``--exec-prefix``
    argument to the :program:`configure` script at build  time.  The value is
    available to Python code as ``sys.exec_prefix``.  It is only useful on Unix.
 
diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst
--- a/Doc/faq/design.rst
+++ b/Doc/faq/design.rst
@@ -667,7 +667,7 @@
 Python 2.6 adds an :mod:`abc` module that lets you define Abstract Base Classes
 (ABCs).  You can then use :func:`isinstance` and :func:`issubclass` to check
 whether an instance or a class implements a particular ABC.  The
-:mod:`collections` modules defines a set of useful ABCs such as
+:mod:`collections.abc` module defines a set of useful ABCs such as
 :class:`Iterable`, :class:`Container`, and :class:`MutableMapping`.
 
 For Python, many of the advantages of interface specifications can be obtained
diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst
--- a/Doc/faq/programming.rst
+++ b/Doc/faq/programming.rst
@@ -473,15 +473,6 @@
        ...
        g(x, *args, **kwargs)
 
-In the unlikely case that you care about Python versions older than 2.0, use
-:func:`apply`::
-
-   def f(x, *args, **kwargs):
-       ...
-       kwargs['width'] = '14.3c'
-       ...
-       apply(g, (x,)+args, kwargs)
-
 
 How do I write a function with output parameters (call by reference)?
 ---------------------------------------------------------------------
diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst
--- a/Doc/faq/windows.rst
+++ b/Doc/faq/windows.rst
@@ -543,10 +543,10 @@
 
 If you can't change compilers or flags, try using :c:func:`Py_RunSimpleString`.
 A trick to get it to run an arbitrary file is to construct a call to
-:func:`execfile` with the name of your file as argument.
+:func:`exec` and :func:`open` with the name of your file as argument.
 
 Also note that you can not mix-and-match Debug and Release versions.  If you
-wish to use the Debug Multithreaded DLL, then your module *must* have an "_d"
+wish to use the Debug Multithreaded DLL, then your module *must* have ``_d``
 appended to the base name.
 
 
diff --git a/Doc/glossary.rst b/Doc/glossary.rst
--- a/Doc/glossary.rst
+++ b/Doc/glossary.rst
@@ -434,8 +434,8 @@
 
    mapping
       A container object that supports arbitrary key lookups and implements the
-      methods specified in the :class:`~collections.Mapping` or
-      :class:`~collections.MutableMapping`
+      methods specified in the :class:`~collections.abc.Mapping` or
+      :class:`~collections.abc.MutableMapping`
       :ref:`abstract base classes <collections-abstract-base-classes>`.  Examples
       include :class:`dict`, :class:`collections.defaultdict`,
       :class:`collections.OrderedDict` and :class:`collections.Counter`.
@@ -492,7 +492,7 @@
       :func:`builtins.open` and :func:`os.open` are distinguished by their
       namespaces.  Namespaces also aid readability and maintainability by making
       it clear which module implements a function.  For instance, writing
-      :func:`random.seed` or :func:`itertools.izip` makes it clear that those
+      :func:`random.seed` or :func:`itertools.islice` makes it clear that those
       functions are implemented by the :mod:`random` and :mod:`itertools`
       modules, respectively.
 
diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst
--- a/Doc/howto/logging.rst
+++ b/Doc/howto/logging.rst
@@ -412,10 +412,10 @@
   :meth:`Logger.error`, and :meth:`Logger.critical` all create log records with
   a message and a level that corresponds to their respective method names. The
   message is actually a format string, which may contain the standard string
-  substitution syntax of :const:`%s`, :const:`%d`, :const:`%f`, and so on.  The
+  substitution syntax of ``%s``, ``%d``, ``%f``, and so on.  The
   rest of their arguments is a list of objects that correspond with the
-  substitution fields in the message.  With regard to :const:`**kwargs`, the
-  logging methods care only about a keyword of :const:`exc_info` and use it to
+  substitution fields in the message.  With regard to ``**kwargs``, the
+  logging methods care only about a keyword of ``exc_info`` and use it to
   determine whether to log exception information.
 
 * :meth:`Logger.exception` creates a log message similar to
diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst
--- a/Doc/library/argparse.rst
+++ b/Doc/library/argparse.rst
@@ -155,7 +155,7 @@
      conflicting optionals.
 
    * prog_ - The name of the program (default:
-     :data:`sys.argv[0]`)
+     ``sys.argv[0]``)
 
    * usage_ - The string describing the program usage (default: generated)
 
diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst
--- a/Doc/library/base64.rst
+++ b/Doc/library/base64.rst
@@ -45,8 +45,8 @@
    at least length 2 (additional characters are ignored) which specifies the
    alternative alphabet used instead of the ``+`` and ``/`` characters.
 
-   The decoded string is returned.  A `binascii.Error` is raised if *s* is
-   incorrectly padded.
+   The decoded string is returned.  A :exc:`binascii.Error` exception is raised
+   if *s* is incorrectly padded.
 
    If *validate* is ``False`` (the default), non-base64-alphabet characters are
    discarded prior to the padding check.  If *validate* is ``True``,
diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst
--- a/Doc/library/configparser.rst
+++ b/Doc/library/configparser.rst
@@ -806,17 +806,17 @@
    cfg = configparser.ConfigParser()
    cfg.read('example.cfg')
 
-   # Set the optional `raw` argument of get() to True if you wish to disable
+   # Set the optional *raw* argument of get() to True if you wish to disable
    # interpolation in a single get operation.
    print(cfg.get('Section1', 'foo', raw=False)) # -> "Python is fun!"
    print(cfg.get('Section1', 'foo', raw=True))  # -> "%(bar)s is %(baz)s!"
 
-   # The optional `vars` argument is a dict with members that will take
+   # The optional *vars* argument is a dict with members that will take
    # precedence in interpolation.
    print(cfg.get('Section1', 'foo', vars={'bar': 'Documentation',
                                              'baz': 'evil'}))
 
-   # The optional `fallback` argument can be used to provide a fallback value
+   # The optional *fallback* argument can be used to provide a fallback value
    print(cfg.get('Section1', 'foo'))
          # -> "Python is fun!"
 
diff --git a/Doc/library/email.header.rst b/Doc/library/email.header.rst
--- a/Doc/library/email.header.rst
+++ b/Doc/library/email.header.rst
@@ -141,11 +141,11 @@
       Returns an approximation of the :class:`Header` as a string, using an
       unlimited line length.  All pieces are converted to unicode using the
       specified encoding and joined together appropriately.  Any pieces with a
-      charset of `unknown-8bit` are decoded as `ASCII` using the `replace`
+      charset of ``'unknown-8bit'`` are decoded as ASCII using the ``'replace'``
       error handler.
 
       .. versionchanged:: 3.2
-         Added handling for the `unknown-8bit` charset.
+         Added handling for the ``'unknown-8bit'`` charset.
 
 
    .. method:: __eq__(other)
diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst
--- a/Doc/library/inspect.rst
+++ b/Doc/library/inspect.rst
@@ -575,13 +575,13 @@
 may be called.
 
 For cases where you want passive introspection, like documentation tools, this
-can be inconvenient. `getattr_static` has the same signature as :func:`getattr`
+can be inconvenient. :func:`getattr_static` has the same signature as :func:`getattr`
 but avoids executing code when it fetches attributes.
 
 .. function:: getattr_static(obj, attr, default=None)
 
    Retrieve attributes without triggering dynamic lookup via the
-   descriptor protocol, `__getattr__` or `__getattribute__`.
+   descriptor protocol, :meth:`__getattr__` or :meth:`__getattribute__`.
 
    Note: this function may not be able to retrieve all attributes
    that getattr can fetch (like dynamically created attributes)
@@ -589,12 +589,12 @@
    that raise AttributeError). It can also return descriptors objects
    instead of instance members.
 
-   If the instance `__dict__` is shadowed by another member (for example a
+   If the instance :attr:`__dict__` is shadowed by another member (for example a
    property) then this function will be unable to find instance members.
 
    .. versionadded:: 3.2
 
-`getattr_static` does not resolve descriptors, for example slot descriptors or
+:func:`getattr_static` does not resolve descriptors, for example slot descriptors or
 getset descriptors on objects implemented in C. The descriptor object
 is returned instead of the underlying attribute.
 
diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -293,7 +293,7 @@
 
    As a shortcut, ``python -m unittest`` is the equivalent of
    ``python -m unittest discover``. If you want to pass arguments to test
-   discovery the `discover` sub-command must be used explicitly.
+   discovery the ``discover`` sub-command must be used explicitly.
 
 The ``discover`` sub-command has the following options:
 
@@ -305,11 +305,11 @@
 
 .. cmdoption:: -s directory
 
-   Directory to start discovery ('.' default)
+   Directory to start discovery (``.`` default)
 
 .. cmdoption:: -p pattern
 
-   Pattern to match test files ('test*.py' default)
+   Pattern to match test files (``test*.py`` default)
 
 .. cmdoption:: -t directory
 
@@ -724,8 +724,8 @@
    single test.
 
    .. versionchanged:: 3.2
-      `TestCase` can be instantiated successfully without providing a method
-      name. This makes it easier to experiment with `TestCase` from the
+      :class:`TestCase` can be instantiated successfully without providing a method
+      name. This makes it easier to experiment with :class:`TestCase` from the
       interactive interpreter.
 
    *methodName* defaults to :meth:`runTest`.
@@ -944,17 +944,17 @@
    +---------------------------------------------------------+--------------------------------------+------------+
    | Method                                                  | Checks that                          | New in     |
    +=========================================================+======================================+============+
-   | :meth:`assertRaises(exc, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises `exc`  |            |
+   | :meth:`assertRaises(exc, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises *exc*  |            |
    | <TestCase.assertRaises>`                                |                                      |            |
    +---------------------------------------------------------+--------------------------------------+------------+
-   | :meth:`assertRaisesRegex(exc, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises `exc`  | 3.1        |
-   | <TestCase.assertRaisesRegex>`                           | and the message matches `re`         |            |
+   | :meth:`assertRaisesRegex(exc, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises *exc*  | 3.1        |
+   | <TestCase.assertRaisesRegex>`                           | and the message matches *re*         |            |
    +---------------------------------------------------------+--------------------------------------+------------+
-   | :meth:`assertWarns(warn, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises `warn` | 3.2        |
+   | :meth:`assertWarns(warn, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises *warn* | 3.2        |
    | <TestCase.assertWarns>`                                 |                                      |            |
    +---------------------------------------------------------+--------------------------------------+------------+
-   | :meth:`assertWarnsRegex(warn, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises `warn` | 3.2        |
-   | <TestCase.assertWarnsRegex>`                            | and the message matches `re`         |            |
+   | :meth:`assertWarnsRegex(warn, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises *warn* | 3.2        |
+   | <TestCase.assertWarnsRegex>`                            | and the message matches *re*         |            |
    +---------------------------------------------------------+--------------------------------------+------------+
 
    .. method:: assertRaises(exception, callable, *args, **kwds)
@@ -1116,7 +1116,7 @@
    | :meth:`assertNotRegex(s, re)          | ``not regex.search(s)``        | 3.2          |
    | <TestCase.assertNotRegex>`            |                                |              |
    +---------------------------------------+--------------------------------+--------------+
-   | :meth:`assertCountEqual(a, b)         | `a` and `b` have the same      | 3.2          |
+   | :meth:`assertCountEqual(a, b)         | *a* and *b* have the same      | 3.2          |
    | <TestCase.assertCountEqual>`          | elements in the same number,   |              |
    |                                       | regardless of their order      |              |
    +---------------------------------------+--------------------------------+--------------+
@@ -1911,7 +1911,7 @@
 .. class:: TextTestRunner(stream=None, descriptions=True, verbosity=1, runnerclass=None, warnings=None)
 
    A basic test runner implementation that outputs results to a stream. If *stream*
-   is `None`, the default, `sys.stderr` is used as the output stream. This class
+   is ``None``, the default, :data:`sys.stderr` is used as the output stream. This class
    has a few configurable parameters, but is essentially very simple.  Graphical
    applications which run test suites should provide alternate implementations.
 
@@ -1928,7 +1928,7 @@
       Added the ``warnings`` argument.
 
    .. versionchanged:: 3.2
-      The default stream is set to `sys.stderr` at instantiation time rather
+      The default stream is set to :data:`sys.stderr` at instantiation time rather
       than import time.
 
    .. method:: _makeResult()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:04 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:04 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Minor_improvement_to_extens?=
 =?utf8?q?ions_section_in_setup=2Ecfg=2E?=
Message-ID: <E1QzVvU-00034f-Ge@dinsdale.python.org>

http://hg.python.org/cpython/rev/0cdd174d1bc4
changeset:   72180:0cdd174d1bc4
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 06:29:11 2011 +0200
summary:
  Minor improvement to extensions section in setup.cfg.

The right-hand part in [extension: foo] is now used as the name of the
extension module.  (I changed the separator from = to : and allowed
whitespace to make the sections look nicer.)

files:
  Doc/packaging/setupcfg.rst         |   9 +++++----
  Lib/packaging/config.py            |  11 +++++++----
  Lib/packaging/tests/test_config.py |  16 +++++++++++-----
  3 files changed, 23 insertions(+), 13 deletions(-)


diff --git a/Doc/packaging/setupcfg.rst b/Doc/packaging/setupcfg.rst
--- a/Doc/packaging/setupcfg.rst
+++ b/Doc/packaging/setupcfg.rst
@@ -756,8 +756,7 @@
    [files]
    packages = coconut
 
-   [extension=_fastcoconut]
-   name = coconut._fastcoconut
+   [extension: coconut._fastcoconut]
    language = cxx
    sources = cxx_src/cononut_utils.cxx
              cxx_src/python_module.cxx
@@ -768,8 +767,10 @@
        -DGECODE_VERSION=$(./gecode_version) -- sys.platform != 'win32'
        /DGECODE_VERSION='win32' -- sys.platform == 'win32'
 
-The section name must start with ``extension=``; the righ-hand part is currently
-discarded.  Valid fields and their values are listed in the documentation of the
+The section name must start with ``extension:``; the right-hand part is used as
+the full name (including a parent package, if any) of the extension.  Whitespace
+around the extension name is allowed.
+Valid fields and their values are listed in the documentation of the
 :class:`packaging.compiler.extension.Extension` class; values documented as
 Python lists translate to multi-line values in the configuration file.  In
 addition, multi-line values accept environment markers on each line, after a
diff --git a/Lib/packaging/config.py b/Lib/packaging/config.py
--- a/Lib/packaging/config.py
+++ b/Lib/packaging/config.py
@@ -251,13 +251,16 @@
 
         ext_modules = self.dist.ext_modules
         for section_key in content:
-            labels = section_key.split('=')
+            # no str.partition in 2.4 :(
+            labels = section_key.split(':')
             if len(labels) == 2 and labels[0] == 'extension':
-                # labels[1] not used from now but should be implemented
-                # for extension build dependency
                 values_dct = content[section_key]
+                if 'name' in values_dct:
+                    raise PackagingOptionError(
+                        'extension name should be given as [extension: name], '
+                        'not as key')
                 ext_modules.append(Extension(
-                    values_dct.pop('name'),
+                    labels[1].strip(),
                     _pop_values(values_dct, 'sources'),
                     _pop_values(values_dct, 'include_dirs'),
                     _pop_values(values_dct, 'define_macros'),
diff --git a/Lib/packaging/tests/test_config.py b/Lib/packaging/tests/test_config.py
--- a/Lib/packaging/tests/test_config.py
+++ b/Lib/packaging/tests/test_config.py
@@ -6,7 +6,7 @@
 
 from packaging import command
 from packaging.dist import Distribution
-from packaging.errors import PackagingFileError
+from packaging.errors import PackagingFileError, PackagingOptionError
 from packaging.compiler import new_compiler, _COMPILERS
 from packaging.command.sdist import sdist
 
@@ -100,21 +100,20 @@
 
 # Can not be merged with SETUP_CFG else install_dist
 # command will fail when trying to compile C sources
+# TODO use a DummyCommand to mock build_ext
 EXT_SETUP_CFG = """
 [files]
 packages = one
            two
 
-[extension=speed_coconuts]
-name = one.speed_coconuts
+[extension:one.speed_coconuts]
 sources = c_src/speed_coconuts.c
 extra_link_args = "`gcc -print-file-name=libgcc.a`" -shared
 define_macros = HAVE_CAIRO HAVE_GTK2
 libraries = gecodeint gecodekernel -- sys.platform != 'win32'
     GecodeInt GecodeKernel -- sys.platform == 'win32'
 
-[extension=fast_taunt]
-name = two.fast_taunt
+[extension: two.fast_taunt]
 sources = cxx_src/utils_taunt.cxx
           cxx_src/python_module.cxx
 include_dirs = /usr/include/gecode
@@ -123,7 +122,11 @@
     -DGECODE_VERSION=$(./gecode_version) -- sys.platform != 'win32'
     /DGECODE_VERSION='win32' -- sys.platform == 'win32'
 language = cxx
+"""
 
+EXT_SETUP_CFG_BUGGY_1 = """
+[extension: realname]
+name = crash_here
 """
 
 HOOKS_MODULE = """
@@ -335,6 +338,9 @@
         self.assertEqual(ext.extra_compile_args, cargs)
         self.assertEqual(ext.language, 'cxx')
 
+        self.write_file('setup.cfg', EXT_SETUP_CFG_BUGGY_1)
+        self.assertRaises(PackagingOptionError, self.get_dist)
+
     def test_project_setup_hook_works(self):
         # Bug #11637: ensure the project directory is on sys.path to allow
         # project-specific hooks

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:07 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Minor_improvement_to_extens?=
 =?utf8?q?ions_in_setup=2Ecfg=3A_check_parent_package?=
Message-ID: <E1QzVvX-00038G-Gw@dinsdale.python.org>

http://hg.python.org/cpython/rev/c1949bde461e
changeset:   72181:c1949bde461e
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 07:01:13 2011 +0200
summary:
  Minor improvement to extensions in setup.cfg: check parent package

files:
  Doc/packaging/setupcfg.rst         |   4 +-
  Lib/packaging/config.py            |  20 ++++++++++-
  Lib/packaging/tests/test_config.py |  31 ++++++++++++++++-
  3 files changed, 50 insertions(+), 5 deletions(-)


diff --git a/Doc/packaging/setupcfg.rst b/Doc/packaging/setupcfg.rst
--- a/Doc/packaging/setupcfg.rst
+++ b/Doc/packaging/setupcfg.rst
@@ -769,7 +769,9 @@
 
 The section name must start with ``extension:``; the right-hand part is used as
 the full name (including a parent package, if any) of the extension.  Whitespace
-around the extension name is allowed.
+around the extension name is allowed.  If the extension module is not standalone
+(e.g. ``_bisect``) but part of a package (e.g. ``thing._speedups``), the parent
+package must be listed in the ``packages`` field.
 Valid fields and their values are listed in the documentation of the
 :class:`packaging.compiler.extension.Extension` class; values documented as
 Python lists translate to multi-line values in the configuration file.  In
diff --git a/Lib/packaging/config.py b/Lib/packaging/config.py
--- a/Lib/packaging/config.py
+++ b/Lib/packaging/config.py
@@ -16,6 +16,19 @@
 from packaging.markers import interpret
 
 
+def _check_name(name, packages):
+    if '.' not in name:
+        return
+    parts = name.split('.')
+    modname = parts[-1]
+    parent = '.'.join(parts[:-1])
+    if parent not in packages:
+        # we could log a warning instead of raising, but what's the use
+        # of letting people build modules they can't import?
+        raise PackagingOptionError(
+            'parent package for extension %r not found' % name)
+
+
 def _pop_values(values_dct, key):
     """Remove values from the dictionary and convert them as a list"""
     vals_str = values_dct.pop(key, '')
@@ -142,7 +155,8 @@
                         try:
                             hook = resolve_name(line)
                         except ImportError as e:
-                            logger.warning('cannot find setup hook: %s', e.args[0])
+                            logger.warning('cannot find setup hook: %s',
+                                           e.args[0])
                         else:
                             self.setup_hooks.append(hook)
                     self.run_hooks(content)
@@ -259,8 +273,10 @@
                     raise PackagingOptionError(
                         'extension name should be given as [extension: name], '
                         'not as key')
+                name = labels[1].strip()
+                _check_name(name, self.dist.packages)
                 ext_modules.append(Extension(
-                    labels[1].strip(),
+                    name,
                     _pop_values(values_dct, 'sources'),
                     _pop_values(values_dct, 'include_dirs'),
                     _pop_values(values_dct, 'define_macros'),
diff --git a/Lib/packaging/tests/test_config.py b/Lib/packaging/tests/test_config.py
--- a/Lib/packaging/tests/test_config.py
+++ b/Lib/packaging/tests/test_config.py
@@ -105,6 +105,7 @@
 [files]
 packages = one
            two
+           parent.undeclared
 
 [extension:one.speed_coconuts]
 sources = c_src/speed_coconuts.c
@@ -122,6 +123,11 @@
     -DGECODE_VERSION=$(./gecode_version) -- sys.platform != 'win32'
     /DGECODE_VERSION='win32' -- sys.platform == 'win32'
 language = cxx
+
+# corner case: if the parent package of an extension is declared but
+# not its grandparent, it's legal
+[extension: parent.undeclared._speed]
+sources = parent/undeclared/_speed.c
 """
 
 EXT_SETUP_CFG_BUGGY_1 = """
@@ -129,6 +135,21 @@
 name = crash_here
 """
 
+EXT_SETUP_CFG_BUGGY_2 = """
+[files]
+packages = ham
+
+[extension: spam.eggs]
+"""
+
+EXT_SETUP_CFG_BUGGY_3 = """
+[files]
+packages = ok
+           ok.works
+
+[extension: ok.works.breaks._ext]
+"""
+
 HOOKS_MODULE = """
 import logging
 
@@ -314,7 +335,7 @@
         dist = self.get_dist()
 
         ext_modules = dict((mod.name, mod) for mod in dist.ext_modules)
-        self.assertEqual(len(ext_modules), 2)
+        self.assertEqual(len(ext_modules), 3)
         ext = ext_modules.get('one.speed_coconuts')
         self.assertEqual(ext.sources, ['c_src/speed_coconuts.c'])
         self.assertEqual(ext.define_macros, ['HAVE_CAIRO', 'HAVE_GTK2'])
@@ -341,6 +362,12 @@
         self.write_file('setup.cfg', EXT_SETUP_CFG_BUGGY_1)
         self.assertRaises(PackagingOptionError, self.get_dist)
 
+        self.write_file('setup.cfg', EXT_SETUP_CFG_BUGGY_2)
+        self.assertRaises(PackagingOptionError, self.get_dist)
+
+        self.write_file('setup.cfg', EXT_SETUP_CFG_BUGGY_3)
+        self.assertRaises(PackagingOptionError, self.get_dist)
+
     def test_project_setup_hook_works(self):
         # Bug #11637: ensure the project directory is on sys.path to allow
         # project-specific hooks
@@ -364,7 +391,7 @@
         self.write_setup({
             'setup-hooks': '\n  packaging.tests.test_config.first_hook'
                            '\n  packaging.tests.test_config.missing_hook'
-                           '\n  packaging.tests.test_config.third_hook'
+                           '\n  packaging.tests.test_config.third_hook',
         })
         self.write_file('README', 'yeah')
         dist = self.get_dist()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:08 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:08 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_missing_name_in_shutil?=
	=?utf8?b?Ll9fYWxsX18=?=
Message-ID: <E1QzVvY-00038O-C0@dinsdale.python.org>

http://hg.python.org/cpython/rev/b166dc85f604
changeset:   72182:b166dc85f604
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 08:31:51 2011 +0200
summary:
  Add missing name in shutil.__all__

files:
  Lib/shutil.py |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Lib/shutil.py b/Lib/shutil.py
--- a/Lib/shutil.py
+++ b/Lib/shutil.py
@@ -35,7 +35,7 @@
            "register_archive_format", "unregister_archive_format",
            "get_unpack_formats", "register_unpack_format",
            "unregister_unpack_format", "unpack_archive",
-           "ignore_patterns"]
+           "ignore_patterns", "chown"]
            # disk_usage is added later, if available on the platform
 
 class Error(EnvironmentError):
@@ -791,6 +791,7 @@
         used = total - free
         return _ntuple_diskusage(total, used, free)
 
+
 def chown(path, user=None, group=None):
     """Change owner user and group of the given path.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:08 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:08 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Add_version_num?=
 =?utf8?q?ber_for_versionchanged_directive_=28backport_from_3=2E3=29?=
Message-ID: <E1QzVvY-00038S-UD@dinsdale.python.org>

http://hg.python.org/cpython/rev/e6f18c055ae6
changeset:   72183:e6f18c055ae6
branch:      3.2
parent:      72177:fcb037cf261b
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 05:55:26 2011 +0200
summary:
  Add version number for versionchanged directive (backport from 3.3)

files:
  Doc/library/unittest.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -723,7 +723,7 @@
    Here, we create two instances of :class:`WidgetTestCase`, each of which runs a
    single test.
 
-   .. versionchanged::
+   .. versionchanged:: 3.2
       :class:`TestCase` can be instantiated successfully without providing a method
       name. This makes it easier to experiment with :class:`TestCase` from the
       interactive interpreter.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:09 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:09 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Document_that_T?=
 =?utf8?q?rue/False/None_don=E2=80=99t_use_=3Akeyword=3A_in_doc=2E?=
Message-ID: <E1QzVvZ-00038W-Fa@dinsdale.python.org>

http://hg.python.org/cpython/rev/f2e0481f28db
changeset:   72184:f2e0481f28db
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 18:45:50 2011 +0200
summary:
  Document that True/False/None don?t use :keyword: in doc.

This was discussed some months ago on python-dev.  Having tons of links
to the definition of True would be annoying, contrary to links to e.g.
the nonlocal or with statements doc.

files:
  Doc/documenting/markup.rst |  5 ++++-
  1 files changed, 4 insertions(+), 1 deletions(-)


diff --git a/Doc/documenting/markup.rst b/Doc/documenting/markup.rst
--- a/Doc/documenting/markup.rst
+++ b/Doc/documenting/markup.rst
@@ -513,7 +513,10 @@
 
 .. describe:: keyword
 
-   The name of a keyword in Python.
+   The name of a Python keyword.  Using this role will generate a link to the
+   documentation of the keyword.  ``True``, ``False`` and ``None`` do not use
+   this role, but simple code markup (````True````), given that they're
+   fundamental to the language and should be known to any programmer.
 
 .. describe:: mailheader
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:17 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:17 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Document_that_f?=
 =?utf8?q?ormat_string_don=E2=80=99t_support_arbitrary_dictonary_keys=2E?=
Message-ID: <E1QzVvh-0003AW-5m@dinsdale.python.org>

http://hg.python.org/cpython/rev/b2a948f42f92
changeset:   72185:b2a948f42f92
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 18:59:06 2011 +0200
summary:
  Document that format string don?t support arbitrary dictonary keys.

Text adapted from the PEP.  Addition requested by Terry J. Reedy on
2011-02-23 on python-dev.

files:
  Doc/library/string.rst |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Doc/library/string.rst b/Doc/library/string.rst
--- a/Doc/library/string.rst
+++ b/Doc/library/string.rst
@@ -216,6 +216,8 @@
 it refers to a named keyword argument.  If the numerical arg_names in a format string
 are 0, 1, 2, ... in sequence, they can all be omitted (not just some)
 and the numbers 0, 1, 2, ... will be automatically inserted in that order.
+Because *arg_name* is not quote-delimited, it is not possible to specify arbitrary
+dictionary keys (e.g., the strings ``'10'`` or ``':-]'``) within a format string.
 The *arg_name* can be followed by any number of index or
 attribute expressions. An expression of the form ``'.name'`` selects the named
 attribute using :func:`getattr`, while an expression of the form ``'[index]'``

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:17 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:17 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzEwNDU0OiBhIGZl?=
 =?utf8?q?w_edits_to_compileall_help_messages?=
Message-ID: <E1QzVvh-0003Aa-SC@dinsdale.python.org>

http://hg.python.org/cpython/rev/755a57f987ca
changeset:   72186:755a57f987ca
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 20:00:33 2011 +0200
summary:
  #10454: a few edits to compileall help messages

files:
  Lib/compileall.py |  14 +++++++-------
  1 files changed, 7 insertions(+), 7 deletions(-)


diff --git a/Lib/compileall.py b/Lib/compileall.py
--- a/Lib/compileall.py
+++ b/Lib/compileall.py
@@ -142,7 +142,7 @@
 
     Arguments (all optional):
 
-    skip_curdir: if true, skip current directory (default true)
+    skip_curdir: if true, skip current directory (default True)
     maxlevels:   max recursion level (default 0)
     force: as for compile_dir() (default False)
     quiet: as for compile_dir() (default False)
@@ -177,17 +177,17 @@
                         help='use legacy (pre-PEP3147) compiled file locations')
     parser.add_argument('-d', metavar='DESTDIR',  dest='ddir', default=None,
                         help=('directory to prepend to file paths for use in '
-                              'compile time tracebacks and in runtime '
+                              'compile-time tracebacks and in runtime '
                               'tracebacks in cases where the source file is '
                               'unavailable'))
     parser.add_argument('-x', metavar='REGEXP', dest='rx', default=None,
-                        help=('skip files matching the regular expression. '
-                              'The regexp is searched for in the full path '
-                              'to each file considered for compilation.'))
+                        help=('skip files matching the regular expression; '
+                              'the regexp is searched for in the full path '
+                              'of each file considered for compilation'))
     parser.add_argument('-i', metavar='FILE', dest='flist',
                         help=('add all the files and directories listed in '
-                              'FILE to the list considered for compilation. '
-                              'If "-", names are read from stdin.'))
+                              'FILE to the list considered for compilation; '
+                              'if "-", names are read from stdin'))
     parser.add_argument('compile_dest', metavar='FILE|DIR', nargs='*',
                         help=('zero or more file and directory names '
                               'to compile; if no arguments given, defaults '

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:21 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:21 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Remove_obsolete?=
	=?utf8?q?_comment?=
Message-ID: <E1QzVvl-0003Ah-5D@dinsdale.python.org>

http://hg.python.org/cpython/rev/ae8491fbb838
changeset:   72187:ae8491fbb838
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 22:06:23 2011 +0200
summary:
  Remove obsolete comment

files:
  Lib/pipes.py |  2 --
  1 files changed, 0 insertions(+), 2 deletions(-)


diff --git a/Lib/pipes.py b/Lib/pipes.py
--- a/Lib/pipes.py
+++ b/Lib/pipes.py
@@ -54,8 +54,6 @@
 
 To create a new template object initialized to a given one:
    t2 = t.clone()
-
-For an example, see the function test() at the end of the file.
 """                                     # '
 
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:21 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:21 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_a_few_links?=
 =?utf8?q?_in_the_table_of_built-in_functions_=28=2312298=29?=
Message-ID: <E1QzVvl-0003Al-T2@dinsdale.python.org>

http://hg.python.org/cpython/rev/7a05cb3beddf
changeset:   72188:7a05cb3beddf
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 23:08:55 2011 +0200
summary:
  Fix a few links in the table of built-in functions (#12298)

files:
  Doc/library/functions.rst |  23 ++++++++++++++++++++---
  1 files changed, 20 insertions(+), 3 deletions(-)


diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -10,7 +10,7 @@
 ===================  =================  ==================  ================  ====================
 ..                   ..                 Built-in Functions  ..                ..
 ===================  =================  ==================  ================  ====================
-:func:`abs`          :func:`dict`       :func:`help`        :func:`min`       :func:`setattr`
+:func:`abs`          |func-dict|_       :func:`help`        :func:`min`       :func:`setattr`
 :func:`all`          :func:`dir`        :func:`hex`         :func:`next`      :func:`slice`
 :func:`any`          :func:`divmod`     :func:`id`          :func:`object`    :func:`sorted`
 :func:`ascii`        :func:`enumerate`  :func:`input`       :func:`oct`       :func:`staticmethod`
@@ -19,13 +19,22 @@
 :func:`bytearray`    :func:`filter`     :func:`issubclass`  :func:`pow`       :func:`super`
 :func:`bytes`        :func:`float`      :func:`iter`        :func:`print`     :func:`tuple`
 :func:`callable`     :func:`format`     :func:`len`         :func:`property`  :func:`type`
-:func:`chr`          :func:`frozenset`  :func:`list`        :func:`range`     :func:`vars`
+:func:`chr`          |func-frozenset|_  :func:`list`        :func:`range`     :func:`vars`
 :func:`classmethod`  :func:`getattr`    :func:`locals`      :func:`repr`      :func:`zip`
 :func:`compile`      :func:`globals`    :func:`map`         :func:`reversed`  :func:`__import__`
 :func:`complex`      :func:`hasattr`    :func:`max`         :func:`round`
-:func:`delattr`      :func:`hash`       :func:`memoryview`  :func:`set`
+:func:`delattr`      :func:`hash`       |func-memoryview|_  |func-set|_
 ===================  =================  ==================  ================  ====================
 
+.. using :func:`dict` would create a link to another page, so local targets are
+   used, with replacement texts to make the output in the table consistent
+
+.. |func-dict| replace:: ``dict()``
+.. |func-frozenset| replace:: ``frozenset()``
+.. |func-memoryview| replace:: ``memoryview()``
+.. |func-set| replace:: ``set()``
+
+
 .. function:: abs(x)
 
    Return the absolute value of a number.  The argument may be an
@@ -248,6 +257,7 @@
    example, ``delattr(x, 'foobar')`` is equivalent to ``del x.foobar``.
 
 
+.. _func-dict:
 .. function:: dict([arg])
    :noindex:
 
@@ -491,6 +501,7 @@
 
    The float type is described in :ref:`typesnumeric`.
 
+
 .. function:: format(value[, format_spec])
 
    .. index::
@@ -511,6 +522,8 @@
    :exc:`TypeError` exception is raised if the method is not found or if either
    the *format_spec* or the return value are not strings.
 
+
+.. _func-frozenset:
 .. function:: frozenset([iterable])
    :noindex:
 
@@ -717,6 +730,8 @@
    such as ``sorted(iterable, key=keyfunc, reverse=True)[0]`` and
    ``heapq.nlargest(1, iterable, key=keyfunc)``.
 
+
+.. _func-memoryview:
 .. function:: memoryview(obj)
    :noindex:
 
@@ -1108,6 +1123,8 @@
       can't be represented exactly as a float.  See :ref:`tut-fp-issues` for
       more information.
 
+
+.. _func-set:
 .. function:: set([iterable])
    :noindex:
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:23 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:23 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Add_links_from_?=
 =?utf8?q?library/functions_to_other_docs=2E?=
Message-ID: <E1QzVvn-0003Aq-Vg@dinsdale.python.org>

http://hg.python.org/cpython/rev/58dd7addef3a
changeset:   72189:58dd7addef3a
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 23:10:36 2011 +0200
summary:
  Add links from library/functions to other docs.

Suggested by Terry J. Reedy in #12298.

files:
  Doc/library/functions.rst |  13 +++++++------
  Doc/library/stdtypes.rst  |   4 ++++
  2 files changed, 11 insertions(+), 6 deletions(-)


diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -83,11 +83,12 @@
 
 .. function:: bool([x])
 
-   Convert a value to a Boolean, using the standard truth testing procedure.  If
-   *x* is false or omitted, this returns :const:`False`; otherwise it returns
-   :const:`True`. :class:`bool` is also a class, which is a subclass of
-   :class:`int`. Class :class:`bool` cannot be subclassed further.  Its only
-   instances are :const:`False` and :const:`True`.
+   Convert a value to a Boolean, using the standard :ref:`truth testing
+   procedure <truth>`.  If *x* is false or omitted, this returns ``False``;
+   otherwise it returns ``True``. :class:`bool` is also a class, which is a
+   subclass of :class:`int` (see :ref:`typesnumeric`).  Class :class:`bool`
+   cannot be subclassed further.  Its only instances are ``False`` and
+   ``True`` (see :ref:`bltin-boolean-values`).
 
    .. index:: pair: Boolean; type
 
@@ -1055,7 +1056,7 @@
 
    Range objects implement the :class:`collections.Sequence` ABC, and provide
    features such as containment tests, element index lookup, slicing and
-   support for negative indices:
+   support for negative indices (see :ref:`typesseq`):
 
       >>> r = range(0, 20, 2)
       >>> r
diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -2712,6 +2712,8 @@
 It is written as ``Ellipsis`` or ``...``.
 
 
+.. _bltin-notimplemented-object:
+
 The NotImplemented Object
 -------------------------
 
@@ -2722,6 +2724,8 @@
 It is written as ``NotImplemented``.
 
 
+.. _bltin-boolean-values:
+
 Boolean Values
 --------------
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:24 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:24 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Minor=3A_tweak_docstrings_a?=
 =?utf8?q?nd_=5F=5Fall=5F=5F_in_packaging=2Etests=2Esupport?=
Message-ID: <E1QzVvo-0003Au-Qx@dinsdale.python.org>

http://hg.python.org/cpython/rev/9526cd63632f
changeset:   72190:9526cd63632f
parent:      72182:b166dc85f604
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 23:48:13 2011 +0200
summary:
  Minor: tweak docstrings and __all__ in packaging.tests.support

files:
  Lib/packaging/tests/support.py |  40 ++++++++++++++-------
  1 files changed, 27 insertions(+), 13 deletions(-)


diff --git a/Lib/packaging/tests/support.py b/Lib/packaging/tests/support.py
--- a/Lib/packaging/tests/support.py
+++ b/Lib/packaging/tests/support.py
@@ -1,5 +1,8 @@
 """Support code for packaging test cases.
 
+*This module should not be considered public: its content and API may
+change in incompatible ways.*
+
 A few helper classes are provided: LoggingCatcher, TempdirManager and
 EnvironRestorer. They are written to be used as mixins::
 
@@ -7,6 +10,7 @@
     from packaging.tests.support import LoggingCatcher
 
     class SomeTestCase(LoggingCatcher, unittest.TestCase):
+        ...
 
 If you need to define a setUp method on your test class, you have to
 call the mixin class' setUp method or it won't work (same thing for
@@ -14,17 +18,18 @@
 
         def setUp(self):
             super(SomeTestCase, self).setUp()
-            ... # other setup code
+            ...  # other setup code
 
 Also provided is a DummyCommand class, useful to mock commands in the
-tests of another command that needs them, a create_distribution function
-and a skip_unless_symlink decorator.
+tests of another command that needs them, for example to fake
+compilation in build_ext (this requires that the mock build_ext command
+be injected into the distribution object's command_obj dictionary).
 
-Also provided is a DummyCommand class, useful to mock commands in the
-tests of another command that needs them, a create_distribution function
-and a skip_unless_symlink decorator.
+For tests that need to compile an extension module, use the
+copy_xxmodule_c and fixup_build_ext functions.
 
 Each class or function has a docstring to explain its purpose and usage.
+Existing tests should also be used as examples.
 """
 
 import os
@@ -39,9 +44,17 @@
 from packaging.tests import unittest
 from test.support import requires_zlib, unlink
 
-__all__ = ['LoggingCatcher', 'TempdirManager', 'EnvironRestorer',
-           'DummyCommand', 'unittest', 'create_distribution',
-           'skip_unless_symlink', 'requires_zlib', 'copy_xxmodule_c']
+# define __all__ to make pydoc more useful
+__all__ = [
+    # TestCase mixins
+    'LoggingCatcher', 'TempdirManager', 'EnvironRestorer',
+    # mocks
+    'DummyCommand', 'TestDistribution',
+    # misc. functions and decorators
+    'fake_dec', 'create_distribution', 'copy_xxmodule_c', 'fixup_build_ext',
+    # imported from this module for backport purposes
+    'unittest', 'requires_zlib', 'skip_unless_symlink',
+]
 
 
 logger = logging.getLogger('packaging')
@@ -233,6 +246,8 @@
     Useful for mocking one dependency command in the tests for another
     command, see e.g. the dummy build command in test_build_scripts.
     """
+    # XXX does not work with dist.get_reinitialized_command, which typechecks
+    # and wants a finalized attribute
 
     def __init__(self, **kwargs):
         for kw, val in kwargs.items():
@@ -308,10 +323,9 @@
 def fixup_build_ext(cmd):
     """Function needed to make build_ext tests pass.
 
-    When Python was build with --enable-shared on Unix, -L. is not good
-    enough to find the libpython<blah>.so.  This is because regrtest runs
-    it under a tempdir, not in the top level where the .so lives.  By the
-    time we've gotten here, Python's already been chdir'd to the tempdir.
+    When Python was built with --enable-shared on Unix, -L. is not enough to
+    find libpython<blah>.so, because regrtest runs in a tempdir, not in the
+    source directory where the .so lives.
 
     When Python was built with in debug mode on Windows, build_ext commands
     need their debug attribute set, and it is not done automatically for

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:25 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:25 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_typo_=28was?=
 =?utf8?q?_build=29_and_remove_redundancy_in_docstring?=
Message-ID: <E1QzVvp-0003Ay-Hi@dinsdale.python.org>

http://hg.python.org/cpython/rev/941106bcd327
changeset:   72191:941106bcd327
branch:      3.2
parent:      72189:58dd7addef3a
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 23:37:56 2011 +0200
summary:
  Fix typo (was build) and remove redundancy in docstring

files:
  Lib/distutils/tests/support.py |  7 +++----
  1 files changed, 3 insertions(+), 4 deletions(-)


diff --git a/Lib/distutils/tests/support.py b/Lib/distutils/tests/support.py
--- a/Lib/distutils/tests/support.py
+++ b/Lib/distutils/tests/support.py
@@ -175,10 +175,9 @@
 def fixup_build_ext(cmd):
     """Function needed to make build_ext tests pass.
 
-    When Python was build with --enable-shared on Unix, -L. is not good
-    enough to find the libpython<blah>.so.  This is because regrtest runs
-    it under a tempdir, not in the top level where the .so lives.  By the
-    time we've gotten here, Python's already been chdir'd to the tempdir.
+    When Python was built with --enable-shared on Unix, -L. is not enough to
+    find libpython<blah>.so, because regrtest runs in a tempdir, not in the
+    source directory where the .so lives.
 
     When Python was built with in debug mode on Windows, build_ext commands
     need their debug attribute set, and it is not done automatically for

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:26 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:26 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_doc_changes_from_3=2E2_=28=2310454=2C_=2312298=29?=
Message-ID: <E1QzVvq-0003B2-6I@dinsdale.python.org>

http://hg.python.org/cpython/rev/892e0ee4ca32
changeset:   72192:892e0ee4ca32
parent:      72190:9526cd63632f
parent:      72191:941106bcd327
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Sep 02 00:03:20 2011 +0200
summary:
  Merge doc changes from 3.2 (#10454, #12298)

files:
  Doc/documenting/markup.rst     |   5 ++-
  Doc/library/functions.rst      |  36 ++++++++++++++++-----
  Doc/library/stdtypes.rst       |   4 ++
  Doc/library/string.rst         |   2 +
  Lib/compileall.py              |  14 ++++----
  Lib/distutils/tests/support.py |   7 +--
  Lib/pipes.py                   |   2 -
  7 files changed, 47 insertions(+), 23 deletions(-)


diff --git a/Doc/documenting/markup.rst b/Doc/documenting/markup.rst
--- a/Doc/documenting/markup.rst
+++ b/Doc/documenting/markup.rst
@@ -513,7 +513,10 @@
 
 .. describe:: keyword
 
-   The name of a keyword in Python.
+   The name of a Python keyword.  Using this role will generate a link to the
+   documentation of the keyword.  ``True``, ``False`` and ``None`` do not use
+   this role, but simple code markup (````True````), given that they're
+   fundamental to the language and should be known to any programmer.
 
 .. describe:: mailheader
 
diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -10,7 +10,7 @@
 ===================  =================  ==================  ================  ====================
 ..                   ..                 Built-in Functions  ..                ..
 ===================  =================  ==================  ================  ====================
-:func:`abs`          :func:`dict`       :func:`help`        :func:`min`       :func:`setattr`
+:func:`abs`          |func-dict|_       :func:`help`        :func:`min`       :func:`setattr`
 :func:`all`          :func:`dir`        :func:`hex`         :func:`next`      :func:`slice`
 :func:`any`          :func:`divmod`     :func:`id`          :func:`object`    :func:`sorted`
 :func:`ascii`        :func:`enumerate`  :func:`input`       :func:`oct`       :func:`staticmethod`
@@ -19,13 +19,22 @@
 :func:`bytearray`    :func:`filter`     :func:`issubclass`  :func:`pow`       :func:`super`
 :func:`bytes`        :func:`float`      :func:`iter`        :func:`print`     :func:`tuple`
 :func:`callable`     :func:`format`     :func:`len`         :func:`property`  :func:`type`
-:func:`chr`          :func:`frozenset`  :func:`list`        :func:`range`     :func:`vars`
+:func:`chr`          |func-frozenset|_  :func:`list`        :func:`range`     :func:`vars`
 :func:`classmethod`  :func:`getattr`    :func:`locals`      :func:`repr`      :func:`zip`
 :func:`compile`      :func:`globals`    :func:`map`         :func:`reversed`  :func:`__import__`
 :func:`complex`      :func:`hasattr`    :func:`max`         :func:`round`
-:func:`delattr`      :func:`hash`       :func:`memoryview`  :func:`set`
+:func:`delattr`      :func:`hash`       |func-memoryview|_  |func-set|_
 ===================  =================  ==================  ================  ====================
 
+.. using :func:`dict` would create a link to another page, so local targets are
+   used, with replacement texts to make the output in the table consistent
+
+.. |func-dict| replace:: ``dict()``
+.. |func-frozenset| replace:: ``frozenset()``
+.. |func-memoryview| replace:: ``memoryview()``
+.. |func-set| replace:: ``set()``
+
+
 .. function:: abs(x)
 
    Return the absolute value of a number.  The argument may be an
@@ -74,11 +83,12 @@
 
 .. function:: bool([x])
 
-   Convert a value to a Boolean, using the standard truth testing procedure.  If
-   *x* is false or omitted, this returns :const:`False`; otherwise it returns
-   :const:`True`. :class:`bool` is also a class, which is a subclass of
-   :class:`int`. Class :class:`bool` cannot be subclassed further.  Its only
-   instances are :const:`False` and :const:`True`.
+   Convert a value to a Boolean, using the standard :ref:`truth testing
+   procedure <truth>`.  If *x* is false or omitted, this returns ``False``;
+   otherwise it returns ``True``. :class:`bool` is also a class, which is a
+   subclass of :class:`int` (see :ref:`typesnumeric`).  Class :class:`bool`
+   cannot be subclassed further.  Its only instances are ``False`` and
+   ``True`` (see :ref:`bltin-boolean-values`).
 
    .. index:: pair: Boolean; type
 
@@ -248,6 +258,7 @@
    example, ``delattr(x, 'foobar')`` is equivalent to ``del x.foobar``.
 
 
+.. _func-dict:
 .. function:: dict([arg])
    :noindex:
 
@@ -491,6 +502,7 @@
 
    The float type is described in :ref:`typesnumeric`.
 
+
 .. function:: format(value[, format_spec])
 
    .. index::
@@ -511,6 +523,8 @@
    :exc:`TypeError` exception is raised if the method is not found or if either
    the *format_spec* or the return value are not strings.
 
+
+.. _func-frozenset:
 .. function:: frozenset([iterable])
    :noindex:
 
@@ -717,6 +731,8 @@
    such as ``sorted(iterable, key=keyfunc, reverse=True)[0]`` and
    ``heapq.nlargest(1, iterable, key=keyfunc)``.
 
+
+.. _func-memoryview:
 .. function:: memoryview(obj)
    :noindex:
 
@@ -1040,7 +1056,7 @@
 
    Range objects implement the :class:`collections.Sequence` ABC, and provide
    features such as containment tests, element index lookup, slicing and
-   support for negative indices:
+   support for negative indices (see :ref:`typesseq`):
 
       >>> r = range(0, 20, 2)
       >>> r
@@ -1108,6 +1124,8 @@
       can't be represented exactly as a float.  See :ref:`tut-fp-issues` for
       more information.
 
+
+.. _func-set:
 .. function:: set([iterable])
    :noindex:
 
diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -2724,6 +2724,8 @@
 It is written as ``Ellipsis`` or ``...``.
 
 
+.. _bltin-notimplemented-object:
+
 The NotImplemented Object
 -------------------------
 
@@ -2735,6 +2737,8 @@
 It is written as ``NotImplemented``.
 
 
+.. _bltin-boolean-values:
+
 Boolean Values
 --------------
 
diff --git a/Doc/library/string.rst b/Doc/library/string.rst
--- a/Doc/library/string.rst
+++ b/Doc/library/string.rst
@@ -216,6 +216,8 @@
 it refers to a named keyword argument.  If the numerical arg_names in a format string
 are 0, 1, 2, ... in sequence, they can all be omitted (not just some)
 and the numbers 0, 1, 2, ... will be automatically inserted in that order.
+Because *arg_name* is not quote-delimited, it is not possible to specify arbitrary
+dictionary keys (e.g., the strings ``'10'`` or ``':-]'``) within a format string.
 The *arg_name* can be followed by any number of index or
 attribute expressions. An expression of the form ``'.name'`` selects the named
 attribute using :func:`getattr`, while an expression of the form ``'[index]'``
diff --git a/Lib/compileall.py b/Lib/compileall.py
--- a/Lib/compileall.py
+++ b/Lib/compileall.py
@@ -142,7 +142,7 @@
 
     Arguments (all optional):
 
-    skip_curdir: if true, skip current directory (default true)
+    skip_curdir: if true, skip current directory (default True)
     maxlevels:   max recursion level (default 0)
     force: as for compile_dir() (default False)
     quiet: as for compile_dir() (default False)
@@ -177,17 +177,17 @@
                         help='use legacy (pre-PEP3147) compiled file locations')
     parser.add_argument('-d', metavar='DESTDIR',  dest='ddir', default=None,
                         help=('directory to prepend to file paths for use in '
-                              'compile time tracebacks and in runtime '
+                              'compile-time tracebacks and in runtime '
                               'tracebacks in cases where the source file is '
                               'unavailable'))
     parser.add_argument('-x', metavar='REGEXP', dest='rx', default=None,
-                        help=('skip files matching the regular expression. '
-                              'The regexp is searched for in the full path '
-                              'to each file considered for compilation.'))
+                        help=('skip files matching the regular expression; '
+                              'the regexp is searched for in the full path '
+                              'of each file considered for compilation'))
     parser.add_argument('-i', metavar='FILE', dest='flist',
                         help=('add all the files and directories listed in '
-                              'FILE to the list considered for compilation. '
-                              'If "-", names are read from stdin.'))
+                              'FILE to the list considered for compilation; '
+                              'if "-", names are read from stdin'))
     parser.add_argument('compile_dest', metavar='FILE|DIR', nargs='*',
                         help=('zero or more file and directory names '
                               'to compile; if no arguments given, defaults '
diff --git a/Lib/distutils/tests/support.py b/Lib/distutils/tests/support.py
--- a/Lib/distutils/tests/support.py
+++ b/Lib/distutils/tests/support.py
@@ -175,10 +175,9 @@
 def fixup_build_ext(cmd):
     """Function needed to make build_ext tests pass.
 
-    When Python was build with --enable-shared on Unix, -L. is not good
-    enough to find the libpython<blah>.so.  This is because regrtest runs
-    it under a tempdir, not in the top level where the .so lives.  By the
-    time we've gotten here, Python's already been chdir'd to the tempdir.
+    When Python was built with --enable-shared on Unix, -L. is not enough to
+    find libpython<blah>.so, because regrtest runs in a tempdir, not in the
+    source directory where the .so lives.
 
     When Python was built with in debug mode on Windows, build_ext commands
     need their debug attribute set, and it is not done automatically for
diff --git a/Lib/pipes.py b/Lib/pipes.py
--- a/Lib/pipes.py
+++ b/Lib/pipes.py
@@ -54,8 +54,6 @@
 
 To create a new template object initialized to a given one:
    t2 = t.clone()
-
-For an example, see the function test() at the end of the file.
 """                                     # '
 
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:27 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:27 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Branch_merge?=
Message-ID: <E1QzVvr-0003B7-48@dinsdale.python.org>

http://hg.python.org/cpython/rev/017548088eee
changeset:   72193:017548088eee
parent:      72174:a8748022504f
parent:      72192:892e0ee4ca32
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Sep 02 17:30:55 2011 +0200
summary:
  Branch merge

files:
  Doc/c-api/init.rst                   |    4 +-
  Doc/documenting/markup.rst           |    5 +-
  Doc/faq/design.rst                   |    2 +-
  Doc/faq/programming.rst              |    9 -
  Doc/faq/windows.rst                  |    4 +-
  Doc/glossary.rst                     |    6 +-
  Doc/howto/logging.rst                |    6 +-
  Doc/library/argparse.rst             |    2 +-
  Doc/library/base64.rst               |    4 +-
  Doc/library/configparser.rst         |    6 +-
  Doc/library/email.header.rst         |    4 +-
  Doc/library/functions.rst            |   36 ++-
  Doc/library/inspect.rst              |    8 +-
  Doc/library/packaging.util.rst       |   37 ---
  Doc/library/stdtypes.rst             |    4 +
  Doc/library/string.rst               |    2 +
  Doc/library/unittest.rst             |   28 +-
  Doc/packaging/setupcfg.rst           |   11 +-
  Lib/compileall.py                    |   14 +-
  Lib/distutils/tests/support.py       |    7 +-
  Lib/packaging/command/build_py.py    |    2 +-
  Lib/packaging/command/install_lib.py |    2 +-
  Lib/packaging/command/register.py    |    1 -
  Lib/packaging/command/upload.py      |    1 -
  Lib/packaging/config.py              |   31 ++-
  Lib/packaging/create.py              |    6 +-
  Lib/packaging/tests/support.py       |   40 ++-
  Lib/packaging/tests/test_config.py   |   47 +++-
  Lib/packaging/tests/test_util.py     |    9 +-
  Lib/packaging/util.py                |  163 +++-----------
  Lib/pipes.py                         |    2 -
  Lib/shutil.py                        |    3 +-
  32 files changed, 226 insertions(+), 280 deletions(-)


diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -122,7 +122,7 @@
    program name is ``'/usr/local/bin/python'``, the prefix is ``'/usr/local'``. The
    returned string points into static storage; the caller should not modify its
    value.  This corresponds to the :makevar:`prefix` variable in the top-level
-   :file:`Makefile` and the :option:`--prefix` argument to the :program:`configure`
+   :file:`Makefile` and the ``--prefix`` argument to the :program:`configure`
    script at build time.  The value is available to Python code as ``sys.prefix``.
    It is only useful on Unix.  See also the next function.
 
@@ -135,7 +135,7 @@
    program name is ``'/usr/local/bin/python'``, the exec-prefix is
    ``'/usr/local'``.  The returned string points into static storage; the caller
    should not modify its value.  This corresponds to the :makevar:`exec_prefix`
-   variable in the top-level :file:`Makefile` and the :option:`--exec-prefix`
+   variable in the top-level :file:`Makefile` and the ``--exec-prefix``
    argument to the :program:`configure` script at build  time.  The value is
    available to Python code as ``sys.exec_prefix``.  It is only useful on Unix.
 
diff --git a/Doc/documenting/markup.rst b/Doc/documenting/markup.rst
--- a/Doc/documenting/markup.rst
+++ b/Doc/documenting/markup.rst
@@ -513,7 +513,10 @@
 
 .. describe:: keyword
 
-   The name of a keyword in Python.
+   The name of a Python keyword.  Using this role will generate a link to the
+   documentation of the keyword.  ``True``, ``False`` and ``None`` do not use
+   this role, but simple code markup (````True````), given that they're
+   fundamental to the language and should be known to any programmer.
 
 .. describe:: mailheader
 
diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst
--- a/Doc/faq/design.rst
+++ b/Doc/faq/design.rst
@@ -667,7 +667,7 @@
 Python 2.6 adds an :mod:`abc` module that lets you define Abstract Base Classes
 (ABCs).  You can then use :func:`isinstance` and :func:`issubclass` to check
 whether an instance or a class implements a particular ABC.  The
-:mod:`collections` modules defines a set of useful ABCs such as
+:mod:`collections.abc` module defines a set of useful ABCs such as
 :class:`Iterable`, :class:`Container`, and :class:`MutableMapping`.
 
 For Python, many of the advantages of interface specifications can be obtained
diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst
--- a/Doc/faq/programming.rst
+++ b/Doc/faq/programming.rst
@@ -473,15 +473,6 @@
        ...
        g(x, *args, **kwargs)
 
-In the unlikely case that you care about Python versions older than 2.0, use
-:func:`apply`::
-
-   def f(x, *args, **kwargs):
-       ...
-       kwargs['width'] = '14.3c'
-       ...
-       apply(g, (x,)+args, kwargs)
-
 
 How do I write a function with output parameters (call by reference)?
 ---------------------------------------------------------------------
diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst
--- a/Doc/faq/windows.rst
+++ b/Doc/faq/windows.rst
@@ -543,10 +543,10 @@
 
 If you can't change compilers or flags, try using :c:func:`Py_RunSimpleString`.
 A trick to get it to run an arbitrary file is to construct a call to
-:func:`execfile` with the name of your file as argument.
+:func:`exec` and :func:`open` with the name of your file as argument.
 
 Also note that you can not mix-and-match Debug and Release versions.  If you
-wish to use the Debug Multithreaded DLL, then your module *must* have an "_d"
+wish to use the Debug Multithreaded DLL, then your module *must* have ``_d``
 appended to the base name.
 
 
diff --git a/Doc/glossary.rst b/Doc/glossary.rst
--- a/Doc/glossary.rst
+++ b/Doc/glossary.rst
@@ -434,8 +434,8 @@
 
    mapping
       A container object that supports arbitrary key lookups and implements the
-      methods specified in the :class:`~collections.Mapping` or
-      :class:`~collections.MutableMapping`
+      methods specified in the :class:`~collections.abc.Mapping` or
+      :class:`~collections.abc.MutableMapping`
       :ref:`abstract base classes <collections-abstract-base-classes>`.  Examples
       include :class:`dict`, :class:`collections.defaultdict`,
       :class:`collections.OrderedDict` and :class:`collections.Counter`.
@@ -492,7 +492,7 @@
       :func:`builtins.open` and :func:`os.open` are distinguished by their
       namespaces.  Namespaces also aid readability and maintainability by making
       it clear which module implements a function.  For instance, writing
-      :func:`random.seed` or :func:`itertools.izip` makes it clear that those
+      :func:`random.seed` or :func:`itertools.islice` makes it clear that those
       functions are implemented by the :mod:`random` and :mod:`itertools`
       modules, respectively.
 
diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst
--- a/Doc/howto/logging.rst
+++ b/Doc/howto/logging.rst
@@ -412,10 +412,10 @@
   :meth:`Logger.error`, and :meth:`Logger.critical` all create log records with
   a message and a level that corresponds to their respective method names. The
   message is actually a format string, which may contain the standard string
-  substitution syntax of :const:`%s`, :const:`%d`, :const:`%f`, and so on.  The
+  substitution syntax of ``%s``, ``%d``, ``%f``, and so on.  The
   rest of their arguments is a list of objects that correspond with the
-  substitution fields in the message.  With regard to :const:`**kwargs`, the
-  logging methods care only about a keyword of :const:`exc_info` and use it to
+  substitution fields in the message.  With regard to ``**kwargs``, the
+  logging methods care only about a keyword of ``exc_info`` and use it to
   determine whether to log exception information.
 
 * :meth:`Logger.exception` creates a log message similar to
diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst
--- a/Doc/library/argparse.rst
+++ b/Doc/library/argparse.rst
@@ -155,7 +155,7 @@
      conflicting optionals.
 
    * prog_ - The name of the program (default:
-     :data:`sys.argv[0]`)
+     ``sys.argv[0]``)
 
    * usage_ - The string describing the program usage (default: generated)
 
diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst
--- a/Doc/library/base64.rst
+++ b/Doc/library/base64.rst
@@ -45,8 +45,8 @@
    at least length 2 (additional characters are ignored) which specifies the
    alternative alphabet used instead of the ``+`` and ``/`` characters.
 
-   The decoded string is returned.  A `binascii.Error` is raised if *s* is
-   incorrectly padded.
+   The decoded string is returned.  A :exc:`binascii.Error` exception is raised
+   if *s* is incorrectly padded.
 
    If *validate* is ``False`` (the default), non-base64-alphabet characters are
    discarded prior to the padding check.  If *validate* is ``True``,
diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst
--- a/Doc/library/configparser.rst
+++ b/Doc/library/configparser.rst
@@ -806,17 +806,17 @@
    cfg = configparser.ConfigParser()
    cfg.read('example.cfg')
 
-   # Set the optional `raw` argument of get() to True if you wish to disable
+   # Set the optional *raw* argument of get() to True if you wish to disable
    # interpolation in a single get operation.
    print(cfg.get('Section1', 'foo', raw=False)) # -> "Python is fun!"
    print(cfg.get('Section1', 'foo', raw=True))  # -> "%(bar)s is %(baz)s!"
 
-   # The optional `vars` argument is a dict with members that will take
+   # The optional *vars* argument is a dict with members that will take
    # precedence in interpolation.
    print(cfg.get('Section1', 'foo', vars={'bar': 'Documentation',
                                              'baz': 'evil'}))
 
-   # The optional `fallback` argument can be used to provide a fallback value
+   # The optional *fallback* argument can be used to provide a fallback value
    print(cfg.get('Section1', 'foo'))
          # -> "Python is fun!"
 
diff --git a/Doc/library/email.header.rst b/Doc/library/email.header.rst
--- a/Doc/library/email.header.rst
+++ b/Doc/library/email.header.rst
@@ -141,11 +141,11 @@
       Returns an approximation of the :class:`Header` as a string, using an
       unlimited line length.  All pieces are converted to unicode using the
       specified encoding and joined together appropriately.  Any pieces with a
-      charset of `unknown-8bit` are decoded as `ASCII` using the `replace`
+      charset of ``'unknown-8bit'`` are decoded as ASCII using the ``'replace'``
       error handler.
 
       .. versionchanged:: 3.2
-         Added handling for the `unknown-8bit` charset.
+         Added handling for the ``'unknown-8bit'`` charset.
 
 
    .. method:: __eq__(other)
diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -10,7 +10,7 @@
 ===================  =================  ==================  ================  ====================
 ..                   ..                 Built-in Functions  ..                ..
 ===================  =================  ==================  ================  ====================
-:func:`abs`          :func:`dict`       :func:`help`        :func:`min`       :func:`setattr`
+:func:`abs`          |func-dict|_       :func:`help`        :func:`min`       :func:`setattr`
 :func:`all`          :func:`dir`        :func:`hex`         :func:`next`      :func:`slice`
 :func:`any`          :func:`divmod`     :func:`id`          :func:`object`    :func:`sorted`
 :func:`ascii`        :func:`enumerate`  :func:`input`       :func:`oct`       :func:`staticmethod`
@@ -19,13 +19,22 @@
 :func:`bytearray`    :func:`filter`     :func:`issubclass`  :func:`pow`       :func:`super`
 :func:`bytes`        :func:`float`      :func:`iter`        :func:`print`     :func:`tuple`
 :func:`callable`     :func:`format`     :func:`len`         :func:`property`  :func:`type`
-:func:`chr`          :func:`frozenset`  :func:`list`        :func:`range`     :func:`vars`
+:func:`chr`          |func-frozenset|_  :func:`list`        :func:`range`     :func:`vars`
 :func:`classmethod`  :func:`getattr`    :func:`locals`      :func:`repr`      :func:`zip`
 :func:`compile`      :func:`globals`    :func:`map`         :func:`reversed`  :func:`__import__`
 :func:`complex`      :func:`hasattr`    :func:`max`         :func:`round`
-:func:`delattr`      :func:`hash`       :func:`memoryview`  :func:`set`
+:func:`delattr`      :func:`hash`       |func-memoryview|_  |func-set|_
 ===================  =================  ==================  ================  ====================
 
+.. using :func:`dict` would create a link to another page, so local targets are
+   used, with replacement texts to make the output in the table consistent
+
+.. |func-dict| replace:: ``dict()``
+.. |func-frozenset| replace:: ``frozenset()``
+.. |func-memoryview| replace:: ``memoryview()``
+.. |func-set| replace:: ``set()``
+
+
 .. function:: abs(x)
 
    Return the absolute value of a number.  The argument may be an
@@ -74,11 +83,12 @@
 
 .. function:: bool([x])
 
-   Convert a value to a Boolean, using the standard truth testing procedure.  If
-   *x* is false or omitted, this returns :const:`False`; otherwise it returns
-   :const:`True`. :class:`bool` is also a class, which is a subclass of
-   :class:`int`. Class :class:`bool` cannot be subclassed further.  Its only
-   instances are :const:`False` and :const:`True`.
+   Convert a value to a Boolean, using the standard :ref:`truth testing
+   procedure <truth>`.  If *x* is false or omitted, this returns ``False``;
+   otherwise it returns ``True``. :class:`bool` is also a class, which is a
+   subclass of :class:`int` (see :ref:`typesnumeric`).  Class :class:`bool`
+   cannot be subclassed further.  Its only instances are ``False`` and
+   ``True`` (see :ref:`bltin-boolean-values`).
 
    .. index:: pair: Boolean; type
 
@@ -248,6 +258,7 @@
    example, ``delattr(x, 'foobar')`` is equivalent to ``del x.foobar``.
 
 
+.. _func-dict:
 .. function:: dict([arg])
    :noindex:
 
@@ -491,6 +502,7 @@
 
    The float type is described in :ref:`typesnumeric`.
 
+
 .. function:: format(value[, format_spec])
 
    .. index::
@@ -511,6 +523,8 @@
    :exc:`TypeError` exception is raised if the method is not found or if either
    the *format_spec* or the return value are not strings.
 
+
+.. _func-frozenset:
 .. function:: frozenset([iterable])
    :noindex:
 
@@ -717,6 +731,8 @@
    such as ``sorted(iterable, key=keyfunc, reverse=True)[0]`` and
    ``heapq.nlargest(1, iterable, key=keyfunc)``.
 
+
+.. _func-memoryview:
 .. function:: memoryview(obj)
    :noindex:
 
@@ -1040,7 +1056,7 @@
 
    Range objects implement the :class:`collections.Sequence` ABC, and provide
    features such as containment tests, element index lookup, slicing and
-   support for negative indices:
+   support for negative indices (see :ref:`typesseq`):
 
       >>> r = range(0, 20, 2)
       >>> r
@@ -1108,6 +1124,8 @@
       can't be represented exactly as a float.  See :ref:`tut-fp-issues` for
       more information.
 
+
+.. _func-set:
 .. function:: set([iterable])
    :noindex:
 
diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst
--- a/Doc/library/inspect.rst
+++ b/Doc/library/inspect.rst
@@ -575,13 +575,13 @@
 may be called.
 
 For cases where you want passive introspection, like documentation tools, this
-can be inconvenient. `getattr_static` has the same signature as :func:`getattr`
+can be inconvenient. :func:`getattr_static` has the same signature as :func:`getattr`
 but avoids executing code when it fetches attributes.
 
 .. function:: getattr_static(obj, attr, default=None)
 
    Retrieve attributes without triggering dynamic lookup via the
-   descriptor protocol, `__getattr__` or `__getattribute__`.
+   descriptor protocol, :meth:`__getattr__` or :meth:`__getattribute__`.
 
    Note: this function may not be able to retrieve all attributes
    that getattr can fetch (like dynamically created attributes)
@@ -589,12 +589,12 @@
    that raise AttributeError). It can also return descriptors objects
    instead of instance members.
 
-   If the instance `__dict__` is shadowed by another member (for example a
+   If the instance :attr:`__dict__` is shadowed by another member (for example a
    property) then this function will be unable to find instance members.
 
    .. versionadded:: 3.2
 
-`getattr_static` does not resolve descriptors, for example slot descriptors or
+:func:`getattr_static` does not resolve descriptors, for example slot descriptors or
 getset descriptors on objects implemented in C. The descriptor object
 is returned instead of the underlying attribute.
 
diff --git a/Doc/library/packaging.util.rst b/Doc/library/packaging.util.rst
--- a/Doc/library/packaging.util.rst
+++ b/Doc/library/packaging.util.rst
@@ -90,34 +90,6 @@
    Search the path for a given executable name.
 
 
-.. function:: subst_vars(s, local_vars)
-
-   Perform shell/Perl-style variable substitution on *s*.  Every occurrence of
-   ``$`` followed by a name is considered a variable, and variable is
-   substituted by the value found in the *local_vars* dictionary, or in
-   ``os.environ`` if it's not in *local_vars*. *os.environ* is first
-   checked/augmented to guarantee that it contains certain values: see
-   :func:`check_environ`.  Raise :exc:`ValueError` for any variables not found
-   in either *local_vars* or ``os.environ``.
-
-   Note that this is not a fully-fledged string interpolation function. A valid
-   ``$variable`` can consist only of upper and lower case letters, numbers and
-   an underscore. No { } or ( ) style quoting is available.
-
-
-.. function:: split_quoted(s)
-
-   Split a string up according to Unix shell-like rules for quotes and
-   backslashes. In short: words are delimited by spaces, as long as those spaces
-   are not escaped by a backslash, or inside a quoted string. Single and double
-   quotes are equivalent, and the quote characters can be backslash-escaped.
-   The backslash is stripped from any two-character escape sequence, leaving
-   only the escaped character.  The quote characters are stripped from any
-   quoted string.  Returns a list of words.
-
-   .. TODO Should probably be moved into the standard library.
-
-
 .. function:: execute(func, args[, msg=None, verbose=0, dry_run=0])
 
    Perform some action that affects the outside world (for instance, writing to
@@ -175,12 +147,3 @@
    figure out to use direct compilation or not (see the source for details).
    The *direct* flag is used by the script generated in indirect mode; unless
    you know what you're doing, leave it set to ``None``.
-
-
-.. function:: rfc822_escape(header)
-
-   Return a version of *header* escaped for inclusion in an :rfc:`822` header, by
-   ensuring there are 8 spaces space after each newline.  Note that it does no
-   other modification of the string.
-
-   .. TODO this _can_ be replaced
diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -2724,6 +2724,8 @@
 It is written as ``Ellipsis`` or ``...``.
 
 
+.. _bltin-notimplemented-object:
+
 The NotImplemented Object
 -------------------------
 
@@ -2735,6 +2737,8 @@
 It is written as ``NotImplemented``.
 
 
+.. _bltin-boolean-values:
+
 Boolean Values
 --------------
 
diff --git a/Doc/library/string.rst b/Doc/library/string.rst
--- a/Doc/library/string.rst
+++ b/Doc/library/string.rst
@@ -216,6 +216,8 @@
 it refers to a named keyword argument.  If the numerical arg_names in a format string
 are 0, 1, 2, ... in sequence, they can all be omitted (not just some)
 and the numbers 0, 1, 2, ... will be automatically inserted in that order.
+Because *arg_name* is not quote-delimited, it is not possible to specify arbitrary
+dictionary keys (e.g., the strings ``'10'`` or ``':-]'``) within a format string.
 The *arg_name* can be followed by any number of index or
 attribute expressions. An expression of the form ``'.name'`` selects the named
 attribute using :func:`getattr`, while an expression of the form ``'[index]'``
diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -293,7 +293,7 @@
 
    As a shortcut, ``python -m unittest`` is the equivalent of
    ``python -m unittest discover``. If you want to pass arguments to test
-   discovery the `discover` sub-command must be used explicitly.
+   discovery the ``discover`` sub-command must be used explicitly.
 
 The ``discover`` sub-command has the following options:
 
@@ -305,11 +305,11 @@
 
 .. cmdoption:: -s directory
 
-   Directory to start discovery ('.' default)
+   Directory to start discovery (``.`` default)
 
 .. cmdoption:: -p pattern
 
-   Pattern to match test files ('test*.py' default)
+   Pattern to match test files (``test*.py`` default)
 
 .. cmdoption:: -t directory
 
@@ -724,8 +724,8 @@
    single test.
 
    .. versionchanged:: 3.2
-      `TestCase` can be instantiated successfully without providing a method
-      name. This makes it easier to experiment with `TestCase` from the
+      :class:`TestCase` can be instantiated successfully without providing a method
+      name. This makes it easier to experiment with :class:`TestCase` from the
       interactive interpreter.
 
    *methodName* defaults to :meth:`runTest`.
@@ -944,17 +944,17 @@
    +---------------------------------------------------------+--------------------------------------+------------+
    | Method                                                  | Checks that                          | New in     |
    +=========================================================+======================================+============+
-   | :meth:`assertRaises(exc, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises `exc`  |            |
+   | :meth:`assertRaises(exc, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises *exc*  |            |
    | <TestCase.assertRaises>`                                |                                      |            |
    +---------------------------------------------------------+--------------------------------------+------------+
-   | :meth:`assertRaisesRegex(exc, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises `exc`  | 3.1        |
-   | <TestCase.assertRaisesRegex>`                           | and the message matches `re`         |            |
+   | :meth:`assertRaisesRegex(exc, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises *exc*  | 3.1        |
+   | <TestCase.assertRaisesRegex>`                           | and the message matches *re*         |            |
    +---------------------------------------------------------+--------------------------------------+------------+
-   | :meth:`assertWarns(warn, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises `warn` | 3.2        |
+   | :meth:`assertWarns(warn, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises *warn* | 3.2        |
    | <TestCase.assertWarns>`                                 |                                      |            |
    +---------------------------------------------------------+--------------------------------------+------------+
-   | :meth:`assertWarnsRegex(warn, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises `warn` | 3.2        |
-   | <TestCase.assertWarnsRegex>`                            | and the message matches `re`         |            |
+   | :meth:`assertWarnsRegex(warn, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises *warn* | 3.2        |
+   | <TestCase.assertWarnsRegex>`                            | and the message matches *re*         |            |
    +---------------------------------------------------------+--------------------------------------+------------+
 
    .. method:: assertRaises(exception, callable, *args, **kwds)
@@ -1116,7 +1116,7 @@
    | :meth:`assertNotRegex(s, re)          | ``not regex.search(s)``        | 3.2          |
    | <TestCase.assertNotRegex>`            |                                |              |
    +---------------------------------------+--------------------------------+--------------+
-   | :meth:`assertCountEqual(a, b)         | `a` and `b` have the same      | 3.2          |
+   | :meth:`assertCountEqual(a, b)         | *a* and *b* have the same      | 3.2          |
    | <TestCase.assertCountEqual>`          | elements in the same number,   |              |
    |                                       | regardless of their order      |              |
    +---------------------------------------+--------------------------------+--------------+
@@ -1911,7 +1911,7 @@
 .. class:: TextTestRunner(stream=None, descriptions=True, verbosity=1, runnerclass=None, warnings=None)
 
    A basic test runner implementation that outputs results to a stream. If *stream*
-   is `None`, the default, `sys.stderr` is used as the output stream. This class
+   is ``None``, the default, :data:`sys.stderr` is used as the output stream. This class
    has a few configurable parameters, but is essentially very simple.  Graphical
    applications which run test suites should provide alternate implementations.
 
@@ -1928,7 +1928,7 @@
       Added the ``warnings`` argument.
 
    .. versionchanged:: 3.2
-      The default stream is set to `sys.stderr` at instantiation time rather
+      The default stream is set to :data:`sys.stderr` at instantiation time rather
       than import time.
 
    .. method:: _makeResult()
diff --git a/Doc/packaging/setupcfg.rst b/Doc/packaging/setupcfg.rst
--- a/Doc/packaging/setupcfg.rst
+++ b/Doc/packaging/setupcfg.rst
@@ -756,8 +756,7 @@
    [files]
    packages = coconut
 
-   [extension=_fastcoconut]
-   name = coconut._fastcoconut
+   [extension: coconut._fastcoconut]
    language = cxx
    sources = cxx_src/cononut_utils.cxx
              cxx_src/python_module.cxx
@@ -768,8 +767,12 @@
        -DGECODE_VERSION=$(./gecode_version) -- sys.platform != 'win32'
        /DGECODE_VERSION='win32' -- sys.platform == 'win32'
 
-The section name must start with ``extension=``; the righ-hand part is currently
-discarded.  Valid fields and their values are listed in the documentation of the
+The section name must start with ``extension:``; the right-hand part is used as
+the full name (including a parent package, if any) of the extension.  Whitespace
+around the extension name is allowed.  If the extension module is not standalone
+(e.g. ``_bisect``) but part of a package (e.g. ``thing._speedups``), the parent
+package must be listed in the ``packages`` field.
+Valid fields and their values are listed in the documentation of the
 :class:`packaging.compiler.extension.Extension` class; values documented as
 Python lists translate to multi-line values in the configuration file.  In
 addition, multi-line values accept environment markers on each line, after a
diff --git a/Lib/compileall.py b/Lib/compileall.py
--- a/Lib/compileall.py
+++ b/Lib/compileall.py
@@ -142,7 +142,7 @@
 
     Arguments (all optional):
 
-    skip_curdir: if true, skip current directory (default true)
+    skip_curdir: if true, skip current directory (default True)
     maxlevels:   max recursion level (default 0)
     force: as for compile_dir() (default False)
     quiet: as for compile_dir() (default False)
@@ -177,17 +177,17 @@
                         help='use legacy (pre-PEP3147) compiled file locations')
     parser.add_argument('-d', metavar='DESTDIR',  dest='ddir', default=None,
                         help=('directory to prepend to file paths for use in '
-                              'compile time tracebacks and in runtime '
+                              'compile-time tracebacks and in runtime '
                               'tracebacks in cases where the source file is '
                               'unavailable'))
     parser.add_argument('-x', metavar='REGEXP', dest='rx', default=None,
-                        help=('skip files matching the regular expression. '
-                              'The regexp is searched for in the full path '
-                              'to each file considered for compilation.'))
+                        help=('skip files matching the regular expression; '
+                              'the regexp is searched for in the full path '
+                              'of each file considered for compilation'))
     parser.add_argument('-i', metavar='FILE', dest='flist',
                         help=('add all the files and directories listed in '
-                              'FILE to the list considered for compilation. '
-                              'If "-", names are read from stdin.'))
+                              'FILE to the list considered for compilation; '
+                              'if "-", names are read from stdin'))
     parser.add_argument('compile_dest', metavar='FILE|DIR', nargs='*',
                         help=('zero or more file and directory names '
                               'to compile; if no arguments given, defaults '
diff --git a/Lib/distutils/tests/support.py b/Lib/distutils/tests/support.py
--- a/Lib/distutils/tests/support.py
+++ b/Lib/distutils/tests/support.py
@@ -175,10 +175,9 @@
 def fixup_build_ext(cmd):
     """Function needed to make build_ext tests pass.
 
-    When Python was build with --enable-shared on Unix, -L. is not good
-    enough to find the libpython<blah>.so.  This is because regrtest runs
-    it under a tempdir, not in the top level where the .so lives.  By the
-    time we've gotten here, Python's already been chdir'd to the tempdir.
+    When Python was built with --enable-shared on Unix, -L. is not enough to
+    find libpython<blah>.so, because regrtest runs in a tempdir, not in the
+    source directory where the .so lives.
 
     When Python was built with in debug mode on Windows, build_ext commands
     need their debug attribute set, and it is not done automatically for
diff --git a/Lib/packaging/command/build_py.py b/Lib/packaging/command/build_py.py
--- a/Lib/packaging/command/build_py.py
+++ b/Lib/packaging/command/build_py.py
@@ -393,7 +393,7 @@
                            self.get_command_name())
             return
 
-        from packaging.util import byte_compile
+        from packaging.util import byte_compile  # FIXME use compileall
         prefix = self.build_lib
         if prefix[-1] != os.sep:
             prefix = prefix + os.sep
diff --git a/Lib/packaging/command/install_lib.py b/Lib/packaging/command/install_lib.py
--- a/Lib/packaging/command/install_lib.py
+++ b/Lib/packaging/command/install_lib.py
@@ -122,7 +122,7 @@
                            self.get_command_name())
             return
 
-        from packaging.util import byte_compile
+        from packaging.util import byte_compile  # FIXME use compileall
 
         # Get the "--root" directory supplied to the "install_dist" command,
         # and use it as a prefix to strip off the purported filename
diff --git a/Lib/packaging/command/register.py b/Lib/packaging/command/register.py
--- a/Lib/packaging/command/register.py
+++ b/Lib/packaging/command/register.py
@@ -2,7 +2,6 @@
 
 # Contributed by Richard Jones
 
-import io
 import getpass
 import urllib.error
 import urllib.parse
diff --git a/Lib/packaging/command/upload.py b/Lib/packaging/command/upload.py
--- a/Lib/packaging/command/upload.py
+++ b/Lib/packaging/command/upload.py
@@ -5,7 +5,6 @@
 import logging
 import platform
 import urllib.parse
-from io import BytesIO
 from base64 import standard_b64encode
 from hashlib import md5
 from urllib.error import HTTPError
diff --git a/Lib/packaging/config.py b/Lib/packaging/config.py
--- a/Lib/packaging/config.py
+++ b/Lib/packaging/config.py
@@ -16,6 +16,19 @@
 from packaging.markers import interpret
 
 
+def _check_name(name, packages):
+    if '.' not in name:
+        return
+    parts = name.split('.')
+    modname = parts[-1]
+    parent = '.'.join(parts[:-1])
+    if parent not in packages:
+        # we could log a warning instead of raising, but what's the use
+        # of letting people build modules they can't import?
+        raise PackagingOptionError(
+            'parent package for extension %r not found' % name)
+
+
 def _pop_values(values_dct, key):
     """Remove values from the dictionary and convert them as a list"""
     vals_str = values_dct.pop(key, '')
@@ -142,7 +155,8 @@
                         try:
                             hook = resolve_name(line)
                         except ImportError as e:
-                            logger.warning('cannot find setup hook: %s', e.args[0])
+                            logger.warning('cannot find setup hook: %s',
+                                           e.args[0])
                         else:
                             self.setup_hooks.append(hook)
                     self.run_hooks(content)
@@ -216,7 +230,7 @@
             for data in files.get('package_data', []):
                 data = data.split('=')
                 if len(data) != 2:
-                    continue  # XXX error should never pass silently
+                    continue  # FIXME errors should never pass silently
                 key, value = data
                 self.dist.package_data[key.strip()] = value.strip()
 
@@ -251,13 +265,18 @@
 
         ext_modules = self.dist.ext_modules
         for section_key in content:
-            labels = section_key.split('=')
+            # no str.partition in 2.4 :(
+            labels = section_key.split(':')
             if len(labels) == 2 and labels[0] == 'extension':
-                # labels[1] not used from now but should be implemented
-                # for extension build dependency
                 values_dct = content[section_key]
+                if 'name' in values_dct:
+                    raise PackagingOptionError(
+                        'extension name should be given as [extension: name], '
+                        'not as key')
+                name = labels[1].strip()
+                _check_name(name, self.dist.packages)
                 ext_modules.append(Extension(
-                    values_dct.pop('name'),
+                    name,
                     _pop_values(values_dct, 'sources'),
                     _pop_values(values_dct, 'include_dirs'),
                     _pop_values(values_dct, 'define_macros'),
diff --git a/Lib/packaging/create.py b/Lib/packaging/create.py
--- a/Lib/packaging/create.py
+++ b/Lib/packaging/create.py
@@ -36,7 +36,7 @@
 from packaging.version import is_valid_version
 
 _FILENAME = 'setup.cfg'
-_DEFAULT_CFG = '.pypkgcreate'
+_DEFAULT_CFG = '.pypkgcreate'  # FIXME use a section in user .pydistutils.cfg
 
 _helptext = {
     'name': '''
@@ -127,6 +127,10 @@
         print('\nERROR: You must select "Y" or "N".\n')
 
 
+# XXX use util.ask
+# FIXME: if prompt ends with '?', don't add ':'
+
+
 def ask(question, default=None, helptext=None, required=True,
         lengthy=False, multiline=False):
     prompt = '%s: ' % (question,)
diff --git a/Lib/packaging/tests/support.py b/Lib/packaging/tests/support.py
--- a/Lib/packaging/tests/support.py
+++ b/Lib/packaging/tests/support.py
@@ -1,5 +1,8 @@
 """Support code for packaging test cases.
 
+*This module should not be considered public: its content and API may
+change in incompatible ways.*
+
 A few helper classes are provided: LoggingCatcher, TempdirManager and
 EnvironRestorer. They are written to be used as mixins::
 
@@ -7,6 +10,7 @@
     from packaging.tests.support import LoggingCatcher
 
     class SomeTestCase(LoggingCatcher, unittest.TestCase):
+        ...
 
 If you need to define a setUp method on your test class, you have to
 call the mixin class' setUp method or it won't work (same thing for
@@ -14,17 +18,18 @@
 
         def setUp(self):
             super(SomeTestCase, self).setUp()
-            ... # other setup code
+            ...  # other setup code
 
 Also provided is a DummyCommand class, useful to mock commands in the
-tests of another command that needs them, a create_distribution function
-and a skip_unless_symlink decorator.
+tests of another command that needs them, for example to fake
+compilation in build_ext (this requires that the mock build_ext command
+be injected into the distribution object's command_obj dictionary).
 
-Also provided is a DummyCommand class, useful to mock commands in the
-tests of another command that needs them, a create_distribution function
-and a skip_unless_symlink decorator.
+For tests that need to compile an extension module, use the
+copy_xxmodule_c and fixup_build_ext functions.
 
 Each class or function has a docstring to explain its purpose and usage.
+Existing tests should also be used as examples.
 """
 
 import os
@@ -39,9 +44,17 @@
 from packaging.tests import unittest
 from test.support import requires_zlib, unlink
 
-__all__ = ['LoggingCatcher', 'TempdirManager', 'EnvironRestorer',
-           'DummyCommand', 'unittest', 'create_distribution',
-           'skip_unless_symlink', 'requires_zlib', 'copy_xxmodule_c']
+# define __all__ to make pydoc more useful
+__all__ = [
+    # TestCase mixins
+    'LoggingCatcher', 'TempdirManager', 'EnvironRestorer',
+    # mocks
+    'DummyCommand', 'TestDistribution',
+    # misc. functions and decorators
+    'fake_dec', 'create_distribution', 'copy_xxmodule_c', 'fixup_build_ext',
+    # imported from this module for backport purposes
+    'unittest', 'requires_zlib', 'skip_unless_symlink',
+]
 
 
 logger = logging.getLogger('packaging')
@@ -233,6 +246,8 @@
     Useful for mocking one dependency command in the tests for another
     command, see e.g. the dummy build command in test_build_scripts.
     """
+    # XXX does not work with dist.get_reinitialized_command, which typechecks
+    # and wants a finalized attribute
 
     def __init__(self, **kwargs):
         for kw, val in kwargs.items():
@@ -308,10 +323,9 @@
 def fixup_build_ext(cmd):
     """Function needed to make build_ext tests pass.
 
-    When Python was build with --enable-shared on Unix, -L. is not good
-    enough to find the libpython<blah>.so.  This is because regrtest runs
-    it under a tempdir, not in the top level where the .so lives.  By the
-    time we've gotten here, Python's already been chdir'd to the tempdir.
+    When Python was built with --enable-shared on Unix, -L. is not enough to
+    find libpython<blah>.so, because regrtest runs in a tempdir, not in the
+    source directory where the .so lives.
 
     When Python was built with in debug mode on Windows, build_ext commands
     need their debug attribute set, and it is not done automatically for
diff --git a/Lib/packaging/tests/test_config.py b/Lib/packaging/tests/test_config.py
--- a/Lib/packaging/tests/test_config.py
+++ b/Lib/packaging/tests/test_config.py
@@ -6,7 +6,7 @@
 
 from packaging import command
 from packaging.dist import Distribution
-from packaging.errors import PackagingFileError
+from packaging.errors import PackagingFileError, PackagingOptionError
 from packaging.compiler import new_compiler, _COMPILERS
 from packaging.command.sdist import sdist
 
@@ -100,21 +100,21 @@
 
 # Can not be merged with SETUP_CFG else install_dist
 # command will fail when trying to compile C sources
+# TODO use a DummyCommand to mock build_ext
 EXT_SETUP_CFG = """
 [files]
 packages = one
            two
+           parent.undeclared
 
-[extension=speed_coconuts]
-name = one.speed_coconuts
+[extension:one.speed_coconuts]
 sources = c_src/speed_coconuts.c
 extra_link_args = "`gcc -print-file-name=libgcc.a`" -shared
 define_macros = HAVE_CAIRO HAVE_GTK2
 libraries = gecodeint gecodekernel -- sys.platform != 'win32'
     GecodeInt GecodeKernel -- sys.platform == 'win32'
 
-[extension=fast_taunt]
-name = two.fast_taunt
+[extension: two.fast_taunt]
 sources = cxx_src/utils_taunt.cxx
           cxx_src/python_module.cxx
 include_dirs = /usr/include/gecode
@@ -124,6 +124,30 @@
     /DGECODE_VERSION='win32' -- sys.platform == 'win32'
 language = cxx
 
+# corner case: if the parent package of an extension is declared but
+# not its grandparent, it's legal
+[extension: parent.undeclared._speed]
+sources = parent/undeclared/_speed.c
+"""
+
+EXT_SETUP_CFG_BUGGY_1 = """
+[extension: realname]
+name = crash_here
+"""
+
+EXT_SETUP_CFG_BUGGY_2 = """
+[files]
+packages = ham
+
+[extension: spam.eggs]
+"""
+
+EXT_SETUP_CFG_BUGGY_3 = """
+[files]
+packages = ok
+           ok.works
+
+[extension: ok.works.breaks._ext]
 """
 
 HOOKS_MODULE = """
@@ -311,7 +335,7 @@
         dist = self.get_dist()
 
         ext_modules = dict((mod.name, mod) for mod in dist.ext_modules)
-        self.assertEqual(len(ext_modules), 2)
+        self.assertEqual(len(ext_modules), 3)
         ext = ext_modules.get('one.speed_coconuts')
         self.assertEqual(ext.sources, ['c_src/speed_coconuts.c'])
         self.assertEqual(ext.define_macros, ['HAVE_CAIRO', 'HAVE_GTK2'])
@@ -335,6 +359,15 @@
         self.assertEqual(ext.extra_compile_args, cargs)
         self.assertEqual(ext.language, 'cxx')
 
+        self.write_file('setup.cfg', EXT_SETUP_CFG_BUGGY_1)
+        self.assertRaises(PackagingOptionError, self.get_dist)
+
+        self.write_file('setup.cfg', EXT_SETUP_CFG_BUGGY_2)
+        self.assertRaises(PackagingOptionError, self.get_dist)
+
+        self.write_file('setup.cfg', EXT_SETUP_CFG_BUGGY_3)
+        self.assertRaises(PackagingOptionError, self.get_dist)
+
     def test_project_setup_hook_works(self):
         # Bug #11637: ensure the project directory is on sys.path to allow
         # project-specific hooks
@@ -358,7 +391,7 @@
         self.write_setup({
             'setup-hooks': '\n  packaging.tests.test_config.first_hook'
                            '\n  packaging.tests.test_config.missing_hook'
-                           '\n  packaging.tests.test_config.third_hook'
+                           '\n  packaging.tests.test_config.third_hook',
         })
         self.write_file('README', 'yeah')
         dist = self.get_dist()
diff --git a/Lib/packaging/tests/test_util.py b/Lib/packaging/tests/test_util.py
--- a/Lib/packaging/tests/test_util.py
+++ b/Lib/packaging/tests/test_util.py
@@ -15,7 +15,7 @@
 from packaging import util
 from packaging.dist import Distribution
 from packaging.util import (
-    convert_path, change_root, split_quoted, strtobool, rfc822_escape,
+    convert_path, change_root, split_quoted, strtobool,
     get_compiler_versions, _MAC_OS_X_LD_VERSION, byte_compile, find_packages,
     spawn, get_pypirc_path, generate_pypirc, read_pypirc, resolve_name, iglob,
     RICH_GLOB, egginfo_to_distinfo, is_setuptools, is_distutils, is_packaging,
@@ -255,13 +255,6 @@
         for n in no:
             self.assertFalse(strtobool(n))
 
-    def test_rfc822_escape(self):
-        header = 'I am a\npoor\nlonesome\nheader\n'
-        res = rfc822_escape(header)
-        wanted = ('I am a%(8s)spoor%(8s)slonesome%(8s)s'
-                  'header%(8s)s') % {'8s': '\n' + 8 * ' '}
-        self.assertEqual(res, wanted)
-
     def test_find_exe_version(self):
         # the ld version scheme under MAC OS is:
         #   ^@(#)PROGRAM:ld  PROJECT:ld64-VERSION
diff --git a/Lib/packaging/util.py b/Lib/packaging/util.py
--- a/Lib/packaging/util.py
+++ b/Lib/packaging/util.py
@@ -8,8 +8,6 @@
 import shutil
 import string
 import hashlib
-import tarfile
-import zipfile
 import posixpath
 import subprocess
 import sysconfig
@@ -23,6 +21,30 @@
                               PackagingByteCompileError, PackagingExecError,
                               InstallationException, PackagingInternalError)
 
+__all__ = [
+    # file dependencies
+    'newer', 'newer_group',
+    # helpers for commands (dry-run system)
+    'execute', 'write_file',
+    # spawning programs
+    'find_executable', 'spawn',
+    # path manipulation
+    'convert_path', 'change_root',
+    # 2to3 conversion
+    'Mixin2to3', 'run_2to3',
+    # packaging compatibility helpers
+    'cfg_to_args', 'generate_setup_py',
+    'egginfo_to_distinfo',
+    'get_install_method',
+    # misc
+    'ask', 'check_environ', 'encode_multipart', 'resolve_name',
+    # querying for information  TODO move to sysconfig
+    'get_compiler_versions', 'get_platform', 'set_platform',
+    # configuration  TODO move to packaging.config
+    'get_pypirc_path', 'read_pypirc', 'generate_pypirc',
+    'strtobool', 'split_multiline',
+]
+
 _PLATFORM = None
 _DEFAULT_INSTALLER = 'packaging'
 
@@ -152,31 +174,6 @@
     _environ_checked = True
 
 
-def subst_vars(s, local_vars):
-    """Perform shell/Perl-style variable substitution on 'string'.
-
-    Every occurrence of '$' followed by a name is considered a variable, and
-    variable is substituted by the value found in the 'local_vars'
-    dictionary, or in 'os.environ' if it's not in 'local_vars'.
-    'os.environ' is first checked/augmented to guarantee that it contains
-    certain values: see 'check_environ()'.  Raise ValueError for any
-    variables not found in either 'local_vars' or 'os.environ'.
-    """
-    check_environ()
-
-    def _subst(match, local_vars=local_vars):
-        var_name = match.group(1)
-        if var_name in local_vars:
-            return str(local_vars[var_name])
-        else:
-            return os.environ[var_name]
-
-    try:
-        return re.sub(r'\$([a-zA-Z_][a-zA-Z_0-9]*)', _subst, s)
-    except KeyError as var:
-        raise ValueError("invalid variable '$%s'" % var)
-
-
 # Needed by 'split_quoted()'
 _wordchars_re = _squote_re = _dquote_re = None
 
@@ -188,6 +185,8 @@
     _dquote_re = re.compile(r'"(?:[^"\\]|\\.)*"')
 
 
+# TODO replace with shlex.split after testing
+
 def split_quoted(s):
     """Split a string up according to Unix shell-like rules for quotes and
     backslashes.
@@ -435,15 +434,6 @@
                               file, cfile_base)
 
 
-def rfc822_escape(header):
-    """Return a form of *header* suitable for inclusion in an RFC 822-header.
-
-    This function ensures there are 8 spaces after each newline.
-    """
-    lines = header.split('\n')
-    sep = '\n' + 8 * ' '
-    return sep.join(lines)
-
 _RE_VERSION = re.compile('(\d+\.\d+(\.\d+)*)')
 _MAC_OS_X_LD_VERSION = re.compile('^@\(#\)PROGRAM:ld  '
                                   'PROJECT:ld64-((\d+)(\.\d+)*)')
@@ -543,6 +533,10 @@
     """Create *filename* and write *contents* to it.
 
     *contents* is a sequence of strings without line terminators.
+
+    This functions is not intended to replace the usual with open + write
+    idiom in all cases, only with Command.execute, which runs depending on
+    the dry_run argument and also logs its arguments).
     """
     with open(filename, "w") as f:
         for line in contents:
@@ -562,6 +556,7 @@
 
 
 def _under(path, root):
+    # XXX use os.path
     path = path.split(os.sep)
     root = root.split(os.sep)
     if len(root) > len(path):
@@ -664,103 +659,11 @@
     return base, ext
 
 
-def unzip_file(filename, location, flatten=True):
-    """Unzip the file *filename* into the *location* directory."""
-    if not os.path.exists(location):
-        os.makedirs(location)
-    with open(filename, 'rb') as zipfp:
-        zip = zipfile.ZipFile(zipfp)
-        leading = has_leading_dir(zip.namelist()) and flatten
-        for name in zip.namelist():
-            data = zip.read(name)
-            fn = name
-            if leading:
-                fn = split_leading_dir(name)[1]
-            fn = os.path.join(location, fn)
-            dir = os.path.dirname(fn)
-            if not os.path.exists(dir):
-                os.makedirs(dir)
-            if fn.endswith('/') or fn.endswith('\\'):
-                # A directory
-                if not os.path.exists(fn):
-                    os.makedirs(fn)
-            else:
-                with open(fn, 'wb') as fp:
-                    fp.write(data)
-
-
-def untar_file(filename, location):
-    """Untar the file *filename* into the *location* directory."""
-    if not os.path.exists(location):
-        os.makedirs(location)
-    if filename.lower().endswith('.gz') or filename.lower().endswith('.tgz'):
-        mode = 'r:gz'
-    elif (filename.lower().endswith('.bz2')
-          or filename.lower().endswith('.tbz')):
-        mode = 'r:bz2'
-    elif filename.lower().endswith('.tar'):
-        mode = 'r'
-    else:
-        mode = 'r:*'
-    with tarfile.open(filename, mode) as tar:
-        leading = has_leading_dir(member.name for member in tar.getmembers())
-        for member in tar.getmembers():
-            fn = member.name
-            if leading:
-                fn = split_leading_dir(fn)[1]
-            path = os.path.join(location, fn)
-            if member.isdir():
-                if not os.path.exists(path):
-                    os.makedirs(path)
-            else:
-                try:
-                    fp = tar.extractfile(member)
-                except (KeyError, AttributeError):
-                    # Some corrupt tar files seem to produce this
-                    # (specifically bad symlinks)
-                    continue
-                try:
-                    if not os.path.exists(os.path.dirname(path)):
-                        os.makedirs(os.path.dirname(path))
-                        with open(path, 'wb') as destfp:
-                            shutil.copyfileobj(fp, destfp)
-                finally:
-                    fp.close()
-
-
-def has_leading_dir(paths):
-    """Return true if all the paths have the same leading path name.
-
-    In other words, check that everything is in one subdirectory in an
-    archive.
-    """
-    common_prefix = None
-    for path in paths:
-        prefix, rest = split_leading_dir(path)
-        if not prefix:
-            return False
-        elif common_prefix is None:
-            common_prefix = prefix
-        elif prefix != common_prefix:
-            return False
-    return True
-
-
-def split_leading_dir(path):
-    path = str(path)
-    path = path.lstrip('/').lstrip('\\')
-    if '/' in path and (('\\' in path and path.find('/') < path.find('\\'))
-                        or '\\' not in path):
-        return path.split('/', 1)
-    elif '\\' in path:
-        return path.split('\\', 1)
-    else:
-        return path, ''
-
 if sys.platform == 'darwin':
     _cfg_target = None
     _cfg_target_split = None
 
+
 def spawn(cmd, search_path=True, verbose=0, dry_run=False, env=None):
     """Run another program specified as a command list 'cmd' in a new process.
 
@@ -1510,7 +1413,7 @@
     for key, values in fields:
         # handle multiple entries for the same name
         if not isinstance(values, (tuple, list)):
-            values=[values]
+            values = [values]
 
         for value in values:
             l.extend((
diff --git a/Lib/pipes.py b/Lib/pipes.py
--- a/Lib/pipes.py
+++ b/Lib/pipes.py
@@ -54,8 +54,6 @@
 
 To create a new template object initialized to a given one:
    t2 = t.clone()
-
-For an example, see the function test() at the end of the file.
 """                                     # '
 
 
diff --git a/Lib/shutil.py b/Lib/shutil.py
--- a/Lib/shutil.py
+++ b/Lib/shutil.py
@@ -35,7 +35,7 @@
            "register_archive_format", "unregister_archive_format",
            "get_unpack_formats", "register_unpack_format",
            "unregister_unpack_format", "unpack_archive",
-           "ignore_patterns"]
+           "ignore_patterns", "chown"]
            # disk_usage is added later, if available on the platform
 
 class Error(EnvironmentError):
@@ -791,6 +791,7 @@
         used = total - free
         return _ntuple_diskusage(total, used, free)
 
+
 def chown(path, user=None, group=None):
     """Change owner user and group of the given path.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:27 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:27 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMik6?=
	=?utf8?q?_Branch_merge?=
Message-ID: <E1QzVvr-0003BB-P6@dinsdale.python.org>

http://hg.python.org/cpython/rev/83c16d258fec
changeset:   72194:83c16d258fec
branch:      3.2
parent:      72173:fe7777f1ed14
parent:      72191:941106bcd327
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Sep 02 17:30:36 2011 +0200
summary:
  Branch merge

files:
  Doc/c-api/init.rst             |   4 +-
  Doc/documenting/markup.rst     |   5 ++-
  Doc/faq/design.rst             |   2 +-
  Doc/faq/programming.rst        |   9 -----
  Doc/faq/windows.rst            |   4 +-
  Doc/glossary.rst               |   2 +-
  Doc/howto/logging.rst          |   6 +-
  Doc/library/argparse.rst       |   2 +-
  Doc/library/base64.rst         |   4 +-
  Doc/library/configparser.rst   |   6 +-
  Doc/library/email.header.rst   |   4 +-
  Doc/library/functions.rst      |  36 ++++++++++++++++-----
  Doc/library/inspect.rst        |   8 ++--
  Doc/library/stdtypes.rst       |   4 ++
  Doc/library/string.rst         |   2 +
  Doc/library/unittest.rst       |  30 +++++++++---------
  Lib/compileall.py              |  14 ++++----
  Lib/distutils/tests/support.py |   7 +--
  Lib/pipes.py                   |   2 -
  19 files changed, 83 insertions(+), 68 deletions(-)


diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -122,7 +122,7 @@
    program name is ``'/usr/local/bin/python'``, the prefix is ``'/usr/local'``. The
    returned string points into static storage; the caller should not modify its
    value.  This corresponds to the :makevar:`prefix` variable in the top-level
-   :file:`Makefile` and the :option:`--prefix` argument to the :program:`configure`
+   :file:`Makefile` and the ``--prefix`` argument to the :program:`configure`
    script at build time.  The value is available to Python code as ``sys.prefix``.
    It is only useful on Unix.  See also the next function.
 
@@ -135,7 +135,7 @@
    program name is ``'/usr/local/bin/python'``, the exec-prefix is
    ``'/usr/local'``.  The returned string points into static storage; the caller
    should not modify its value.  This corresponds to the :makevar:`exec_prefix`
-   variable in the top-level :file:`Makefile` and the :option:`--exec-prefix`
+   variable in the top-level :file:`Makefile` and the ``--exec-prefix``
    argument to the :program:`configure` script at build  time.  The value is
    available to Python code as ``sys.exec_prefix``.  It is only useful on Unix.
 
diff --git a/Doc/documenting/markup.rst b/Doc/documenting/markup.rst
--- a/Doc/documenting/markup.rst
+++ b/Doc/documenting/markup.rst
@@ -513,7 +513,10 @@
 
 .. describe:: keyword
 
-   The name of a keyword in Python.
+   The name of a Python keyword.  Using this role will generate a link to the
+   documentation of the keyword.  ``True``, ``False`` and ``None`` do not use
+   this role, but simple code markup (````True````), given that they're
+   fundamental to the language and should be known to any programmer.
 
 .. describe:: mailheader
 
diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst
--- a/Doc/faq/design.rst
+++ b/Doc/faq/design.rst
@@ -667,7 +667,7 @@
 Python 2.6 adds an :mod:`abc` module that lets you define Abstract Base Classes
 (ABCs).  You can then use :func:`isinstance` and :func:`issubclass` to check
 whether an instance or a class implements a particular ABC.  The
-:mod:`collections` modules defines a set of useful ABCs such as
+:mod:`collections` module defines a set of useful ABCs such as
 :class:`Iterable`, :class:`Container`, and :class:`MutableMapping`.
 
 For Python, many of the advantages of interface specifications can be obtained
diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst
--- a/Doc/faq/programming.rst
+++ b/Doc/faq/programming.rst
@@ -473,15 +473,6 @@
        ...
        g(x, *args, **kwargs)
 
-In the unlikely case that you care about Python versions older than 2.0, use
-:func:`apply`::
-
-   def f(x, *args, **kwargs):
-       ...
-       kwargs['width'] = '14.3c'
-       ...
-       apply(g, (x,)+args, kwargs)
-
 
 How do I write a function with output parameters (call by reference)?
 ---------------------------------------------------------------------
diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst
--- a/Doc/faq/windows.rst
+++ b/Doc/faq/windows.rst
@@ -543,10 +543,10 @@
 
 If you can't change compilers or flags, try using :c:func:`Py_RunSimpleString`.
 A trick to get it to run an arbitrary file is to construct a call to
-:func:`execfile` with the name of your file as argument.
+:func:`exec` and :func:`open` with the name of your file as argument.
 
 Also note that you can not mix-and-match Debug and Release versions.  If you
-wish to use the Debug Multithreaded DLL, then your module *must* have an "_d"
+wish to use the Debug Multithreaded DLL, then your module *must* have ``_d``
 appended to the base name.
 
 
diff --git a/Doc/glossary.rst b/Doc/glossary.rst
--- a/Doc/glossary.rst
+++ b/Doc/glossary.rst
@@ -492,7 +492,7 @@
       :func:`builtins.open` and :func:`os.open` are distinguished by their
       namespaces.  Namespaces also aid readability and maintainability by making
       it clear which module implements a function.  For instance, writing
-      :func:`random.seed` or :func:`itertools.izip` makes it clear that those
+      :func:`random.seed` or :func:`itertools.islice` makes it clear that those
       functions are implemented by the :mod:`random` and :mod:`itertools`
       modules, respectively.
 
diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst
--- a/Doc/howto/logging.rst
+++ b/Doc/howto/logging.rst
@@ -412,10 +412,10 @@
   :meth:`Logger.error`, and :meth:`Logger.critical` all create log records with
   a message and a level that corresponds to their respective method names. The
   message is actually a format string, which may contain the standard string
-  substitution syntax of :const:`%s`, :const:`%d`, :const:`%f`, and so on.  The
+  substitution syntax of ``%s``, ``%d``, ``%f``, and so on.  The
   rest of their arguments is a list of objects that correspond with the
-  substitution fields in the message.  With regard to :const:`**kwargs`, the
-  logging methods care only about a keyword of :const:`exc_info` and use it to
+  substitution fields in the message.  With regard to ``**kwargs``, the
+  logging methods care only about a keyword of ``exc_info`` and use it to
   determine whether to log exception information.
 
 * :meth:`Logger.exception` creates a log message similar to
diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst
--- a/Doc/library/argparse.rst
+++ b/Doc/library/argparse.rst
@@ -155,7 +155,7 @@
      conflicting optionals.
 
    * prog_ - The name of the program (default:
-     :data:`sys.argv[0]`)
+     ``sys.argv[0]``)
 
    * usage_ - The string describing the program usage (default: generated)
 
diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst
--- a/Doc/library/base64.rst
+++ b/Doc/library/base64.rst
@@ -45,8 +45,8 @@
    at least length 2 (additional characters are ignored) which specifies the
    alternative alphabet used instead of the ``+`` and ``/`` characters.
 
-   The decoded string is returned.  A `binascii.Error` is raised if *s* is
-   incorrectly padded.
+   The decoded string is returned.  A :exc:`binascii.Error` exception is raised
+   if *s* is incorrectly padded.
 
    If *validate* is ``False`` (the default), non-base64-alphabet characters are
    discarded prior to the padding check.  If *validate* is ``True``,
diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst
--- a/Doc/library/configparser.rst
+++ b/Doc/library/configparser.rst
@@ -806,17 +806,17 @@
    cfg = configparser.ConfigParser()
    cfg.read('example.cfg')
 
-   # Set the optional `raw` argument of get() to True if you wish to disable
+   # Set the optional *raw* argument of get() to True if you wish to disable
    # interpolation in a single get operation.
    print(cfg.get('Section1', 'foo', raw=False)) # -> "Python is fun!"
    print(cfg.get('Section1', 'foo', raw=True))  # -> "%(bar)s is %(baz)s!"
 
-   # The optional `vars` argument is a dict with members that will take
+   # The optional *vars* argument is a dict with members that will take
    # precedence in interpolation.
    print(cfg.get('Section1', 'foo', vars={'bar': 'Documentation',
                                              'baz': 'evil'}))
 
-   # The optional `fallback` argument can be used to provide a fallback value
+   # The optional *fallback* argument can be used to provide a fallback value
    print(cfg.get('Section1', 'foo'))
          # -> "Python is fun!"
 
diff --git a/Doc/library/email.header.rst b/Doc/library/email.header.rst
--- a/Doc/library/email.header.rst
+++ b/Doc/library/email.header.rst
@@ -141,11 +141,11 @@
       Returns an approximation of the :class:`Header` as a string, using an
       unlimited line length.  All pieces are converted to unicode using the
       specified encoding and joined together appropriately.  Any pieces with a
-      charset of `unknown-8bit` are decoded as `ASCII` using the `replace`
+      charset of ``'unknown-8bit'`` are decoded as ASCII using the ``'replace'``
       error handler.
 
       .. versionchanged:: 3.2
-         Added handling for the `unknown-8bit` charset.
+         Added handling for the ``'unknown-8bit'`` charset.
 
 
    .. method:: __eq__(other)
diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -10,7 +10,7 @@
 ===================  =================  ==================  ================  ====================
 ..                   ..                 Built-in Functions  ..                ..
 ===================  =================  ==================  ================  ====================
-:func:`abs`          :func:`dict`       :func:`help`        :func:`min`       :func:`setattr`
+:func:`abs`          |func-dict|_       :func:`help`        :func:`min`       :func:`setattr`
 :func:`all`          :func:`dir`        :func:`hex`         :func:`next`      :func:`slice`
 :func:`any`          :func:`divmod`     :func:`id`          :func:`object`    :func:`sorted`
 :func:`ascii`        :func:`enumerate`  :func:`input`       :func:`oct`       :func:`staticmethod`
@@ -19,13 +19,22 @@
 :func:`bytearray`    :func:`filter`     :func:`issubclass`  :func:`pow`       :func:`super`
 :func:`bytes`        :func:`float`      :func:`iter`        :func:`print`     :func:`tuple`
 :func:`callable`     :func:`format`     :func:`len`         :func:`property`  :func:`type`
-:func:`chr`          :func:`frozenset`  :func:`list`        :func:`range`     :func:`vars`
+:func:`chr`          |func-frozenset|_  :func:`list`        :func:`range`     :func:`vars`
 :func:`classmethod`  :func:`getattr`    :func:`locals`      :func:`repr`      :func:`zip`
 :func:`compile`      :func:`globals`    :func:`map`         :func:`reversed`  :func:`__import__`
 :func:`complex`      :func:`hasattr`    :func:`max`         :func:`round`
-:func:`delattr`      :func:`hash`       :func:`memoryview`  :func:`set`
+:func:`delattr`      :func:`hash`       |func-memoryview|_  |func-set|_
 ===================  =================  ==================  ================  ====================
 
+.. using :func:`dict` would create a link to another page, so local targets are
+   used, with replacement texts to make the output in the table consistent
+
+.. |func-dict| replace:: ``dict()``
+.. |func-frozenset| replace:: ``frozenset()``
+.. |func-memoryview| replace:: ``memoryview()``
+.. |func-set| replace:: ``set()``
+
+
 .. function:: abs(x)
 
    Return the absolute value of a number.  The argument may be an
@@ -74,11 +83,12 @@
 
 .. function:: bool([x])
 
-   Convert a value to a Boolean, using the standard truth testing procedure.  If
-   *x* is false or omitted, this returns :const:`False`; otherwise it returns
-   :const:`True`. :class:`bool` is also a class, which is a subclass of
-   :class:`int`. Class :class:`bool` cannot be subclassed further.  Its only
-   instances are :const:`False` and :const:`True`.
+   Convert a value to a Boolean, using the standard :ref:`truth testing
+   procedure <truth>`.  If *x* is false or omitted, this returns ``False``;
+   otherwise it returns ``True``. :class:`bool` is also a class, which is a
+   subclass of :class:`int` (see :ref:`typesnumeric`).  Class :class:`bool`
+   cannot be subclassed further.  Its only instances are ``False`` and
+   ``True`` (see :ref:`bltin-boolean-values`).
 
    .. index:: pair: Boolean; type
 
@@ -248,6 +258,7 @@
    example, ``delattr(x, 'foobar')`` is equivalent to ``del x.foobar``.
 
 
+.. _func-dict:
 .. function:: dict([arg])
    :noindex:
 
@@ -491,6 +502,7 @@
 
    The float type is described in :ref:`typesnumeric`.
 
+
 .. function:: format(value[, format_spec])
 
    .. index::
@@ -511,6 +523,8 @@
    :exc:`TypeError` exception is raised if the method is not found or if either
    the *format_spec* or the return value are not strings.
 
+
+.. _func-frozenset:
 .. function:: frozenset([iterable])
    :noindex:
 
@@ -717,6 +731,8 @@
    such as ``sorted(iterable, key=keyfunc, reverse=True)[0]`` and
    ``heapq.nlargest(1, iterable, key=keyfunc)``.
 
+
+.. _func-memoryview:
 .. function:: memoryview(obj)
    :noindex:
 
@@ -1040,7 +1056,7 @@
 
    Range objects implement the :class:`collections.Sequence` ABC, and provide
    features such as containment tests, element index lookup, slicing and
-   support for negative indices:
+   support for negative indices (see :ref:`typesseq`):
 
       >>> r = range(0, 20, 2)
       >>> r
@@ -1108,6 +1124,8 @@
       can't be represented exactly as a float.  See :ref:`tut-fp-issues` for
       more information.
 
+
+.. _func-set:
 .. function:: set([iterable])
    :noindex:
 
diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst
--- a/Doc/library/inspect.rst
+++ b/Doc/library/inspect.rst
@@ -575,13 +575,13 @@
 may be called.
 
 For cases where you want passive introspection, like documentation tools, this
-can be inconvenient. `getattr_static` has the same signature as :func:`getattr`
+can be inconvenient. :func:`getattr_static` has the same signature as :func:`getattr`
 but avoids executing code when it fetches attributes.
 
 .. function:: getattr_static(obj, attr, default=None)
 
    Retrieve attributes without triggering dynamic lookup via the
-   descriptor protocol, `__getattr__` or `__getattribute__`.
+   descriptor protocol, :meth:`__getattr__` or :meth:`__getattribute__`.
 
    Note: this function may not be able to retrieve all attributes
    that getattr can fetch (like dynamically created attributes)
@@ -589,12 +589,12 @@
    that raise AttributeError). It can also return descriptors objects
    instead of instance members.
 
-   If the instance `__dict__` is shadowed by another member (for example a
+   If the instance :attr:`__dict__` is shadowed by another member (for example a
    property) then this function will be unable to find instance members.
 
    .. versionadded:: 3.2
 
-`getattr_static` does not resolve descriptors, for example slot descriptors or
+:func:`getattr_static` does not resolve descriptors, for example slot descriptors or
 getset descriptors on objects implemented in C. The descriptor object
 is returned instead of the underlying attribute.
 
diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -2712,6 +2712,8 @@
 It is written as ``Ellipsis`` or ``...``.
 
 
+.. _bltin-notimplemented-object:
+
 The NotImplemented Object
 -------------------------
 
@@ -2722,6 +2724,8 @@
 It is written as ``NotImplemented``.
 
 
+.. _bltin-boolean-values:
+
 Boolean Values
 --------------
 
diff --git a/Doc/library/string.rst b/Doc/library/string.rst
--- a/Doc/library/string.rst
+++ b/Doc/library/string.rst
@@ -216,6 +216,8 @@
 it refers to a named keyword argument.  If the numerical arg_names in a format string
 are 0, 1, 2, ... in sequence, they can all be omitted (not just some)
 and the numbers 0, 1, 2, ... will be automatically inserted in that order.
+Because *arg_name* is not quote-delimited, it is not possible to specify arbitrary
+dictionary keys (e.g., the strings ``'10'`` or ``':-]'``) within a format string.
 The *arg_name* can be followed by any number of index or
 attribute expressions. An expression of the form ``'.name'`` selects the named
 attribute using :func:`getattr`, while an expression of the form ``'[index]'``
diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -293,7 +293,7 @@
 
    As a shortcut, ``python -m unittest`` is the equivalent of
    ``python -m unittest discover``. If you want to pass arguments to test
-   discovery the `discover` sub-command must be used explicitly.
+   discovery the ``discover`` sub-command must be used explicitly.
 
 The ``discover`` sub-command has the following options:
 
@@ -305,11 +305,11 @@
 
 .. cmdoption:: -s directory
 
-   Directory to start discovery ('.' default)
+   Directory to start discovery (``.`` default)
 
 .. cmdoption:: -p pattern
 
-   Pattern to match test files ('test*.py' default)
+   Pattern to match test files (``test*.py`` default)
 
 .. cmdoption:: -t directory
 
@@ -723,9 +723,9 @@
    Here, we create two instances of :class:`WidgetTestCase`, each of which runs a
    single test.
 
-   .. versionchanged::
-      `TestCase` can be instantiated successfully without providing a method
-      name. This makes it easier to experiment with `TestCase` from the
+   .. versionchanged:: 3.2
+      :class:`TestCase` can be instantiated successfully without providing a method
+      name. This makes it easier to experiment with :class:`TestCase` from the
       interactive interpreter.
 
    *methodName* defaults to :meth:`runTest`.
@@ -940,17 +940,17 @@
    +---------------------------------------------------------+--------------------------------------+------------+
    | Method                                                  | Checks that                          | New in     |
    +=========================================================+======================================+============+
-   | :meth:`assertRaises(exc, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises `exc`  |            |
+   | :meth:`assertRaises(exc, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises *exc*  |            |
    | <TestCase.assertRaises>`                                |                                      |            |
    +---------------------------------------------------------+--------------------------------------+------------+
-   | :meth:`assertRaisesRegex(exc, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises `exc`  | 3.1        |
-   | <TestCase.assertRaisesRegex>`                           | and the message matches `re`         |            |
+   | :meth:`assertRaisesRegex(exc, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises *exc*  | 3.1        |
+   | <TestCase.assertRaisesRegex>`                           | and the message matches *re*         |            |
    +---------------------------------------------------------+--------------------------------------+------------+
-   | :meth:`assertWarns(warn, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises `warn` | 3.2        |
+   | :meth:`assertWarns(warn, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises *warn* | 3.2        |
    | <TestCase.assertWarns>`                                 |                                      |            |
    +---------------------------------------------------------+--------------------------------------+------------+
-   | :meth:`assertWarnsRegex(warn, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises `warn` | 3.2        |
-   | <TestCase.assertWarnsRegex>`                            | and the message matches `re`         |            |
+   | :meth:`assertWarnsRegex(warn, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises *warn* | 3.2        |
+   | <TestCase.assertWarnsRegex>`                            | and the message matches *re*         |            |
    +---------------------------------------------------------+--------------------------------------+------------+
 
    .. method:: assertRaises(exception, callable, *args, **kwds)
@@ -1092,7 +1092,7 @@
    | :meth:`assertNotRegex(s, re)          | ``not regex.search(s)``        | 3.2          |
    | <TestCase.assertNotRegex>`            |                                |              |
    +---------------------------------------+--------------------------------+--------------+
-   | :meth:`assertCountEqual(a, b)         | `a` and `b` have the same      | 3.2          |
+   | :meth:`assertCountEqual(a, b)         | *a* and *b* have the same      | 3.2          |
    | <TestCase.assertCountEqual>`          | elements in the same number,   |              |
    |                                       | regardless of their order      |              |
    +---------------------------------------+--------------------------------+--------------+
@@ -1887,7 +1887,7 @@
 .. class:: TextTestRunner(stream=None, descriptions=True, verbosity=1, runnerclass=None, warnings=None)
 
    A basic test runner implementation that outputs results to a stream. If *stream*
-   is `None`, the default, `sys.stderr` is used as the output stream. This class
+   is ``None``, the default, :data:`sys.stderr` is used as the output stream. This class
    has a few configurable parameters, but is essentially very simple.  Graphical
    applications which run test suites should provide alternate implementations.
 
@@ -1904,7 +1904,7 @@
       Added the ``warnings`` argument.
 
    .. versionchanged:: 3.2
-      The default stream is set to `sys.stderr` at instantiation time rather
+      The default stream is set to :data:`sys.stderr` at instantiation time rather
       than import time.
 
    .. method:: _makeResult()
diff --git a/Lib/compileall.py b/Lib/compileall.py
--- a/Lib/compileall.py
+++ b/Lib/compileall.py
@@ -142,7 +142,7 @@
 
     Arguments (all optional):
 
-    skip_curdir: if true, skip current directory (default true)
+    skip_curdir: if true, skip current directory (default True)
     maxlevels:   max recursion level (default 0)
     force: as for compile_dir() (default False)
     quiet: as for compile_dir() (default False)
@@ -177,17 +177,17 @@
                         help='use legacy (pre-PEP3147) compiled file locations')
     parser.add_argument('-d', metavar='DESTDIR',  dest='ddir', default=None,
                         help=('directory to prepend to file paths for use in '
-                              'compile time tracebacks and in runtime '
+                              'compile-time tracebacks and in runtime '
                               'tracebacks in cases where the source file is '
                               'unavailable'))
     parser.add_argument('-x', metavar='REGEXP', dest='rx', default=None,
-                        help=('skip files matching the regular expression. '
-                              'The regexp is searched for in the full path '
-                              'to each file considered for compilation.'))
+                        help=('skip files matching the regular expression; '
+                              'the regexp is searched for in the full path '
+                              'of each file considered for compilation'))
     parser.add_argument('-i', metavar='FILE', dest='flist',
                         help=('add all the files and directories listed in '
-                              'FILE to the list considered for compilation. '
-                              'If "-", names are read from stdin.'))
+                              'FILE to the list considered for compilation; '
+                              'if "-", names are read from stdin'))
     parser.add_argument('compile_dest', metavar='FILE|DIR', nargs='*',
                         help=('zero or more file and directory names '
                               'to compile; if no arguments given, defaults '
diff --git a/Lib/distutils/tests/support.py b/Lib/distutils/tests/support.py
--- a/Lib/distutils/tests/support.py
+++ b/Lib/distutils/tests/support.py
@@ -175,10 +175,9 @@
 def fixup_build_ext(cmd):
     """Function needed to make build_ext tests pass.
 
-    When Python was build with --enable-shared on Unix, -L. is not good
-    enough to find the libpython<blah>.so.  This is because regrtest runs
-    it under a tempdir, not in the top level where the .so lives.  By the
-    time we've gotten here, Python's already been chdir'd to the tempdir.
+    When Python was built with --enable-shared on Unix, -L. is not enough to
+    find libpython<blah>.so, because regrtest runs in a tempdir, not in the
+    source directory where the .so lives.
 
     When Python was built with in debug mode on Windows, build_ext commands
     need their debug attribute set, and it is not done automatically for
diff --git a/Lib/pipes.py b/Lib/pipes.py
--- a/Lib/pipes.py
+++ b/Lib/pipes.py
@@ -54,8 +54,6 @@
 
 To create a new template object initialized to a given one:
    t2 = t.clone()
-
-For an example, see the function test() at the end of the file.
 """                                     # '
 
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 17:45:29 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 17:45:29 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge_3=2E2?=
Message-ID: <E1QzVvt-0003BF-9y@dinsdale.python.org>

http://hg.python.org/cpython/rev/321b8372eef6
changeset:   72195:321b8372eef6
parent:      72193:017548088eee
parent:      72194:83c16d258fec
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Sep 02 17:32:30 2011 +0200
summary:
  Merge 3.2

files:
  Parser/asdl_c.py    |  2 +-
  Python/Python-ast.c |  2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py
--- a/Parser/asdl_c.py
+++ b/Parser/asdl_c.py
@@ -807,7 +807,7 @@
 
 static int obj2ast_string(PyObject* obj, PyObject** out, PyArena* arena)
 {
-    if (!PyUnicode_CheckExact(obj)) {
+    if (!PyUnicode_CheckExact(obj) && !PyBytes_CheckExact(obj)) {
         PyErr_SetString(PyExc_TypeError, "AST string must be of type str");
         return 1;
     }
diff --git a/Python/Python-ast.c b/Python/Python-ast.c
--- a/Python/Python-ast.c
+++ b/Python/Python-ast.c
@@ -604,7 +604,7 @@
 
 static int obj2ast_string(PyObject* obj, PyObject** out, PyArena* arena)
 {
-    if (!PyUnicode_CheckExact(obj)) {
+    if (!PyUnicode_CheckExact(obj) && !PyBytes_CheckExact(obj)) {
         PyErr_SetString(PyExc_TypeError, "AST string must be of type str");
         return 1;
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 18:03:03 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 18:03:03 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Avoid_using_the?=
 =?utf8?q?_default_reST_role=2E__Makes_Doc/tools/rstlint=2Epy_happy=2E?=
Message-ID: <E1QzWCt-0004gU-BZ@dinsdale.python.org>

http://hg.python.org/cpython/rev/901c1f025d31
changeset:   72196:901c1f025d31
branch:      2.7
parent:      72153:4dcbae65df3f
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 19:49:31 2011 +0200
summary:
  Avoid using the default reST role.  Makes Doc/tools/rstlint.py happy.

files:
  Doc/documenting/building.rst |  10 +++++-----
  Doc/library/struct.rst       |   4 ++--
  Doc/library/unittest.rst     |  10 +++++-----
  Doc/library/urllib2.rst      |   2 +-
  4 files changed, 13 insertions(+), 13 deletions(-)


diff --git a/Doc/documenting/building.rst b/Doc/documenting/building.rst
--- a/Doc/documenting/building.rst
+++ b/Doc/documenting/building.rst
@@ -17,9 +17,9 @@
    cd Doc
    make html
 
-to check out the necessary toolset in the `tools/` subdirectory and build the
-HTML output files.  To view the generated HTML, point your favorite browser at
-the top-level index `build/html/index.html` after running "make".
+to check out the necessary toolset in the :file:`tools/` subdirectory and build
+the HTML output files.  To view the generated HTML, point your favorite browser
+at the top-level index :file:`build/html/index.html` after running "make".
 
 Available make targets are:
 
@@ -50,10 +50,10 @@
 
  * "pydoc-topics", which builds a Python module containing a dictionary with
    plain text documentation for the labels defined in
-   `tools/sphinxext/pyspecific.py` -- pydoc needs these to show topic and
+   :file:`tools/sphinxext/pyspecific.py` -- pydoc needs these to show topic and
    keyword help.
 
-A "make update" updates the Subversion checkouts in `tools/`.
+A "make update" updates the Subversion checkouts in :file:`tools/`.
 
 
 Without make
diff --git a/Doc/library/struct.rst b/Doc/library/struct.rst
--- a/Doc/library/struct.rst
+++ b/Doc/library/struct.rst
@@ -22,8 +22,8 @@
    alignment is taken into account when unpacking.  This behavior is chosen so
    that the bytes of a packed struct correspond exactly to the layout in memory
    of the corresponding C struct.  To handle platform-independent data formats
-   or omit implicit pad bytes, use `standard` size and alignment instead of
-   `native` size and alignment: see :ref:`struct-alignment` for details.
+   or omit implicit pad bytes, use ``standard`` size and alignment instead of
+   ``native`` size and alignment: see :ref:`struct-alignment` for details.
 
 Functions and Exceptions
 ------------------------
diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -307,7 +307,7 @@
 
     Test discovery loads tests by importing them. Once test discovery has
     found all the test files from the start directory you specify it turns the
-    paths into package names to import. For example `foo/bar/baz.py` will be
+    paths into package names to import. For example :file:`foo/bar/baz.py` will be
     imported as ``foo.bar.baz``.
 
     If you have a package installed globally and attempt test discovery on
@@ -905,11 +905,11 @@
    +---------------------------------------------------------+--------------------------------------+------------+
    | Method                                                  | Checks that                          | New in     |
    +=========================================================+======================================+============+
-   | :meth:`assertRaises(exc, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises `exc`  |            |
+   | :meth:`assertRaises(exc, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises *exc*  |            |
    | <TestCase.assertRaises>`                                |                                      |            |
    +---------------------------------------------------------+--------------------------------------+------------+
-   | :meth:`assertRaisesRegexp(exc, re, fun, *args, **kwds)  | ``fun(*args, **kwds)`` raises `exc`  | 2.7        |
-   | <TestCase.assertRaisesRegexp>`                          | and the message matches `re`         |            |
+   | :meth:`assertRaisesRegexp(exc, re, fun, *args, **kwds)  | ``fun(*args, **kwds)`` raises *exc*  | 2.7        |
+   | <TestCase.assertRaisesRegexp>`                          | and the message matches *re*         |            |
    +---------------------------------------------------------+--------------------------------------+------------+
 
    .. method:: assertRaises(exception, callable, *args, **kwds)
@@ -995,7 +995,7 @@
    | <TestCase.assertItemsEqual>`          | works with unhashable objs     |              |
    +---------------------------------------+--------------------------------+--------------+
    | :meth:`assertDictContainsSubset(a, b) | all the key/value pairs        | 2.7          |
-   | <TestCase.assertDictContainsSubset>`  | in `a` exist in `b`            |              |
+   | <TestCase.assertDictContainsSubset>`  | in *a* exist in *b*            |              |
    +---------------------------------------+--------------------------------+--------------+
 
 
diff --git a/Doc/library/urllib2.rst b/Doc/library/urllib2.rst
--- a/Doc/library/urllib2.rst
+++ b/Doc/library/urllib2.rst
@@ -36,7 +36,7 @@
    :mimetype:`application/x-www-form-urlencoded` format.  The
    :func:`urllib.urlencode` function takes a mapping or sequence of 2-tuples and
    returns a string in this format. urllib2 module sends HTTP/1.1 requests with
-   `Connection:close` header included.
+   ``Connection:close`` header included.
 
    The optional *timeout* parameter specifies a timeout in seconds for blocking
    operations like the connection attempt (if not specified, the global default

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 18:03:04 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 18:03:04 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_some_markup?=
	=?utf8?q?_and_one_typo?=
Message-ID: <E1QzWCu-0004gY-2m@dinsdale.python.org>

http://hg.python.org/cpython/rev/af04764ed7e4
changeset:   72197:af04764ed7e4
branch:      2.7
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 19:54:05 2011 +0200
summary:
  Fix some markup and one typo

files:
  Doc/c-api/init.rst       |  4 ++--
  Doc/faq/design.rst       |  2 +-
  Doc/faq/windows.rst      |  2 +-
  Doc/library/argparse.rst |  2 +-
  4 files changed, 5 insertions(+), 5 deletions(-)


diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -126,7 +126,7 @@
    program name is ``'/usr/local/bin/python'``, the prefix is ``'/usr/local'``. The
    returned string points into static storage; the caller should not modify its
    value.  This corresponds to the :makevar:`prefix` variable in the top-level
-   :file:`Makefile` and the :option:`--prefix` argument to the :program:`configure`
+   :file:`Makefile` and the ``--prefix`` argument to the :program:`configure`
    script at build time.  The value is available to Python code as ``sys.prefix``.
    It is only useful on Unix.  See also the next function.
 
@@ -139,7 +139,7 @@
    program name is ``'/usr/local/bin/python'``, the exec-prefix is
    ``'/usr/local'``.  The returned string points into static storage; the caller
    should not modify its value.  This corresponds to the :makevar:`exec_prefix`
-   variable in the top-level :file:`Makefile` and the :option:`--exec-prefix`
+   variable in the top-level :file:`Makefile` and the ``--exec-prefix``
    argument to the :program:`configure` script at build  time.  The value is
    available to Python code as ``sys.exec_prefix``.  It is only useful on Unix.
 
diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst
--- a/Doc/faq/design.rst
+++ b/Doc/faq/design.rst
@@ -684,7 +684,7 @@
 Python 2.6 adds an :mod:`abc` module that lets you define Abstract Base Classes
 (ABCs).  You can then use :func:`isinstance` and :func:`issubclass` to check
 whether an instance or a class implements a particular ABC.  The
-:mod:`collections` modules defines a set of useful ABCs such as
+:mod:`collections` module defines a set of useful ABCs such as
 :class:`Iterable`, :class:`Container`, and :class:`MutableMapping`.
 
 For Python, many of the advantages of interface specifications can be obtained
diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst
--- a/Doc/faq/windows.rst
+++ b/Doc/faq/windows.rst
@@ -542,7 +542,7 @@
 :func:`execfile` with the name of your file as argument.
 
 Also note that you can not mix-and-match Debug and Release versions.  If you
-wish to use the Debug Multithreaded DLL, then your module *must* have an "_d"
+wish to use the Debug Multithreaded DLL, then your module *must* have ``_d``
 appended to the base name.
 
 
diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst
--- a/Doc/library/argparse.rst
+++ b/Doc/library/argparse.rst
@@ -153,7 +153,7 @@
      conflicting optionals.
 
    * prog_ - The name of the program (default:
-     :data:`sys.argv[0]`)
+     ``sys.argv[0]``)
 
    * usage_ - The string describing the program usage (default: generated)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 18:03:04 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 18:03:04 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Document_that_T?=
 =?utf8?q?rue/False/None_don=E2=80=99t_use_=3Akeyword=3A_in_doc?=
Message-ID: <E1QzWCu-0004gd-Ph@dinsdale.python.org>

http://hg.python.org/cpython/rev/d4b8dc7e0699
changeset:   72198:d4b8dc7e0699
branch:      2.7
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 19:56:04 2011 +0200
summary:
  Document that True/False/None don?t use :keyword: in doc

files:
  Doc/documenting/markup.rst |  5 ++++-
  1 files changed, 4 insertions(+), 1 deletions(-)


diff --git a/Doc/documenting/markup.rst b/Doc/documenting/markup.rst
--- a/Doc/documenting/markup.rst
+++ b/Doc/documenting/markup.rst
@@ -472,7 +472,10 @@
 
 .. describe:: keyword
 
-   The name of a keyword in Python.
+   The name of a Python keyword.  Using this role will generate a link to the
+   documentation of the keyword.  ``True``, ``False`` and ``None`` do not use
+   this role, but simple code markup (````True````), given that they're
+   fundamental to the language and should be known to any programmer.
 
 .. describe:: mailheader
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 18:03:05 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 18:03:05 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Document_that_f?=
 =?utf8?q?ormat_string_don=E2=80=99t_support_arbitrary_dictonary_keys=2E?=
Message-ID: <E1QzWCv-0004gh-GR@dinsdale.python.org>

http://hg.python.org/cpython/rev/e75272e9c12d
changeset:   72199:e75272e9c12d
branch:      2.7
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 19:57:01 2011 +0200
summary:
  Document that format string don?t support arbitrary dictonary keys.

Text adapted from the PEP.  Addition requested by Terry J. Reedy on
2011-02-23 on python-dev.

files:
  Doc/library/string.rst |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Doc/library/string.rst b/Doc/library/string.rst
--- a/Doc/library/string.rst
+++ b/Doc/library/string.rst
@@ -248,6 +248,8 @@
 it refers to a named keyword argument.  If the numerical arg_names in a format string
 are 0, 1, 2, ... in sequence, they can all be omitted (not just some)
 and the numbers 0, 1, 2, ... will be automatically inserted in that order.
+Because *arg_name* is not quote-delimited, it is not possible to specify arbitrary
+dictionary keys (e.g., the strings ``'10'`` or ``':-]'``) within a format string.
 The *arg_name* can be followed by any number of index or
 attribute expressions. An expression of the form ``'.name'`` selects the named
 attribute using :func:`getattr`, while an expression of the form ``'[index]'``

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 18:03:06 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 18:03:06 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Clarify_compile?=
 =?utf8?q?all_command-line_options_=28=2310454=29=2E?=
Message-ID: <E1QzWCw-0004gl-72@dinsdale.python.org>

http://hg.python.org/cpython/rev/4ae85348e3e8
changeset:   72200:4ae85348e3e8
branch:      2.7
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 20:04:50 2011 +0200
summary:
  Clarify compileall command-line options (#10454).

Backport of R. David Murray?s 3.2 patch.  The code is ugly, with print
statements split across lines, but the output is readable.

files:
  Lib/compileall.py |  39 +++++++++++++++++++++++-----------
  1 files changed, 26 insertions(+), 13 deletions(-)


diff --git a/Lib/compileall.py b/Lib/compileall.py
--- a/Lib/compileall.py
+++ b/Lib/compileall.py
@@ -1,4 +1,4 @@
-"""Module/script to "compile" all .py files to .pyc (or .pyo) file.
+"""Module/script to byte-compile all .py files to .pyc (or .pyo) files.
 
 When called as a script with arguments, this compiles the directories
 given as arguments recursively; the -l option prevents it from
@@ -26,8 +26,8 @@
 
     dir:       the directory to byte-compile
     maxlevels: maximum recursion level (default 10)
-    ddir:      if given, purported directory name (this is the
-               directory name that will show up in error messages)
+    ddir:      the directory that will be prepended to the path to the
+               file as it is compiled into each byte-code file.
     force:     if 1, force compilation, even if timestamps are up-to-date
     quiet:     if 1, be quiet during compilation
     """
@@ -64,8 +64,8 @@
     Arguments (only fullname is required):
 
     fullname:  the file to byte-compile
-    ddir:      if given, purported directory name (this is the
-               directory name that will show up in error messages)
+    ddir:      if given, the directory name compiled in to the
+               byte-code file.
     force:     if 1, force compilation, even if timestamps are up-to-date
     quiet:     if 1, be quiet during compilation
     """
@@ -157,14 +157,27 @@
         print msg
         print "usage: python compileall.py [-l] [-f] [-q] [-d destdir] " \
               "[-x regexp] [-i list] [directory|file ...]"
-        print "-l: don't recurse down"
+        print
+        print "arguments: zero or more file and directory names to compile; " \
+              "if no arguments given, "
+        print "           defaults to the equivalent of -l sys.path"
+        print
+        print "options:"
+        print "-l: don't recurse into subdirectories"
         print "-f: force rebuild even if timestamps are up-to-date"
-        print "-q: quiet operation"
-        print "-d destdir: purported directory name for error messages"
-        print "   if no directory arguments, -l sys.path is assumed"
-        print "-x regexp: skip files matching the regular expression regexp"
-        print "   the regexp is searched for in the full path of the file"
-        print "-i list: expand list with its content (file and directory names)"
+        print "-q: output only error messages"
+        print "-d destdir: directory to prepend to file paths for use in " \
+              "compile-time tracebacks and in"
+        print "            runtime tracebacks in cases where the source " \
+              "file is unavailable"
+        print "-x regexp: skip files matching the regular expression regexp; " \
+              "the regexp is searched for"
+        print "           in the full path of each file considered for " \
+              "compilation"
+        print "-i file: add all the files and directories listed in file to " \
+              "the list considered for"
+        print '         compilation; if "-", names are read from stdin'
+
         sys.exit(2)
     maxlevels = 10
     ddir = None
@@ -205,7 +218,7 @@
         else:
             success = compile_path()
     except KeyboardInterrupt:
-        print "\n[interrupt]"
+        print "\n[interrupted]"
         success = 0
     return success
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 18:03:06 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 18:03:06 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Remove_obsolete?=
	=?utf8?q?_comment?=
Message-ID: <E1QzWCw-0004gp-Ub@dinsdale.python.org>

http://hg.python.org/cpython/rev/6d0da4a4d677
changeset:   72201:6d0da4a4d677
branch:      2.7
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 22:06:49 2011 +0200
summary:
  Remove obsolete comment

files:
  Lib/pipes.py |  2 --
  1 files changed, 0 insertions(+), 2 deletions(-)


diff --git a/Lib/pipes.py b/Lib/pipes.py
--- a/Lib/pipes.py
+++ b/Lib/pipes.py
@@ -54,8 +54,6 @@
 
 To create a new template object initialized to a given one:
    t2 = t.clone()
-
-For an example, see the function test() at the end of the file.
 """                                     # '
 
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 18:03:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 02 Sep 2011 18:03:07 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?=
	=?utf8?q?_Branch_merge?=
Message-ID: <E1QzWCx-0004gu-Hx@dinsdale.python.org>

http://hg.python.org/cpython/rev/f08d4f338b0c
changeset:   72202:f08d4f338b0c
branch:      2.7
parent:      72171:85a12278de69
parent:      72201:6d0da4a4d677
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Sep 02 17:33:10 2011 +0200
summary:
  Branch merge

files:
  Doc/c-api/init.rst           |   4 +-
  Doc/documenting/building.rst |  10 +++---
  Doc/documenting/markup.rst   |   5 ++-
  Doc/faq/design.rst           |   2 +-
  Doc/faq/windows.rst          |   2 +-
  Doc/library/argparse.rst     |   2 +-
  Doc/library/string.rst       |   2 +
  Doc/library/struct.rst       |   4 +-
  Doc/library/unittest.rst     |  10 +++---
  Doc/library/urllib2.rst      |   2 +-
  Lib/compileall.py            |  39 ++++++++++++++++--------
  Lib/pipes.py                 |   2 -
  12 files changed, 50 insertions(+), 34 deletions(-)


diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -126,7 +126,7 @@
    program name is ``'/usr/local/bin/python'``, the prefix is ``'/usr/local'``. The
    returned string points into static storage; the caller should not modify its
    value.  This corresponds to the :makevar:`prefix` variable in the top-level
-   :file:`Makefile` and the :option:`--prefix` argument to the :program:`configure`
+   :file:`Makefile` and the ``--prefix`` argument to the :program:`configure`
    script at build time.  The value is available to Python code as ``sys.prefix``.
    It is only useful on Unix.  See also the next function.
 
@@ -139,7 +139,7 @@
    program name is ``'/usr/local/bin/python'``, the exec-prefix is
    ``'/usr/local'``.  The returned string points into static storage; the caller
    should not modify its value.  This corresponds to the :makevar:`exec_prefix`
-   variable in the top-level :file:`Makefile` and the :option:`--exec-prefix`
+   variable in the top-level :file:`Makefile` and the ``--exec-prefix``
    argument to the :program:`configure` script at build  time.  The value is
    available to Python code as ``sys.exec_prefix``.  It is only useful on Unix.
 
diff --git a/Doc/documenting/building.rst b/Doc/documenting/building.rst
--- a/Doc/documenting/building.rst
+++ b/Doc/documenting/building.rst
@@ -17,9 +17,9 @@
    cd Doc
    make html
 
-to check out the necessary toolset in the `tools/` subdirectory and build the
-HTML output files.  To view the generated HTML, point your favorite browser at
-the top-level index `build/html/index.html` after running "make".
+to check out the necessary toolset in the :file:`tools/` subdirectory and build
+the HTML output files.  To view the generated HTML, point your favorite browser
+at the top-level index :file:`build/html/index.html` after running "make".
 
 Available make targets are:
 
@@ -50,10 +50,10 @@
 
  * "pydoc-topics", which builds a Python module containing a dictionary with
    plain text documentation for the labels defined in
-   `tools/sphinxext/pyspecific.py` -- pydoc needs these to show topic and
+   :file:`tools/sphinxext/pyspecific.py` -- pydoc needs these to show topic and
    keyword help.
 
-A "make update" updates the Subversion checkouts in `tools/`.
+A "make update" updates the Subversion checkouts in :file:`tools/`.
 
 
 Without make
diff --git a/Doc/documenting/markup.rst b/Doc/documenting/markup.rst
--- a/Doc/documenting/markup.rst
+++ b/Doc/documenting/markup.rst
@@ -472,7 +472,10 @@
 
 .. describe:: keyword
 
-   The name of a keyword in Python.
+   The name of a Python keyword.  Using this role will generate a link to the
+   documentation of the keyword.  ``True``, ``False`` and ``None`` do not use
+   this role, but simple code markup (````True````), given that they're
+   fundamental to the language and should be known to any programmer.
 
 .. describe:: mailheader
 
diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst
--- a/Doc/faq/design.rst
+++ b/Doc/faq/design.rst
@@ -684,7 +684,7 @@
 Python 2.6 adds an :mod:`abc` module that lets you define Abstract Base Classes
 (ABCs).  You can then use :func:`isinstance` and :func:`issubclass` to check
 whether an instance or a class implements a particular ABC.  The
-:mod:`collections` modules defines a set of useful ABCs such as
+:mod:`collections` module defines a set of useful ABCs such as
 :class:`Iterable`, :class:`Container`, and :class:`MutableMapping`.
 
 For Python, many of the advantages of interface specifications can be obtained
diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst
--- a/Doc/faq/windows.rst
+++ b/Doc/faq/windows.rst
@@ -542,7 +542,7 @@
 :func:`execfile` with the name of your file as argument.
 
 Also note that you can not mix-and-match Debug and Release versions.  If you
-wish to use the Debug Multithreaded DLL, then your module *must* have an "_d"
+wish to use the Debug Multithreaded DLL, then your module *must* have ``_d``
 appended to the base name.
 
 
diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst
--- a/Doc/library/argparse.rst
+++ b/Doc/library/argparse.rst
@@ -153,7 +153,7 @@
      conflicting optionals.
 
    * prog_ - The name of the program (default:
-     :data:`sys.argv[0]`)
+     ``sys.argv[0]``)
 
    * usage_ - The string describing the program usage (default: generated)
 
diff --git a/Doc/library/string.rst b/Doc/library/string.rst
--- a/Doc/library/string.rst
+++ b/Doc/library/string.rst
@@ -248,6 +248,8 @@
 it refers to a named keyword argument.  If the numerical arg_names in a format string
 are 0, 1, 2, ... in sequence, they can all be omitted (not just some)
 and the numbers 0, 1, 2, ... will be automatically inserted in that order.
+Because *arg_name* is not quote-delimited, it is not possible to specify arbitrary
+dictionary keys (e.g., the strings ``'10'`` or ``':-]'``) within a format string.
 The *arg_name* can be followed by any number of index or
 attribute expressions. An expression of the form ``'.name'`` selects the named
 attribute using :func:`getattr`, while an expression of the form ``'[index]'``
diff --git a/Doc/library/struct.rst b/Doc/library/struct.rst
--- a/Doc/library/struct.rst
+++ b/Doc/library/struct.rst
@@ -22,8 +22,8 @@
    alignment is taken into account when unpacking.  This behavior is chosen so
    that the bytes of a packed struct correspond exactly to the layout in memory
    of the corresponding C struct.  To handle platform-independent data formats
-   or omit implicit pad bytes, use `standard` size and alignment instead of
-   `native` size and alignment: see :ref:`struct-alignment` for details.
+   or omit implicit pad bytes, use ``standard`` size and alignment instead of
+   ``native`` size and alignment: see :ref:`struct-alignment` for details.
 
 Functions and Exceptions
 ------------------------
diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -307,7 +307,7 @@
 
     Test discovery loads tests by importing them. Once test discovery has
     found all the test files from the start directory you specify it turns the
-    paths into package names to import. For example `foo/bar/baz.py` will be
+    paths into package names to import. For example :file:`foo/bar/baz.py` will be
     imported as ``foo.bar.baz``.
 
     If you have a package installed globally and attempt test discovery on
@@ -905,11 +905,11 @@
    +---------------------------------------------------------+--------------------------------------+------------+
    | Method                                                  | Checks that                          | New in     |
    +=========================================================+======================================+============+
-   | :meth:`assertRaises(exc, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises `exc`  |            |
+   | :meth:`assertRaises(exc, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises *exc*  |            |
    | <TestCase.assertRaises>`                                |                                      |            |
    +---------------------------------------------------------+--------------------------------------+------------+
-   | :meth:`assertRaisesRegexp(exc, re, fun, *args, **kwds)  | ``fun(*args, **kwds)`` raises `exc`  | 2.7        |
-   | <TestCase.assertRaisesRegexp>`                          | and the message matches `re`         |            |
+   | :meth:`assertRaisesRegexp(exc, re, fun, *args, **kwds)  | ``fun(*args, **kwds)`` raises *exc*  | 2.7        |
+   | <TestCase.assertRaisesRegexp>`                          | and the message matches *re*         |            |
    +---------------------------------------------------------+--------------------------------------+------------+
 
    .. method:: assertRaises(exception, callable, *args, **kwds)
@@ -995,7 +995,7 @@
    | <TestCase.assertItemsEqual>`          | works with unhashable objs     |              |
    +---------------------------------------+--------------------------------+--------------+
    | :meth:`assertDictContainsSubset(a, b) | all the key/value pairs        | 2.7          |
-   | <TestCase.assertDictContainsSubset>`  | in `a` exist in `b`            |              |
+   | <TestCase.assertDictContainsSubset>`  | in *a* exist in *b*            |              |
    +---------------------------------------+--------------------------------+--------------+
 
 
diff --git a/Doc/library/urllib2.rst b/Doc/library/urllib2.rst
--- a/Doc/library/urllib2.rst
+++ b/Doc/library/urllib2.rst
@@ -36,7 +36,7 @@
    :mimetype:`application/x-www-form-urlencoded` format.  The
    :func:`urllib.urlencode` function takes a mapping or sequence of 2-tuples and
    returns a string in this format. urllib2 module sends HTTP/1.1 requests with
-   `Connection:close` header included.
+   ``Connection:close`` header included.
 
    The optional *timeout* parameter specifies a timeout in seconds for blocking
    operations like the connection attempt (if not specified, the global default
diff --git a/Lib/compileall.py b/Lib/compileall.py
--- a/Lib/compileall.py
+++ b/Lib/compileall.py
@@ -1,4 +1,4 @@
-"""Module/script to "compile" all .py files to .pyc (or .pyo) file.
+"""Module/script to byte-compile all .py files to .pyc (or .pyo) files.
 
 When called as a script with arguments, this compiles the directories
 given as arguments recursively; the -l option prevents it from
@@ -26,8 +26,8 @@
 
     dir:       the directory to byte-compile
     maxlevels: maximum recursion level (default 10)
-    ddir:      if given, purported directory name (this is the
-               directory name that will show up in error messages)
+    ddir:      the directory that will be prepended to the path to the
+               file as it is compiled into each byte-code file.
     force:     if 1, force compilation, even if timestamps are up-to-date
     quiet:     if 1, be quiet during compilation
     """
@@ -64,8 +64,8 @@
     Arguments (only fullname is required):
 
     fullname:  the file to byte-compile
-    ddir:      if given, purported directory name (this is the
-               directory name that will show up in error messages)
+    ddir:      if given, the directory name compiled in to the
+               byte-code file.
     force:     if 1, force compilation, even if timestamps are up-to-date
     quiet:     if 1, be quiet during compilation
     """
@@ -157,14 +157,27 @@
         print msg
         print "usage: python compileall.py [-l] [-f] [-q] [-d destdir] " \
               "[-x regexp] [-i list] [directory|file ...]"
-        print "-l: don't recurse down"
+        print
+        print "arguments: zero or more file and directory names to compile; " \
+              "if no arguments given, "
+        print "           defaults to the equivalent of -l sys.path"
+        print
+        print "options:"
+        print "-l: don't recurse into subdirectories"
         print "-f: force rebuild even if timestamps are up-to-date"
-        print "-q: quiet operation"
-        print "-d destdir: purported directory name for error messages"
-        print "   if no directory arguments, -l sys.path is assumed"
-        print "-x regexp: skip files matching the regular expression regexp"
-        print "   the regexp is searched for in the full path of the file"
-        print "-i list: expand list with its content (file and directory names)"
+        print "-q: output only error messages"
+        print "-d destdir: directory to prepend to file paths for use in " \
+              "compile-time tracebacks and in"
+        print "            runtime tracebacks in cases where the source " \
+              "file is unavailable"
+        print "-x regexp: skip files matching the regular expression regexp; " \
+              "the regexp is searched for"
+        print "           in the full path of each file considered for " \
+              "compilation"
+        print "-i file: add all the files and directories listed in file to " \
+              "the list considered for"
+        print '         compilation; if "-", names are read from stdin'
+
         sys.exit(2)
     maxlevels = 10
     ddir = None
@@ -205,7 +218,7 @@
         else:
             success = compile_path()
     except KeyboardInterrupt:
-        print "\n[interrupt]"
+        print "\n[interrupted]"
         success = 0
     return success
 
diff --git a/Lib/pipes.py b/Lib/pipes.py
--- a/Lib/pipes.py
+++ b/Lib/pipes.py
@@ -54,8 +54,6 @@
 
 To create a new template object initialized to a given one:
    t2 = t.clone()
-
-For an example, see the function test() at the end of the file.
 """                                     # '
 
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 20:09:53 2011
From: python-checkins at python.org (sandro.tosi)
Date: Fri, 02 Sep 2011 20:09:53 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzEyNzgxOiBNZW50?=
 =?utf8?q?ion_SO=5FREUSEADDR_flag_near_socket_examples?=
Message-ID: <E1QzYBd-0008Fv-Kj@dinsdale.python.org>

http://hg.python.org/cpython/rev/fe60689d6a2e
changeset:   72203:fe60689d6a2e
branch:      2.7
user:        Sandro Tosi <sandro.tosi at gmail.com>
date:        Fri Sep 02 20:04:20 2011 +0200
summary:
  #12781: Mention SO_REUSEADDR flag near socket examples

files:
  Doc/library/socket.rst |  19 +++++++++++++++++++
  1 files changed, 19 insertions(+), 0 deletions(-)


diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst
--- a/Doc/library/socket.rst
+++ b/Doc/library/socket.rst
@@ -989,3 +989,22 @@
 
    # disabled promiscuous mode
    s.ioctl(socket.SIO_RCVALL, socket.RCVALL_OFF)
+
+
+Running an example several times with too small delay between executions, could
+lead to this error::
+
+   socket.error: [Errno 98] Address already in use
+
+This is because the previous execution has left the socket in a ``TIME_WAIT``
+state, and can't be immediately reused.
+
+There is a :mod:`socket` flag to set, in order to prevent this,
+:data:`socket.SO_REUSEADDR`::
+
+   s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+   s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+   s.bind((HOST, PORT))
+
+the :data:`SO_REUSEADDR` flag tells the kernel to reuse a local socket in
+``TIME_WAIT`` state, without waiting for its natural timeout to expire.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 20:09:54 2011
From: python-checkins at python.org (sandro.tosi)
Date: Fri, 02 Sep 2011 20:09:54 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzEyNzgxOiBNZW50?=
 =?utf8?q?ion_SO=5FREUSEADDR_flag_near_socket_examples?=
Message-ID: <E1QzYBe-0008Fz-Bt@dinsdale.python.org>

http://hg.python.org/cpython/rev/c4588cd2d59a
changeset:   72204:c4588cd2d59a
branch:      3.2
parent:      72194:83c16d258fec
user:        Sandro Tosi <sandro.tosi at gmail.com>
date:        Fri Sep 02 20:06:31 2011 +0200
summary:
  #12781: Mention SO_REUSEADDR flag near socket examples

files:
  Doc/library/socket.rst |  19 +++++++++++++++++++
  1 files changed, 19 insertions(+), 0 deletions(-)


diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst
--- a/Doc/library/socket.rst
+++ b/Doc/library/socket.rst
@@ -1014,6 +1014,25 @@
    s.ioctl(socket.SIO_RCVALL, socket.RCVALL_OFF)
 
 
+Running an example several times with too small delay between executions, could
+lead to this error::
+
+   socket.error: [Errno 98] Address already in use
+
+This is because the previous execution has left the socket in a ``TIME_WAIT``
+state, and can't be immediately reused.
+
+There is a :mod:`socket` flag to set, in order to prevent this,
+:data:`socket.SO_REUSEADDR`::
+
+   s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+   s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+   s.bind((HOST, PORT))
+
+the :data:`SO_REUSEADDR` flag tells the kernel to reuse a local socket in
+``TIME_WAIT`` state, without waiting for its natural timeout to expire.
+
+
 .. seealso::
 
    For an introduction to socket programming (in C), see the following papers:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 20:09:55 2011
From: python-checkins at python.org (sandro.tosi)
Date: Fri, 02 Sep 2011 20:09:55 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_=2312781=3A_merge_with_3=2E2?=
Message-ID: <E1QzYBf-0008G4-2e@dinsdale.python.org>

http://hg.python.org/cpython/rev/eb9a9a043f13
changeset:   72205:eb9a9a043f13
parent:      72195:321b8372eef6
parent:      72204:c4588cd2d59a
user:        Sandro Tosi <sandro.tosi at gmail.com>
date:        Fri Sep 02 20:07:34 2011 +0200
summary:
  #12781: merge with 3.2

files:
  Doc/library/socket.rst |  19 +++++++++++++++++++
  1 files changed, 19 insertions(+), 0 deletions(-)


diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst
--- a/Doc/library/socket.rst
+++ b/Doc/library/socket.rst
@@ -1239,6 +1239,25 @@
    s.ioctl(socket.SIO_RCVALL, socket.RCVALL_OFF)
 
 
+Running an example several times with too small delay between executions, could
+lead to this error::
+
+   socket.error: [Errno 98] Address already in use
+
+This is because the previous execution has left the socket in a ``TIME_WAIT``
+state, and can't be immediately reused.
+
+There is a :mod:`socket` flag to set, in order to prevent this,
+:data:`socket.SO_REUSEADDR`::
+
+   s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+   s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+   s.bind((HOST, PORT))
+
+the :data:`SO_REUSEADDR` flag tells the kernel to reuse a local socket in
+``TIME_WAIT`` state, without waiting for its natural timeout to expire.
+
+
 .. seealso::
 
    For an introduction to socket programming (in C), see the following papers:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 20:44:12 2011
From: python-checkins at python.org (amaury.forgeotdarc)
Date: Fri, 02 Sep 2011 20:44:12 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyNzY0?=
 =?utf8?q?=3A_Fix_a_crash_in_ctypes_when_the_name_of_a_Structure_field_is_?=
 =?utf8?q?not?=
Message-ID: <E1QzYiq-00035T-HE@dinsdale.python.org>

http://hg.python.org/cpython/rev/b8acee08283c
changeset:   72206:b8acee08283c
branch:      3.2
parent:      72204:c4588cd2d59a
user:        Amaury Forgeot d'Arc <amauryfa at gmail.com>
date:        Fri Sep 02 20:39:40 2011 +0200
summary:
  Issue #12764: Fix a crash in ctypes when the name of a Structure field is not
a string.

files:
  Lib/ctypes/test/test_structures.py |   8 ++++++++
  Misc/NEWS                          |   3 +++
  Modules/_ctypes/stgdict.c          |  17 +++++++++++++++--
  3 files changed, 26 insertions(+), 2 deletions(-)


diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py
--- a/Lib/ctypes/test/test_structures.py
+++ b/Lib/ctypes/test/test_structures.py
@@ -239,6 +239,14 @@
             pass
         self.assertRaises(TypeError, setattr, POINT, "_fields_", [("x", 1), ("y", 2)])
 
+    def test_invalid_name(self):
+        # field name must be string
+        def declare_with_name(name):
+            class S(Structure):
+                _fields_ = [(name, c_int)]
+
+        self.assertRaises(TypeError, declare_with_name, b"x")
+
     def test_intarray_fields(self):
         class SomeInts(Structure):
             _fields_ = [("a", c_int * 4)]
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -193,6 +193,9 @@
 Extension Modules
 -----------------
 
+- Issue #12764: Fix a crash in ctypes when the name of a Structure field is not
+  a string.
+
 - Issue #11241: subclasses of ctypes.Array can now be subclassed.
 
 - Issue #9651: Fix a crash when ctypes.create_string_buffer(0) was passed to
diff --git a/Modules/_ctypes/stgdict.c b/Modules/_ctypes/stgdict.c
--- a/Modules/_ctypes/stgdict.c
+++ b/Modules/_ctypes/stgdict.c
@@ -482,8 +482,21 @@
             char *fieldfmt = dict->format ? dict->format : "B";
             char *fieldname = _PyUnicode_AsString(name);
             char *ptr;
-            Py_ssize_t len = strlen(fieldname) + strlen(fieldfmt);
-            char *buf = alloca(len + 2 + 1);
+            Py_ssize_t len; 
+            char *buf;
+
+            if (fieldname == NULL)
+            {
+                PyErr_Format(PyExc_TypeError,
+                             "structure field name must be string not %s",
+                             name->ob_type->tp_name);
+                                
+                Py_DECREF(pair);
+                return -1;
+            }
+
+            len = strlen(fieldname) + strlen(fieldfmt);
+            buf = alloca(len + 2 + 1);
 
             sprintf(buf, "%s:%s:", fieldfmt, fieldname);
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 20:44:13 2011
From: python-checkins at python.org (amaury.forgeotdarc)
Date: Fri, 02 Sep 2011 20:44:13 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_from_3=2E2=3A_Issue_=2312764=3A_Fix_a_crash_in_ctypes_?=
 =?utf8?q?when_the_name_of_a?=
Message-ID: <E1QzYir-00035d-8x@dinsdale.python.org>

http://hg.python.org/cpython/rev/1ed1ea0f4cd8
changeset:   72207:1ed1ea0f4cd8
parent:      72205:eb9a9a043f13
parent:      72206:b8acee08283c
user:        Amaury Forgeot d'Arc <amauryfa at gmail.com>
date:        Fri Sep 02 20:43:59 2011 +0200
summary:
  Merge from 3.2: Issue #12764: Fix a crash in ctypes when the name of a
Structure field is not a string.

files:
  Lib/ctypes/test/test_structures.py |   8 ++++++++
  Misc/NEWS                          |   3 +++
  Modules/_ctypes/stgdict.c          |  17 +++++++++++++++--
  3 files changed, 26 insertions(+), 2 deletions(-)


diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py
--- a/Lib/ctypes/test/test_structures.py
+++ b/Lib/ctypes/test/test_structures.py
@@ -239,6 +239,14 @@
             pass
         self.assertRaises(TypeError, setattr, POINT, "_fields_", [("x", 1), ("y", 2)])
 
+    def test_invalid_name(self):
+        # field name must be string
+        def declare_with_name(name):
+            class S(Structure):
+                _fields_ = [(name, c_int)]
+
+        self.assertRaises(TypeError, declare_with_name, b"x")
+
     def test_intarray_fields(self):
         class SomeInts(Structure):
             _fields_ = [("a", c_int * 4)]
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1240,6 +1240,9 @@
 Extension Modules
 -----------------
 
+- Issue #12764: Fix a crash in ctypes when the name of a Structure field is not
+  a string.
+
 - Issue #11241: subclasses of ctypes.Array can now be subclassed.
 
 - Issue #9651: Fix a crash when ctypes.create_string_buffer(0) was passed to
diff --git a/Modules/_ctypes/stgdict.c b/Modules/_ctypes/stgdict.c
--- a/Modules/_ctypes/stgdict.c
+++ b/Modules/_ctypes/stgdict.c
@@ -482,8 +482,21 @@
             char *fieldfmt = dict->format ? dict->format : "B";
             char *fieldname = _PyUnicode_AsString(name);
             char *ptr;
-            Py_ssize_t len = strlen(fieldname) + strlen(fieldfmt);
-            char *buf = alloca(len + 2 + 1);
+            Py_ssize_t len; 
+            char *buf;
+
+            if (fieldname == NULL)
+            {
+                PyErr_Format(PyExc_TypeError,
+                             "structure field name must be string not %s",
+                             name->ob_type->tp_name);
+                                
+                Py_DECREF(pair);
+                return -1;
+            }
+
+            len = strlen(fieldname) + strlen(fieldfmt);
+            buf = alloca(len + 2 + 1);
 
             sprintf(buf, "%s:%s:", fieldfmt, fieldname);
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 20:44:15 2011
From: python-checkins at python.org (amaury.forgeotdarc)
Date: Fri, 02 Sep 2011 20:44:15 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyNzY0?=
 =?utf8?q?=3A_Fix_a_crash_in_ctypes_when_the_name_of_a_Structure_field_is_?=
 =?utf8?q?not?=
Message-ID: <E1QzYit-00035n-7J@dinsdale.python.org>

http://hg.python.org/cpython/rev/73827c23cdde
changeset:   72208:73827c23cdde
branch:      2.7
parent:      72203:fe60689d6a2e
user:        Amaury Forgeot d'Arc <amauryfa at gmail.com>
date:        Fri Sep 02 20:32:23 2011 +0200
summary:
  Issue #12764: Fix a crash in ctypes when the name of a Structure field is not
a string.

files:
  Lib/ctypes/test/test_structures.py |   8 ++++++++
  Misc/NEWS                          |   3 +++
  Modules/_ctypes/stgdict.c          |  17 +++++++++++++++--
  3 files changed, 26 insertions(+), 2 deletions(-)


diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py
--- a/Lib/ctypes/test/test_structures.py
+++ b/Lib/ctypes/test/test_structures.py
@@ -239,6 +239,14 @@
             pass
         self.assertRaises(TypeError, setattr, POINT, "_fields_", [("x", 1), ("y", 2)])
 
+    def test_invalid_name(self):
+        # field name must be string
+        def declare_with_name(name):
+            class S(Structure):
+                _fields_ = [(name, c_int)]
+
+        self.assertRaises(TypeError, declare_with_name, u"x\xe9")
+
     def test_intarray_fields(self):
         class SomeInts(Structure):
             _fields_ = [("a", c_int * 4)]
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -180,6 +180,9 @@
 Extension Modules
 -----------------
 
+- Issue #12764: Fix a crash in ctypes when the name of a Structure field is not
+  a string.
+
 - Issue #9651: Fix a crash when ctypes.create_string_buffer(0) was passed to
   some functions like file.write().
 
diff --git a/Modules/_ctypes/stgdict.c b/Modules/_ctypes/stgdict.c
--- a/Modules/_ctypes/stgdict.c
+++ b/Modules/_ctypes/stgdict.c
@@ -494,8 +494,21 @@
             char *fieldfmt = dict->format ? dict->format : "B";
             char *fieldname = PyString_AsString(name);
             char *ptr;
-            Py_ssize_t len = strlen(fieldname) + strlen(fieldfmt);
-            char *buf = alloca(len + 2 + 1);
+            Py_ssize_t len; 
+            char *buf;
+
+            if (fieldname == NULL)
+            {
+                PyErr_Format(PyExc_TypeError,
+                             "structure field name must be string not %s",
+                             name->ob_type->tp_name);
+                                
+                Py_DECREF(pair);
+                return -1;
+            }
+
+            len = strlen(fieldname) + strlen(fieldfmt);
+            buf = alloca(len + 2 + 1);
 
             sprintf(buf, "%s:%s:", fieldfmt, fieldname);
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 21:26:45 2011
From: python-checkins at python.org (sandro.tosi)
Date: Fri, 02 Sep 2011 21:26:45 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Give_credit_to_?=
	=?utf8?q?Adam?=
Message-ID: <E1QzZO1-0006wi-Ar@dinsdale.python.org>

http://hg.python.org/cpython/rev/af09b9d9be93
changeset:   72209:af09b9d9be93
branch:      2.7
user:        Sandro Tosi <sandro.tosi at gmail.com>
date:        Fri Sep 02 21:23:55 2011 +0200
summary:
  Give credit to Adam

files:
  Doc/ACKS.txt |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Doc/ACKS.txt b/Doc/ACKS.txt
--- a/Doc/ACKS.txt
+++ b/Doc/ACKS.txt
@@ -219,6 +219,7 @@
    * Collin Winter
    * Blake Winton
    * Dan Wolfe
+   * Adam Woodbeck
    * Steven Work
    * Thomas Wouters
    * Ka-Ping Yee

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 21:26:46 2011
From: python-checkins at python.org (sandro.tosi)
Date: Fri, 02 Sep 2011 21:26:46 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Give_credit_to_?=
	=?utf8?q?Adam?=
Message-ID: <E1QzZO2-0006ws-2F@dinsdale.python.org>

http://hg.python.org/cpython/rev/ce136cdbc57e
changeset:   72210:ce136cdbc57e
branch:      3.2
parent:      72206:b8acee08283c
user:        Sandro Tosi <sandro.tosi at gmail.com>
date:        Fri Sep 02 21:24:40 2011 +0200
summary:
  Give credit to Adam

files:
  Doc/ACKS.txt |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Doc/ACKS.txt b/Doc/ACKS.txt
--- a/Doc/ACKS.txt
+++ b/Doc/ACKS.txt
@@ -225,6 +225,7 @@
    * Collin Winter
    * Blake Winton
    * Dan Wolfe
+   * Adam Woodbeck
    * Steven Work
    * Thomas Wouters
    * Ka-Ping Yee

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 21:26:46 2011
From: python-checkins at python.org (sandro.tosi)
Date: Fri, 02 Sep 2011 21:26:46 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_merge_with_3=2E2?=
Message-ID: <E1QzZO2-0006xP-Om@dinsdale.python.org>

http://hg.python.org/cpython/rev/874739f36be8
changeset:   72211:874739f36be8
parent:      72207:1ed1ea0f4cd8
parent:      72210:ce136cdbc57e
user:        Sandro Tosi <sandro.tosi at gmail.com>
date:        Fri Sep 02 21:24:55 2011 +0200
summary:
  merge with 3.2

files:
  Doc/ACKS.txt |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Doc/ACKS.txt b/Doc/ACKS.txt
--- a/Doc/ACKS.txt
+++ b/Doc/ACKS.txt
@@ -226,6 +226,7 @@
    * Collin Winter
    * Blake Winton
    * Dan Wolfe
+   * Adam Woodbeck
    * Steven Work
    * Thomas Wouters
    * Ka-Ping Yee

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 23:08:26 2011
From: python-checkins at python.org (amaury.forgeotdarc)
Date: Fri, 02 Sep 2011 23:08:26 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_ctypes=3A_Slightly_better_e?=
 =?utf8?q?rror_message_when_a_struct_field_name_is_not_a_string=2E?=
Message-ID: <E1QzayQ-0007bT-8l@dinsdale.python.org>

http://hg.python.org/cpython/rev/447b50d8ac23
changeset:   72212:447b50d8ac23
user:        Amaury Forgeot d'Arc <amauryfa at gmail.com>
date:        Fri Sep 02 23:07:54 2011 +0200
summary:
  ctypes: Slightly better error message when a struct field name is not a string.

files:
  Modules/_ctypes/stgdict.c |  12 +++++-------
  1 files changed, 5 insertions(+), 7 deletions(-)


diff --git a/Modules/_ctypes/stgdict.c b/Modules/_ctypes/stgdict.c
--- a/Modules/_ctypes/stgdict.c
+++ b/Modules/_ctypes/stgdict.c
@@ -426,9 +426,9 @@
         StgDictObject *dict;
         int bitsize = 0;
 
-        if (!pair || !PyArg_ParseTuple(pair, "OO|i", &name, &desc, &bitsize)) {
-            PyErr_SetString(PyExc_AttributeError,
-                            "'_fields_' must be a sequence of pairs");
+        if (!pair || !PyArg_ParseTuple(pair, "UO|i", &name, &desc, &bitsize)) {
+            PyErr_SetString(PyExc_TypeError,
+                            "'_fields_' must be a sequence of (name, C type) pairs");
             Py_XDECREF(pair);
             return -1;
         }
@@ -478,6 +478,7 @@
             }
         } else
             bitsize = 0;
+
         if (isStruct && !isPacked) {
             char *fieldfmt = dict->format ? dict->format : "B";
             char *fieldname = _PyUnicode_AsString(name);
@@ -487,10 +488,6 @@
 
             if (fieldname == NULL)
             {
-                PyErr_Format(PyExc_TypeError,
-                             "structure field name must be string not %s",
-                             name->ob_type->tp_name);
-                                
                 Py_DECREF(pair);
                 return -1;
             }
@@ -509,6 +506,7 @@
                 return -1;
             }
         }
+
         if (isStruct) {
             prop = PyCField_FromDesc(desc, i,
                                    &field_size, bitsize, &bitofs,

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 23:20:18 2011
From: python-checkins at python.org (lukasz.langa)
Date: Fri, 02 Sep 2011 23:20:18 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_removed_mislead?=
	=?utf8?q?ing_editing_leftovers?=
Message-ID: <E1Qzb9u-00009L-0Q@dinsdale.python.org>

http://hg.python.org/cpython/rev/14c97947e87c
changeset:   72213:14c97947e87c
branch:      3.2
parent:      72210:ce136cdbc57e
user:        ?ukasz Langa <lukasz at langa.pl>
date:        Fri Sep 02 23:17:39 2011 +0200
summary:
  removed misleading editing leftovers

files:
  Doc/library/configparser.rst |  4 ----
  1 files changed, 0 insertions(+), 4 deletions(-)


diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst
--- a/Doc/library/configparser.rst
+++ b/Doc/library/configparser.rst
@@ -865,10 +865,6 @@
    Comments can be indented. When *inline_comment_prefixes* is given, it will be
    used as the set of substrings that prefix comments in non-empty lines.
 
-   line and inline comments.  For backwards compatibility, the default value for
-   *comment_prefixes* is a special value that indicates that ``;`` and ``#`` can
-   start whole line comments while only ``;`` can start inline comments.
-
    When *strict* is ``True`` (the default), the parser won't allow for
    any section or option duplicates while reading from a single source (file,
    string or dictionary), raising :exc:`DuplicateSectionError` or

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 23:20:18 2011
From: python-checkins at python.org (lukasz.langa)
Date: Fri, 02 Sep 2011 23:20:18 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_merged_configparser_documentation_leftovers_cleanup_from_3?=
 =?utf8?q?=2E2?=
Message-ID: <E1Qzb9u-00009z-OQ@dinsdale.python.org>

http://hg.python.org/cpython/rev/306a2ed78c19
changeset:   72214:306a2ed78c19
parent:      72211:874739f36be8
parent:      72213:14c97947e87c
user:        ?ukasz Langa <lukasz at langa.pl>
date:        Fri Sep 02 23:18:39 2011 +0200
summary:
  merged configparser documentation leftovers cleanup from 3.2

files:
  Doc/library/configparser.rst |  4 ----
  1 files changed, 0 insertions(+), 4 deletions(-)


diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst
--- a/Doc/library/configparser.rst
+++ b/Doc/library/configparser.rst
@@ -865,10 +865,6 @@
    Comments can be indented. When *inline_comment_prefixes* is given, it will be
    used as the set of substrings that prefix comments in non-empty lines.
 
-   line and inline comments.  For backwards compatibility, the default value for
-   *comment_prefixes* is a special value that indicates that ``;`` and ``#`` can
-   start whole line comments while only ``;`` can start inline comments.
-
    When *strict* is ``True`` (the default), the parser won't allow for
    any section or option duplicates while reading from a single source (file,
    string or dictionary), raising :exc:`DuplicateSectionError` or

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  2 23:20:19 2011
From: python-checkins at python.org (lukasz.langa)
Date: Fri, 02 Sep 2011 23:20:19 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
 =?utf8?q?=29=3A_oops=2C_someone_beat_me_to_it=3A_merging_minor_configpars?=
 =?utf8?q?er_documentation_cleanups?=
Message-ID: <E1Qzb9v-0000A3-FO@dinsdale.python.org>

http://hg.python.org/cpython/rev/6374b4ffe00c
changeset:   72215:6374b4ffe00c
parent:      72212:447b50d8ac23
parent:      72214:306a2ed78c19
user:        ?ukasz Langa <lukasz at langa.pl>
date:        Fri Sep 02 23:19:52 2011 +0200
summary:
  oops, someone beat me to it: merging minor configparser documentation cleanups

files:
  Doc/library/configparser.rst |  4 ----
  1 files changed, 0 insertions(+), 4 deletions(-)


diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst
--- a/Doc/library/configparser.rst
+++ b/Doc/library/configparser.rst
@@ -865,10 +865,6 @@
    Comments can be indented. When *inline_comment_prefixes* is given, it will be
    used as the set of substrings that prefix comments in non-empty lines.
 
-   line and inline comments.  For backwards compatibility, the default value for
-   *comment_prefixes* is a special value that indicates that ``;`` and ``#`` can
-   start whole line comments while only ``;`` can start inline comments.
-
    When *strict* is ``True`` (the default), the parser won't allow for
    any section or option duplicates while reading from a single source (file,
    string or dictionary), raising :exc:`DuplicateSectionError` or

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Sat Sep  3 05:19:06 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sat, 03 Sep 2011 05:19:06 +0200
Subject: [Python-checkins] Daily reference leaks (6374b4ffe00c): sum=0
Message-ID: <E1Qzgl8-0003aQ-TY@ap.vmr.nerim.net>

results for 6374b4ffe00c on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogXUlct8', '-x']

From python-checkins at python.org  Sat Sep  3 15:32:30 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Sat, 03 Sep 2011 15:32:30 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogYWRkIGEgX19kaWN0?=
 =?utf8?q?=5F=5F_descr_for_IOBase_=28closes_=2312878=29?=
Message-ID: <E1QzqKk-00048y-TN@dinsdale.python.org>

http://hg.python.org/cpython/rev/8ec8a4579788
changeset:   72216:8ec8a4579788
branch:      3.2
parent:      72213:14c97947e87c
user:        Benjamin Peterson <benjamin at python.org>
date:        Sat Sep 03 09:26:20 2011 -0400
summary:
  add a __dict__ descr for IOBase (closes #12878)

files:
  Lib/test/test_io.py  |  11 +++++++++++
  Misc/NEWS            |   2 ++
  Modules/_io/iobase.c |  14 ++++++++++++++
  3 files changed, 27 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py
--- a/Lib/test/test_io.py
+++ b/Lib/test/test_io.py
@@ -610,6 +610,17 @@
         self.assertEqual(rawio.read(2), None)
         self.assertEqual(rawio.read(2), b"")
 
+    def test_types_have_dict(self):
+        test = (
+            self.IOBase(),
+            self.RawIOBase(),
+            self.TextIOBase(),
+            self.StringIO(),
+            self.BytesIO()
+        )
+        for obj in test:
+            self.assertTrue(hasattr(obj, "__dict__"))
+
 class CIOTest(IOTest):
 
     def test_IOBase_finalize(self):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -28,6 +28,8 @@
 Library
 -------
 
+- Issue #12878: Expose a __dict__ attribute on io.IOBase and its subclasses.
+
 - Issue #12636: IDLE reads the coding cookie when executing a Python script.
 
 - Issue #10946: The distutils commands bdist_dumb, bdist_wininst and bdist_msi
diff --git a/Modules/_io/iobase.c b/Modules/_io/iobase.c
--- a/Modules/_io/iobase.c
+++ b/Modules/_io/iobase.c
@@ -156,6 +156,19 @@
     return PyBool_FromLong(IS_CLOSED(self));
 }
 
+static PyObject *
+iobase_get_dict(PyObject *self)
+{
+    PyObject **dictptr = _PyObject_GetDictPtr(self);
+    PyObject *dict;
+    assert(dictptr);
+    dict = *dictptr;
+    if (dict == NULL)
+        dict = *dictptr = PyDict_New();
+    Py_XINCREF(dict);
+    return dict;
+}
+
 PyObject *
 _PyIOBase_check_closed(PyObject *self, PyObject *args)
 {
@@ -691,6 +704,7 @@
 };
 
 static PyGetSetDef iobase_getset[] = {
+    {"__dict__", iobase_get_dict, NULL, NULL},
     {"closed", (getter)iobase_closed_get, NULL, NULL},
     {NULL}
 };

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep  3 15:32:31 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Sat, 03 Sep 2011 15:32:31 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?b?OiBtZXJnZSAzLjIgKCMxMjg3OCk=?=
Message-ID: <E1QzqKl-000493-LK@dinsdale.python.org>

http://hg.python.org/cpython/rev/fe0497bd7354
changeset:   72217:fe0497bd7354
parent:      72215:6374b4ffe00c
parent:      72216:8ec8a4579788
user:        Benjamin Peterson <benjamin at python.org>
date:        Sat Sep 03 09:32:24 2011 -0400
summary:
  merge 3.2 (#12878)

files:
  Lib/test/test_io.py  |  11 +++++++++++
  Misc/NEWS            |   2 ++
  Modules/_io/iobase.c |  14 ++++++++++++++
  3 files changed, 27 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py
--- a/Lib/test/test_io.py
+++ b/Lib/test/test_io.py
@@ -610,6 +610,17 @@
         self.assertEqual(rawio.read(2), None)
         self.assertEqual(rawio.read(2), b"")
 
+    def test_types_have_dict(self):
+        test = (
+            self.IOBase(),
+            self.RawIOBase(),
+            self.TextIOBase(),
+            self.StringIO(),
+            self.BytesIO()
+        )
+        for obj in test:
+            self.assertTrue(hasattr(obj, "__dict__"))
+
 class CIOTest(IOTest):
 
     def test_IOBase_finalize(self):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -271,6 +271,8 @@
 Library
 -------
 
+- Issue #12878: Expose a __dict__ attribute on io.IOBase and its subclasses.
+
 - Issue #12636: IDLE reads the coding cookie when executing a Python script.
 
 - Issue #12494: On error, call(), check_call(), check_output() and
diff --git a/Modules/_io/iobase.c b/Modules/_io/iobase.c
--- a/Modules/_io/iobase.c
+++ b/Modules/_io/iobase.c
@@ -156,6 +156,19 @@
     return PyBool_FromLong(IS_CLOSED(self));
 }
 
+static PyObject *
+iobase_get_dict(PyObject *self)
+{
+    PyObject **dictptr = _PyObject_GetDictPtr(self);
+    PyObject *dict;
+    assert(dictptr);
+    dict = *dictptr;
+    if (dict == NULL)
+        dict = *dictptr = PyDict_New();
+    Py_XINCREF(dict);
+    return dict;
+}
+
 PyObject *
 _PyIOBase_check_closed(PyObject *self, PyObject *args)
 {
@@ -691,6 +704,7 @@
 };
 
 static PyGetSetDef iobase_getset[] = {
+    {"__dict__", iobase_get_dict, NULL, NULL},
     {"closed", (getter)iobase_closed_get, NULL, NULL},
     {NULL}
 };

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Sun Sep  4 05:17:27 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sun, 04 Sep 2011 05:17:27 +0200
Subject: [Python-checkins] Daily reference leaks (fe0497bd7354): sum=0
Message-ID: <E1R03D5-00043n-Px@ap.vmr.nerim.net>

results for fe0497bd7354 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogEcp8mp', '-x']

From python-checkins at python.org  Sun Sep  4 08:28:30 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:28:30 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogVHlwbyBmaXgu?=
Message-ID: <E1R06By-0001jP-2j@dinsdale.python.org>

http://hg.python.org/cpython/rev/2ecaf6aa3fee
changeset:   72218:2ecaf6aa3fee
branch:      3.2
parent:      72216:8ec8a4579788
user:        Georg Brandl <georg at python.org>
date:        Sun Sep 04 08:10:25 2011 +0200
summary:
  Typo fix.

files:
  Misc/NEWS |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -92,7 +92,7 @@
 - Issue #11627: Fix segfault when __new__ on a exception returns a
   non-exception class.
 
-- Issue #12149: Update the method cache after a type's dictionnary gets
+- Issue #12149: Update the method cache after a type's dictionary gets
   cleared by the garbage collector.  This fixes a segfault when an instance
   and its type get caught in a reference cycle, and the instance's
   deallocator calls one of the methods on the type (e.g. when subclassing

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:29:00 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:29:00 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge_with_3=2E2=2E?=
Message-ID: <E1R06CS-0001jd-NH@dinsdale.python.org>

http://hg.python.org/cpython/rev/fdd83a93ba37
changeset:   72219:fdd83a93ba37
parent:      72217:fe0497bd7354
parent:      72218:2ecaf6aa3fee
user:        Georg Brandl <georg at python.org>
date:        Sun Sep 04 08:12:27 2011 +0200
summary:
  Merge with 3.2.

files:
  Misc/NEWS |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -53,7 +53,7 @@
 - Issue #11627: Fix segfault when __new__ on a exception returns a
   non-exception class.
 
-- Issue #12149: Update the method cache after a type's dictionnary gets
+- Issue #12149: Update the method cache after a type's dictionary gets
   cleared by the garbage collector.  This fixes a segfault when an instance
   and its type get caught in a reference cycle, and the instance's
   deallocator calls one of the methods on the type (e.g. when subclassing

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:41:14 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:41:14 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Update_pydoc_to?=
 =?utf8?q?pics_and_suspicious_ignore=2E?=
Message-ID: <E1R06OI-0002Wg-Kr@dinsdale.python.org>

http://hg.python.org/cpython/rev/1fcd8f97f5cc
changeset:   72220:1fcd8f97f5cc
branch:      3.2
parent:      71853:c13abed5d764
user:        Georg Brandl <georg at python.org>
date:        Sat Aug 13 11:33:35 2011 +0200
summary:
  Update pydoc topics and suspicious ignore.

files:
  Doc/tools/sphinxext/susp-ignored.csv |   8 ++++----
  Lib/pydoc_data/topics.py             |  10 +++++-----
  2 files changed, 9 insertions(+), 9 deletions(-)


diff --git a/Doc/tools/sphinxext/susp-ignored.csv b/Doc/tools/sphinxext/susp-ignored.csv
--- a/Doc/tools/sphinxext/susp-ignored.csv
+++ b/Doc/tools/sphinxext/susp-ignored.csv
@@ -217,10 +217,10 @@
 library/xmlrpc.client,103,:pass,http://user:pass at host:port/path
 library/xmlrpc.client,103,:port,http://user:pass at host:port/path
 library/xmlrpc.client,103,:pass,user:pass
-license,717,`,* THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
-license,717,`,* THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND
-license,879,`,"``Software''), to deal in the Software without restriction, including"
-license,879,`,"THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND,"
+license,,`,* THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
+license,,`,* THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND
+license,,`,"``Software''), to deal in the Software without restriction, including"
+license,,`,"THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND,"
 reference/lexical_analysis,704,`,$       ?       `
 whatsnew/2.7,735,:Sunday,'2009:4:Sunday'
 whatsnew/2.7,862,::,"export PYTHONWARNINGS=all,error:::Cookie:0"
diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py
--- a/Lib/pydoc_data/topics.py
+++ b/Lib/pydoc_data/topics.py
@@ -1,4 +1,4 @@
-# Autogenerated by Sphinx on Sun Jul  3 09:27:35 2011
+# Autogenerated by Sphinx on Sat Aug 13 11:28:40 2011
 topics = {'assert': '\nThe ``assert`` statement\n************************\n\nAssert statements are a convenient way to insert debugging assertions\ninto a program:\n\n   assert_stmt ::= "assert" expression ["," expression]\n\nThe simple form, ``assert expression``, is equivalent to\n\n   if __debug__:\n      if not expression: raise AssertionError\n\nThe extended form, ``assert expression1, expression2``, is equivalent\nto\n\n   if __debug__:\n      if not expression1: raise AssertionError(expression2)\n\nThese equivalences assume that ``__debug__`` and ``AssertionError``\nrefer to the built-in variables with those names.  In the current\nimplementation, the built-in variable ``__debug__`` is ``True`` under\nnormal circumstances, ``False`` when optimization is requested\n(command line option -O).  The current code generator emits no code\nfor an assert statement when optimization is requested at compile\ntime.  Note that it is unnecessary to include the source code for the\nexpression that failed in the error message; it will be displayed as\npart of the stack trace.\n\nAssignments to ``__debug__`` are illegal.  The value for the built-in\nvariable is determined when the interpreter starts.\n',
  'assignment': '\nAssignment statements\n*********************\n\nAssignment statements are used to (re)bind names to values and to\nmodify attributes or items of mutable objects:\n\n   assignment_stmt ::= (target_list "=")+ (expression_list | yield_expression)\n   target_list     ::= target ("," target)* [","]\n   target          ::= identifier\n              | "(" target_list ")"\n              | "[" target_list "]"\n              | attributeref\n              | subscription\n              | slicing\n              | "*" target\n\n(See section *Primaries* for the syntax definitions for the last three\nsymbols.)\n\nAn assignment statement evaluates the expression list (remember that\nthis can be a single expression or a comma-separated list, the latter\nyielding a tuple) and assigns the single resulting object to each of\nthe target lists, from left to right.\n\nAssignment is defined recursively depending on the form of the target\n(list). When a target is part of a mutable object (an attribute\nreference, subscription or slicing), the mutable object must\nultimately perform the assignment and decide about its validity, and\nmay raise an exception if the assignment is unacceptable.  The rules\nobserved by various types and the exceptions raised are given with the\ndefinition of the object types (see section *The standard type\nhierarchy*).\n\nAssignment of an object to a target list, optionally enclosed in\nparentheses or square brackets, is recursively defined as follows.\n\n* If the target list is a single target: The object is assigned to\n  that target.\n\n* If the target list is a comma-separated list of targets: The object\n  must be an iterable with the same number of items as there are\n  targets in the target list, and the items are assigned, from left to\n  right, to the corresponding targets.\n\n  * If the target list contains one target prefixed with an asterisk,\n    called a "starred" target: The object must be a sequence with at\n    least as many items as there are targets in the target list, minus\n    one.  The first items of the sequence are assigned, from left to\n    right, to the targets before the starred target.  The final items\n    of the sequence are assigned to the targets after the starred\n    target.  A list of the remaining items in the sequence is then\n    assigned to the starred target (the list can be empty).\n\n  * Else: The object must be a sequence with the same number of items\n    as there are targets in the target list, and the items are\n    assigned, from left to right, to the corresponding targets.\n\nAssignment of an object to a single target is recursively defined as\nfollows.\n\n* If the target is an identifier (name):\n\n  * If the name does not occur in a ``global`` or ``nonlocal``\n    statement in the current code block: the name is bound to the\n    object in the current local namespace.\n\n  * Otherwise: the name is bound to the object in the global namespace\n    or the outer namespace determined by ``nonlocal``, respectively.\n\n  The name is rebound if it was already bound.  This may cause the\n  reference count for the object previously bound to the name to reach\n  zero, causing the object to be deallocated and its destructor (if it\n  has one) to be called.\n\n* If the target is a target list enclosed in parentheses or in square\n  brackets: The object must be an iterable with the same number of\n  items as there are targets in the target list, and its items are\n  assigned, from left to right, to the corresponding targets.\n\n* If the target is an attribute reference: The primary expression in\n  the reference is evaluated.  It should yield an object with\n  assignable attributes; if this is not the case, ``TypeError`` is\n  raised.  That object is then asked to assign the assigned object to\n  the given attribute; if it cannot perform the assignment, it raises\n  an exception (usually but not necessarily ``AttributeError``).\n\n  Note: If the object is a class instance and the attribute reference\n  occurs on both sides of the assignment operator, the RHS expression,\n  ``a.x`` can access either an instance attribute or (if no instance\n  attribute exists) a class attribute.  The LHS target ``a.x`` is\n  always set as an instance attribute, creating it if necessary.\n  Thus, the two occurrences of ``a.x`` do not necessarily refer to the\n  same attribute: if the RHS expression refers to a class attribute,\n  the LHS creates a new instance attribute as the target of the\n  assignment:\n\n     class Cls:\n         x = 3             # class variable\n     inst = Cls()\n     inst.x = inst.x + 1   # writes inst.x as 4 leaving Cls.x as 3\n\n  This description does not necessarily apply to descriptor\n  attributes, such as properties created with ``property()``.\n\n* If the target is a subscription: The primary expression in the\n  reference is evaluated.  It should yield either a mutable sequence\n  object (such as a list) or a mapping object (such as a dictionary).\n  Next, the subscript expression is evaluated.\n\n  If the primary is a mutable sequence object (such as a list), the\n  subscript must yield an integer.  If it is negative, the sequence\'s\n  length is added to it.  The resulting value must be a nonnegative\n  integer less than the sequence\'s length, and the sequence is asked\n  to assign the assigned object to its item with that index.  If the\n  index is out of range, ``IndexError`` is raised (assignment to a\n  subscripted sequence cannot add new items to a list).\n\n  If the primary is a mapping object (such as a dictionary), the\n  subscript must have a type compatible with the mapping\'s key type,\n  and the mapping is then asked to create a key/datum pair which maps\n  the subscript to the assigned object.  This can either replace an\n  existing key/value pair with the same key value, or insert a new\n  key/value pair (if no key with the same value existed).\n\n  For user-defined objects, the ``__setitem__()`` method is called\n  with appropriate arguments.\n\n* If the target is a slicing: The primary expression in the reference\n  is evaluated.  It should yield a mutable sequence object (such as a\n  list).  The assigned object should be a sequence object of the same\n  type.  Next, the lower and upper bound expressions are evaluated,\n  insofar they are present; defaults are zero and the sequence\'s\n  length.  The bounds should evaluate to integers. If either bound is\n  negative, the sequence\'s length is added to it.  The resulting\n  bounds are clipped to lie between zero and the sequence\'s length,\n  inclusive.  Finally, the sequence object is asked to replace the\n  slice with the items of the assigned sequence.  The length of the\n  slice may be different from the length of the assigned sequence,\n  thus changing the length of the target sequence, if the object\n  allows it.\n\n**CPython implementation detail:** In the current implementation, the\nsyntax for targets is taken to be the same as for expressions, and\ninvalid syntax is rejected during the code generation phase, causing\nless detailed error messages.\n\nWARNING: Although the definition of assignment implies that overlaps\nbetween the left-hand side and the right-hand side are \'safe\' (for\nexample ``a, b = b, a`` swaps two variables), overlaps *within* the\ncollection of assigned-to variables are not safe!  For instance, the\nfollowing program prints ``[0, 2]``:\n\n   x = [0, 1]\n   i = 0\n   i, x[i] = 1, 2\n   print(x)\n\nSee also:\n\n   **PEP 3132** - Extended Iterable Unpacking\n      The specification for the ``*target`` feature.\n\n\nAugmented assignment statements\n===============================\n\nAugmented assignment is the combination, in a single statement, of a\nbinary operation and an assignment statement:\n\n   augmented_assignment_stmt ::= augtarget augop (expression_list | yield_expression)\n   augtarget                 ::= identifier | attributeref | subscription | slicing\n   augop                     ::= "+=" | "-=" | "*=" | "/=" | "//=" | "%=" | "**="\n             | ">>=" | "<<=" | "&=" | "^=" | "|="\n\n(See section *Primaries* for the syntax definitions for the last three\nsymbols.)\n\nAn augmented assignment evaluates the target (which, unlike normal\nassignment statements, cannot be an unpacking) and the expression\nlist, performs the binary operation specific to the type of assignment\non the two operands, and assigns the result to the original target.\nThe target is only evaluated once.\n\nAn augmented assignment expression like ``x += 1`` can be rewritten as\n``x = x + 1`` to achieve a similar, but not exactly equal effect. In\nthe augmented version, ``x`` is only evaluated once. Also, when\npossible, the actual operation is performed *in-place*, meaning that\nrather than creating a new object and assigning that to the target,\nthe old object is modified instead.\n\nWith the exception of assigning to tuples and multiple targets in a\nsingle statement, the assignment done by augmented assignment\nstatements is handled the same way as normal assignments. Similarly,\nwith the exception of the possible *in-place* behavior, the binary\noperation performed by augmented assignment is the same as the normal\nbinary operations.\n\nFor targets which are attribute references, the same *caveat about\nclass and instance attributes* applies as for regular assignments.\n',
  'atom-identifiers': '\nIdentifiers (Names)\n*******************\n\nAn identifier occurring as an atom is a name.  See section\n*Identifiers and keywords* for lexical definition and section *Naming\nand binding* for documentation of naming and binding.\n\nWhen the name is bound to an object, evaluation of the atom yields\nthat object. When a name is not bound, an attempt to evaluate it\nraises a ``NameError`` exception.\n\n**Private name mangling:** When an identifier that textually occurs in\na class definition begins with two or more underscore characters and\ndoes not end in two or more underscores, it is considered a *private\nname* of that class. Private names are transformed to a longer form\nbefore code is generated for them.  The transformation inserts the\nclass name in front of the name, with leading underscores removed, and\na single underscore inserted in front of the class name.  For example,\nthe identifier ``__spam`` occurring in a class named ``Ham`` will be\ntransformed to ``_Ham__spam``.  This transformation is independent of\nthe syntactical context in which the identifier is used.  If the\ntransformed name is extremely long (longer than 255 characters),\nimplementation defined truncation may happen.  If the class name\nconsists only of underscores, no transformation is done.\n',
@@ -15,7 +15,7 @@
  'booleans': '\nBoolean operations\n******************\n\n   or_test  ::= and_test | or_test "or" and_test\n   and_test ::= not_test | and_test "and" not_test\n   not_test ::= comparison | "not" not_test\n\nIn the context of Boolean operations, and also when expressions are\nused by control flow statements, the following values are interpreted\nas false: ``False``, ``None``, numeric zero of all types, and empty\nstrings and containers (including strings, tuples, lists,\ndictionaries, sets and frozensets).  All other values are interpreted\nas true.  User-defined objects can customize their truth value by\nproviding a ``__bool__()`` method.\n\nThe operator ``not`` yields ``True`` if its argument is false,\n``False`` otherwise.\n\nThe expression ``x and y`` first evaluates *x*; if *x* is false, its\nvalue is returned; otherwise, *y* is evaluated and the resulting value\nis returned.\n\nThe expression ``x or y`` first evaluates *x*; if *x* is true, its\nvalue is returned; otherwise, *y* is evaluated and the resulting value\nis returned.\n\n(Note that neither ``and`` nor ``or`` restrict the value and type they\nreturn to ``False`` and ``True``, but rather return the last evaluated\nargument.  This is sometimes useful, e.g., if ``s`` is a string that\nshould be replaced by a default value if it is empty, the expression\n``s or \'foo\'`` yields the desired value.  Because ``not`` has to\ninvent a value anyway, it does not bother to return a value of the\nsame type as its argument, so e.g., ``not \'foo\'`` yields ``False``,\nnot ``\'\'``.)\n',
  'break': '\nThe ``break`` statement\n***********************\n\n   break_stmt ::= "break"\n\n``break`` may only occur syntactically nested in a ``for`` or\n``while`` loop, but not nested in a function or class definition\nwithin that loop.\n\nIt terminates the nearest enclosing loop, skipping the optional\n``else`` clause if the loop has one.\n\nIf a ``for`` loop is terminated by ``break``, the loop control target\nkeeps its current value.\n\nWhen ``break`` passes control out of a ``try`` statement with a\n``finally`` clause, that ``finally`` clause is executed before really\nleaving the loop.\n',
  'callable-types': '\nEmulating callable objects\n**************************\n\nobject.__call__(self[, args...])\n\n   Called when the instance is "called" as a function; if this method\n   is defined, ``x(arg1, arg2, ...)`` is a shorthand for\n   ``x.__call__(arg1, arg2, ...)``.\n',
- 'calls': '\nCalls\n*****\n\nA call calls a callable object (e.g., a function) with a possibly\nempty series of arguments:\n\n   call                 ::= primary "(" [argument_list [","] | comprehension] ")"\n   argument_list        ::= positional_arguments ["," keyword_arguments]\n                       ["," "*" expression] ["," keyword_arguments]\n                       ["," "**" expression]\n                     | keyword_arguments ["," "*" expression]\n                       ["," keyword_arguments] ["," "**" expression]\n                     | "*" expression ["," keyword_arguments] ["," "**" expression]\n                     | "**" expression\n   positional_arguments ::= expression ("," expression)*\n   keyword_arguments    ::= keyword_item ("," keyword_item)*\n   keyword_item         ::= identifier "=" expression\n\nA trailing comma may be present after the positional and keyword\narguments but does not affect the semantics.\n\nThe primary must evaluate to a callable object (user-defined\nfunctions, built-in functions, methods of built-in objects, class\nobjects, methods of class instances, and all objects having a\n``__call__()`` method are callable).  All argument expressions are\nevaluated before the call is attempted.  Please refer to section\n*Function definitions* for the syntax of formal parameter lists.\n\nIf keyword arguments are present, they are first converted to\npositional arguments, as follows.  First, a list of unfilled slots is\ncreated for the formal parameters.  If there are N positional\narguments, they are placed in the first N slots.  Next, for each\nkeyword argument, the identifier is used to determine the\ncorresponding slot (if the identifier is the same as the first formal\nparameter name, the first slot is used, and so on).  If the slot is\nalready filled, a ``TypeError`` exception is raised. Otherwise, the\nvalue of the argument is placed in the slot, filling it (even if the\nexpression is ``None``, it fills the slot).  When all arguments have\nbeen processed, the slots that are still unfilled are filled with the\ncorresponding default value from the function definition.  (Default\nvalues are calculated, once, when the function is defined; thus, a\nmutable object such as a list or dictionary used as default value will\nbe shared by all calls that don\'t specify an argument value for the\ncorresponding slot; this should usually be avoided.)  If there are any\nunfilled slots for which no default value is specified, a\n``TypeError`` exception is raised.  Otherwise, the list of filled\nslots is used as the argument list for the call.\n\n**CPython implementation detail:** An implementation may provide\nbuilt-in functions whose positional parameters do not have names, even\nif they are \'named\' for the purpose of documentation, and which\ntherefore cannot be supplied by keyword.  In CPython, this is the case\nfor functions implemented in C that use ``PyArg_ParseTuple()`` to\nparse their arguments.\n\nIf there are more positional arguments than there are formal parameter\nslots, a ``TypeError`` exception is raised, unless a formal parameter\nusing the syntax ``*identifier`` is present; in this case, that formal\nparameter receives a tuple containing the excess positional arguments\n(or an empty tuple if there were no excess positional arguments).\n\nIf any keyword argument does not correspond to a formal parameter\nname, a ``TypeError`` exception is raised, unless a formal parameter\nusing the syntax ``**identifier`` is present; in this case, that\nformal parameter receives a dictionary containing the excess keyword\narguments (using the keywords as keys and the argument values as\ncorresponding values), or a (new) empty dictionary if there were no\nexcess keyword arguments.\n\nIf the syntax ``*expression`` appears in the function call,\n``expression`` must evaluate to a sequence.  Elements from this\nsequence are treated as if they were additional positional arguments;\nif there are positional arguments *x1*,..., *xN*, and ``expression``\nevaluates to a sequence *y1*, ..., *yM*, this is equivalent to a call\nwith M+N positional arguments *x1*, ..., *xN*, *y1*, ..., *yM*.\n\nA consequence of this is that although the ``*expression`` syntax may\nappear *after* some keyword arguments, it is processed *before* the\nkeyword arguments (and the ``**expression`` argument, if any -- see\nbelow).  So:\n\n   >>> def f(a, b):\n   ...  print(a, b)\n   ...\n   >>> f(b=1, *(2,))\n   2 1\n   >>> f(a=1, *(2,))\n   Traceback (most recent call last):\n     File "<stdin>", line 1, in ?\n   TypeError: f() got multiple values for keyword argument \'a\'\n   >>> f(1, *(2,))\n   1 2\n\nIt is unusual for both keyword arguments and the ``*expression``\nsyntax to be used in the same call, so in practice this confusion does\nnot arise.\n\nIf the syntax ``**expression`` appears in the function call,\n``expression`` must evaluate to a mapping, the contents of which are\ntreated as additional keyword arguments.  In the case of a keyword\nappearing in both ``expression`` and as an explicit keyword argument,\na ``TypeError`` exception is raised.\n\nFormal parameters using the syntax ``*identifier`` or ``**identifier``\ncannot be used as positional argument slots or as keyword argument\nnames.\n\nA call always returns some value, possibly ``None``, unless it raises\nan exception.  How this value is computed depends on the type of the\ncallable object.\n\nIf it is---\n\na user-defined function:\n   The code block for the function is executed, passing it the\n   argument list.  The first thing the code block will do is bind the\n   formal parameters to the arguments; this is described in section\n   *Function definitions*.  When the code block executes a ``return``\n   statement, this specifies the return value of the function call.\n\na built-in function or method:\n   The result is up to the interpreter; see *Built-in Functions* for\n   the descriptions of built-in functions and methods.\n\na class object:\n   A new instance of that class is returned.\n\na class instance method:\n   The corresponding user-defined function is called, with an argument\n   list that is one longer than the argument list of the call: the\n   instance becomes the first argument.\n\na class instance:\n   The class must define a ``__call__()`` method; the effect is then\n   the same as if that method was called.\n',
+ 'calls': '\nCalls\n*****\n\nA call calls a callable object (e.g., a function) with a possibly\nempty series of arguments:\n\n   call                 ::= primary "(" [argument_list [","] | comprehension] ")"\n   argument_list        ::= positional_arguments ["," keyword_arguments]\n                       ["," "*" expression] ["," keyword_arguments]\n                       ["," "**" expression]\n                     | keyword_arguments ["," "*" expression]\n                       ["," keyword_arguments] ["," "**" expression]\n                     | "*" expression ["," keyword_arguments] ["," "**" expression]\n                     | "**" expression\n   positional_arguments ::= expression ("," expression)*\n   keyword_arguments    ::= keyword_item ("," keyword_item)*\n   keyword_item         ::= identifier "=" expression\n\nA trailing comma may be present after the positional and keyword\narguments but does not affect the semantics.\n\nThe primary must evaluate to a callable object (user-defined\nfunctions, built-in functions, methods of built-in objects, class\nobjects, methods of class instances, and all objects having a\n``__call__()`` method are callable).  All argument expressions are\nevaluated before the call is attempted.  Please refer to section\n*Function definitions* for the syntax of formal parameter lists.\n\nIf keyword arguments are present, they are first converted to\npositional arguments, as follows.  First, a list of unfilled slots is\ncreated for the formal parameters.  If there are N positional\narguments, they are placed in the first N slots.  Next, for each\nkeyword argument, the identifier is used to determine the\ncorresponding slot (if the identifier is the same as the first formal\nparameter name, the first slot is used, and so on).  If the slot is\nalready filled, a ``TypeError`` exception is raised. Otherwise, the\nvalue of the argument is placed in the slot, filling it (even if the\nexpression is ``None``, it fills the slot).  When all arguments have\nbeen processed, the slots that are still unfilled are filled with the\ncorresponding default value from the function definition.  (Default\nvalues are calculated, once, when the function is defined; thus, a\nmutable object such as a list or dictionary used as default value will\nbe shared by all calls that don\'t specify an argument value for the\ncorresponding slot; this should usually be avoided.)  If there are any\nunfilled slots for which no default value is specified, a\n``TypeError`` exception is raised.  Otherwise, the list of filled\nslots is used as the argument list for the call.\n\n**CPython implementation detail:** An implementation may provide\nbuilt-in functions whose positional parameters do not have names, even\nif they are \'named\' for the purpose of documentation, and which\ntherefore cannot be supplied by keyword.  In CPython, this is the case\nfor functions implemented in C that use ``PyArg_ParseTuple()`` to\nparse their arguments.\n\nIf there are more positional arguments than there are formal parameter\nslots, a ``TypeError`` exception is raised, unless a formal parameter\nusing the syntax ``*identifier`` is present; in this case, that formal\nparameter receives a tuple containing the excess positional arguments\n(or an empty tuple if there were no excess positional arguments).\n\nIf any keyword argument does not correspond to a formal parameter\nname, a ``TypeError`` exception is raised, unless a formal parameter\nusing the syntax ``**identifier`` is present; in this case, that\nformal parameter receives a dictionary containing the excess keyword\narguments (using the keywords as keys and the argument values as\ncorresponding values), or a (new) empty dictionary if there were no\nexcess keyword arguments.\n\nIf the syntax ``*expression`` appears in the function call,\n``expression`` must evaluate to an iterable.  Elements from this\niterable are treated as if they were additional positional arguments;\nif there are positional arguments *x1*, ..., *xN*, and ``expression``\nevaluates to a sequence *y1*, ..., *yM*, this is equivalent to a call\nwith M+N positional arguments *x1*, ..., *xN*, *y1*, ..., *yM*.\n\nA consequence of this is that although the ``*expression`` syntax may\nappear *after* some keyword arguments, it is processed *before* the\nkeyword arguments (and the ``**expression`` argument, if any -- see\nbelow).  So:\n\n   >>> def f(a, b):\n   ...  print(a, b)\n   ...\n   >>> f(b=1, *(2,))\n   2 1\n   >>> f(a=1, *(2,))\n   Traceback (most recent call last):\n     File "<stdin>", line 1, in ?\n   TypeError: f() got multiple values for keyword argument \'a\'\n   >>> f(1, *(2,))\n   1 2\n\nIt is unusual for both keyword arguments and the ``*expression``\nsyntax to be used in the same call, so in practice this confusion does\nnot arise.\n\nIf the syntax ``**expression`` appears in the function call,\n``expression`` must evaluate to a mapping, the contents of which are\ntreated as additional keyword arguments.  In the case of a keyword\nappearing in both ``expression`` and as an explicit keyword argument,\na ``TypeError`` exception is raised.\n\nFormal parameters using the syntax ``*identifier`` or ``**identifier``\ncannot be used as positional argument slots or as keyword argument\nnames.\n\nA call always returns some value, possibly ``None``, unless it raises\nan exception.  How this value is computed depends on the type of the\ncallable object.\n\nIf it is---\n\na user-defined function:\n   The code block for the function is executed, passing it the\n   argument list.  The first thing the code block will do is bind the\n   formal parameters to the arguments; this is described in section\n   *Function definitions*.  When the code block executes a ``return``\n   statement, this specifies the return value of the function call.\n\na built-in function or method:\n   The result is up to the interpreter; see *Built-in Functions* for\n   the descriptions of built-in functions and methods.\n\na class object:\n   A new instance of that class is returned.\n\na class instance method:\n   The corresponding user-defined function is called, with an argument\n   list that is one longer than the argument list of the call: the\n   instance becomes the first argument.\n\na class instance:\n   The class must define a ``__call__()`` method; the effect is then\n   the same as if that method was called.\n',
  'class': '\nClass definitions\n*****************\n\nA class definition defines a class object (see section *The standard\ntype hierarchy*):\n\n   classdef    ::= [decorators] "class" classname [inheritance] ":" suite\n   inheritance ::= "(" [argument_list [","] | comprehension] ")"\n   classname   ::= identifier\n\nA class definition is an executable statement.  The inheritance list\nusually gives a list of base classes (see *Customizing class creation*\nfor more advanced uses), so each item in the list should evaluate to a\nclass object which allows subclassing.  Classes without an inheritance\nlist inherit, by default, from the base class ``object``; hence,\n\n   class Foo:\n       pass\n\nis equivalent to\n\n   class Foo(object):\n       pass\n\nThe class\'s suite is then executed in a new execution frame (see\n*Naming and binding*), using a newly created local namespace and the\noriginal global namespace. (Usually, the suite contains mostly\nfunction definitions.)  When the class\'s suite finishes execution, its\nexecution frame is discarded but its local namespace is saved. [4] A\nclass object is then created using the inheritance list for the base\nclasses and the saved local namespace for the attribute dictionary.\nThe class name is bound to this class object in the original local\nnamespace.\n\nClass creation can be customized heavily using *metaclasses*.\n\nClasses can also be decorated: just like when decorating functions,\n\n   @f1(arg)\n   @f2\n   class Foo: pass\n\nis equivalent to\n\n   class Foo: pass\n   Foo = f1(arg)(f2(Foo))\n\nThe evaluation rules for the decorator expressions are the same as for\nfunction decorators.  The result must be a class object, which is then\nbound to the class name.\n\n**Programmer\'s note:** Variables defined in the class definition are\nclass attributes; they are shared by instances.  Instance attributes\ncan be set in a method with ``self.name = value``.  Both class and\ninstance attributes are accessible through the notation\n"``self.name``", and an instance attribute hides a class attribute\nwith the same name when accessed in this way.  Class attributes can be\nused as defaults for instance attributes, but using mutable values\nthere can lead to unexpected results.  *Descriptors* can be used to\ncreate instance variables with different implementation details.\n\nSee also:\n\n   **PEP 3115** - Metaclasses in Python 3 **PEP 3129** - Class\n   Decorators\n\n-[ Footnotes ]-\n\n[1] The exception is propagated to the invocation stack unless there\n    is a ``finally`` clause which happens to raise another exception.\n    That new exception causes the old one to be lost.\n\n[2] Currently, control "flows off the end" except in the case of an\n    exception or the execution of a ``return``, ``continue``, or\n    ``break`` statement.\n\n[3] A string literal appearing as the first statement in the function\n    body is transformed into the function\'s ``__doc__`` attribute and\n    therefore the function\'s *docstring*.\n\n[4] A string literal appearing as the first statement in the class\n    body is transformed into the namespace\'s ``__doc__`` item and\n    therefore the class\'s *docstring*.\n',
  'comparisons': '\nComparisons\n***********\n\nUnlike C, all comparison operations in Python have the same priority,\nwhich is lower than that of any arithmetic, shifting or bitwise\noperation.  Also unlike C, expressions like ``a < b < c`` have the\ninterpretation that is conventional in mathematics:\n\n   comparison    ::= or_expr ( comp_operator or_expr )*\n   comp_operator ::= "<" | ">" | "==" | ">=" | "<=" | "!="\n                     | "is" ["not"] | ["not"] "in"\n\nComparisons yield boolean values: ``True`` or ``False``.\n\nComparisons can be chained arbitrarily, e.g., ``x < y <= z`` is\nequivalent to ``x < y and y <= z``, except that ``y`` is evaluated\nonly once (but in both cases ``z`` is not evaluated at all when ``x <\ny`` is found to be false).\n\nFormally, if *a*, *b*, *c*, ..., *y*, *z* are expressions and *op1*,\n*op2*, ..., *opN* are comparison operators, then ``a op1 b op2 c ... y\nopN z`` is equivalent to ``a op1 b and b op2 c and ... y opN z``,\nexcept that each expression is evaluated at most once.\n\nNote that ``a op1 b op2 c`` doesn\'t imply any kind of comparison\nbetween *a* and *c*, so that, e.g., ``x < y > z`` is perfectly legal\n(though perhaps not pretty).\n\nThe operators ``<``, ``>``, ``==``, ``>=``, ``<=``, and ``!=`` compare\nthe values of two objects.  The objects need not have the same type.\nIf both are numbers, they are converted to a common type.  Otherwise,\nthe ``==`` and ``!=`` operators *always* consider objects of different\ntypes to be unequal, while the ``<``, ``>``, ``>=`` and ``<=``\noperators raise a ``TypeError`` when comparing objects of different\ntypes that do not implement these operators for the given pair of\ntypes.  You can control comparison behavior of objects of non-built-in\ntypes by defining rich comparison methods like ``__gt__()``, described\nin section *Basic customization*.\n\nComparison of objects of the same type depends on the type:\n\n* Numbers are compared arithmetically.\n\n* The values ``float(\'NaN\')`` and ``Decimal(\'NaN\')`` are special. The\n  are identical to themselves, ``x is x`` but are not equal to\n  themselves, ``x != x``.  Additionally, comparing any value to a\n  not-a-number value will return ``False``.  For example, both ``3 <\n  float(\'NaN\')`` and ``float(\'NaN\') < 3`` will return ``False``.\n\n* Bytes objects are compared lexicographically using the numeric\n  values of their elements.\n\n* Strings are compared lexicographically using the numeric equivalents\n  (the result of the built-in function ``ord()``) of their characters.\n  [3] String and bytes object can\'t be compared!\n\n* Tuples and lists are compared lexicographically using comparison of\n  corresponding elements.  This means that to compare equal, each\n  element must compare equal and the two sequences must be of the same\n  type and have the same length.\n\n  If not equal, the sequences are ordered the same as their first\n  differing elements.  For example, ``[1,2,x] <= [1,2,y]`` has the\n  same value as ``x <= y``.  If the corresponding element does not\n  exist, the shorter sequence is ordered first (for example, ``[1,2] <\n  [1,2,3]``).\n\n* Mappings (dictionaries) compare equal if and only if they have the\n  same ``(key, value)`` pairs. Order comparisons ``(\'<\', \'<=\', \'>=\',\n  \'>\')`` raise ``TypeError``.\n\n* Sets and frozensets define comparison operators to mean subset and\n  superset tests.  Those relations do not define total orderings (the\n  two sets ``{1,2}`` and {2,3} are not equal, nor subsets of one\n  another, nor supersets of one another).  Accordingly, sets are not\n  appropriate arguments for functions which depend on total ordering.\n  For example, ``min()``, ``max()``, and ``sorted()`` produce\n  undefined results given a list of sets as inputs.\n\n* Most other objects of built-in types compare unequal unless they are\n  the same object; the choice whether one object is considered smaller\n  or larger than another one is made arbitrarily but consistently\n  within one execution of a program.\n\nComparison of objects of the differing types depends on whether either\nof the types provide explicit support for the comparison.  Most\nnumeric types can be compared with one another, but comparisons of\n``float`` and ``Decimal`` are not supported to avoid the inevitable\nconfusion arising from representation issues such as ``float(\'1.1\')``\nbeing inexactly represented and therefore not exactly equal to\n``Decimal(\'1.1\')`` which is.  When cross-type comparison is not\nsupported, the comparison method returns ``NotImplemented``.  This can\ncreate the illusion of non-transitivity between supported cross-type\ncomparisons and unsupported comparisons.  For example, ``Decimal(2) ==\n2`` and ``2 == float(2)`` but ``Decimal(2) != float(2)``.\n\nThe operators ``in`` and ``not in`` test for membership.  ``x in s``\nevaluates to true if *x* is a member of *s*, and false otherwise.  ``x\nnot in s`` returns the negation of ``x in s``.  All built-in sequences\nand set types support this as well as dictionary, for which ``in``\ntests whether a the dictionary has a given key. For container types\nsuch as list, tuple, set, frozenset, dict, or collections.deque, the\nexpression ``x in y`` is equivalent to ``any(x is e or x == e for e in\ny)``.\n\nFor the string and bytes types, ``x in y`` is true if and only if *x*\nis a substring of *y*.  An equivalent test is ``y.find(x) != -1``.\nEmpty strings are always considered to be a substring of any other\nstring, so ``"" in "abc"`` will return ``True``.\n\nFor user-defined classes which define the ``__contains__()`` method,\n``x in y`` is true if and only if ``y.__contains__(x)`` is true.\n\nFor user-defined classes which do not define ``__contains__()`` but do\ndefine ``__iter__()``, ``x in y`` is true if some value ``z`` with ``x\n== z`` is produced while iterating over ``y``.  If an exception is\nraised during the iteration, it is as if ``in`` raised that exception.\n\nLastly, the old-style iteration protocol is tried: if a class defines\n``__getitem__()``, ``x in y`` is true if and only if there is a non-\nnegative integer index *i* such that ``x == y[i]``, and all lower\ninteger indices do not raise ``IndexError`` exception.  (If any other\nexception is raised, it is as if ``in`` raised that exception).\n\nThe operator ``not in`` is defined to have the inverse true value of\n``in``.\n\nThe operators ``is`` and ``is not`` test for object identity: ``x is\ny`` is true if and only if *x* and *y* are the same object.  ``x is\nnot y`` yields the inverse truth value. [4]\n',
  'compound': '\nCompound statements\n*******************\n\nCompound statements contain (groups of) other statements; they affect\nor control the execution of those other statements in some way.  In\ngeneral, compound statements span multiple lines, although in simple\nincarnations a whole compound statement may be contained in one line.\n\nThe ``if``, ``while`` and ``for`` statements implement traditional\ncontrol flow constructs.  ``try`` specifies exception handlers and/or\ncleanup code for a group of statements, while the ``with`` statement\nallows the execution of initialization and finalization code around a\nblock of code.  Function and class definitions are also syntactically\ncompound statements.\n\nCompound statements consist of one or more \'clauses.\'  A clause\nconsists of a header and a \'suite.\'  The clause headers of a\nparticular compound statement are all at the same indentation level.\nEach clause header begins with a uniquely identifying keyword and ends\nwith a colon.  A suite is a group of statements controlled by a\nclause.  A suite can be one or more semicolon-separated simple\nstatements on the same line as the header, following the header\'s\ncolon, or it can be one or more indented statements on subsequent\nlines.  Only the latter form of suite can contain nested compound\nstatements; the following is illegal, mostly because it wouldn\'t be\nclear to which ``if`` clause a following ``else`` clause would belong:\n\n   if test1: if test2: print(x)\n\nAlso note that the semicolon binds tighter than the colon in this\ncontext, so that in the following example, either all or none of the\n``print()`` calls are executed:\n\n   if x < y < z: print(x); print(y); print(z)\n\nSummarizing:\n\n   compound_stmt ::= if_stmt\n                     | while_stmt\n                     | for_stmt\n                     | try_stmt\n                     | with_stmt\n                     | funcdef\n                     | classdef\n   suite         ::= stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENT\n   statement     ::= stmt_list NEWLINE | compound_stmt\n   stmt_list     ::= simple_stmt (";" simple_stmt)* [";"]\n\nNote that statements always end in a ``NEWLINE`` possibly followed by\na ``DEDENT``.  Also note that optional continuation clauses always\nbegin with a keyword that cannot start a statement, thus there are no\nambiguities (the \'dangling ``else``\' problem is solved in Python by\nrequiring nested ``if`` statements to be indented).\n\nThe formatting of the grammar rules in the following sections places\neach clause on a separate line for clarity.\n\n\nThe ``if`` statement\n====================\n\nThe ``if`` statement is used for conditional execution:\n\n   if_stmt ::= "if" expression ":" suite\n               ( "elif" expression ":" suite )*\n               ["else" ":" suite]\n\nIt selects exactly one of the suites by evaluating the expressions one\nby one until one is found to be true (see section *Boolean operations*\nfor the definition of true and false); then that suite is executed\n(and no other part of the ``if`` statement is executed or evaluated).\nIf all expressions are false, the suite of the ``else`` clause, if\npresent, is executed.\n\n\nThe ``while`` statement\n=======================\n\nThe ``while`` statement is used for repeated execution as long as an\nexpression is true:\n\n   while_stmt ::= "while" expression ":" suite\n                  ["else" ":" suite]\n\nThis repeatedly tests the expression and, if it is true, executes the\nfirst suite; if the expression is false (which may be the first time\nit is tested) the suite of the ``else`` clause, if present, is\nexecuted and the loop terminates.\n\nA ``break`` statement executed in the first suite terminates the loop\nwithout executing the ``else`` clause\'s suite.  A ``continue``\nstatement executed in the first suite skips the rest of the suite and\ngoes back to testing the expression.\n\n\nThe ``for`` statement\n=====================\n\nThe ``for`` statement is used to iterate over the elements of a\nsequence (such as a string, tuple or list) or other iterable object:\n\n   for_stmt ::= "for" target_list "in" expression_list ":" suite\n                ["else" ":" suite]\n\nThe expression list is evaluated once; it should yield an iterable\nobject.  An iterator is created for the result of the\n``expression_list``.  The suite is then executed once for each item\nprovided by the iterator, in the order of ascending indices.  Each\nitem in turn is assigned to the target list using the standard rules\nfor assignments (see *Assignment statements*), and then the suite is\nexecuted.  When the items are exhausted (which is immediately when the\nsequence is empty or an iterator raises a ``StopIteration``\nexception), the suite in the ``else`` clause, if present, is executed,\nand the loop terminates.\n\nA ``break`` statement executed in the first suite terminates the loop\nwithout executing the ``else`` clause\'s suite.  A ``continue``\nstatement executed in the first suite skips the rest of the suite and\ncontinues with the next item, or with the ``else`` clause if there was\nno next item.\n\nThe suite may assign to the variable(s) in the target list; this does\nnot affect the next item assigned to it.\n\nNames in the target list are not deleted when the loop is finished,\nbut if the sequence is empty, it will not have been assigned to at all\nby the loop.  Hint: the built-in function ``range()`` returns an\niterator of integers suitable to emulate the effect of Pascal\'s ``for\ni := a to b do``; e.g., ``list(range(3))`` returns the list ``[0, 1,\n2]``.\n\nNote: There is a subtlety when the sequence is being modified by the loop\n  (this can only occur for mutable sequences, i.e. lists).  An\n  internal counter is used to keep track of which item is used next,\n  and this is incremented on each iteration.  When this counter has\n  reached the length of the sequence the loop terminates.  This means\n  that if the suite deletes the current (or a previous) item from the\n  sequence, the next item will be skipped (since it gets the index of\n  the current item which has already been treated).  Likewise, if the\n  suite inserts an item in the sequence before the current item, the\n  current item will be treated again the next time through the loop.\n  This can lead to nasty bugs that can be avoided by making a\n  temporary copy using a slice of the whole sequence, e.g.,\n\n     for x in a[:]:\n         if x < 0: a.remove(x)\n\n\nThe ``try`` statement\n=====================\n\nThe ``try`` statement specifies exception handlers and/or cleanup code\nfor a group of statements:\n\n   try_stmt  ::= try1_stmt | try2_stmt\n   try1_stmt ::= "try" ":" suite\n                 ("except" [expression ["as" target]] ":" suite)+\n                 ["else" ":" suite]\n                 ["finally" ":" suite]\n   try2_stmt ::= "try" ":" suite\n                 "finally" ":" suite\n\nThe ``except`` clause(s) specify one or more exception handlers. When\nno exception occurs in the ``try`` clause, no exception handler is\nexecuted. When an exception occurs in the ``try`` suite, a search for\nan exception handler is started.  This search inspects the except\nclauses in turn until one is found that matches the exception.  An\nexpression-less except clause, if present, must be last; it matches\nany exception.  For an except clause with an expression, that\nexpression is evaluated, and the clause matches the exception if the\nresulting object is "compatible" with the exception.  An object is\ncompatible with an exception if it is the class or a base class of the\nexception object or a tuple containing an item compatible with the\nexception.\n\nIf no except clause matches the exception, the search for an exception\nhandler continues in the surrounding code and on the invocation stack.\n[1]\n\nIf the evaluation of an expression in the header of an except clause\nraises an exception, the original search for a handler is canceled and\na search starts for the new exception in the surrounding code and on\nthe call stack (it is treated as if the entire ``try`` statement\nraised the exception).\n\nWhen a matching except clause is found, the exception is assigned to\nthe target specified after the ``as`` keyword in that except clause,\nif present, and the except clause\'s suite is executed.  All except\nclauses must have an executable block.  When the end of this block is\nreached, execution continues normally after the entire try statement.\n(This means that if two nested handlers exist for the same exception,\nand the exception occurs in the try clause of the inner handler, the\nouter handler will not handle the exception.)\n\nWhen an exception has been assigned using ``as target``, it is cleared\nat the end of the except clause.  This is as if\n\n   except E as N:\n       foo\n\nwas translated to\n\n   except E as N:\n       try:\n           foo\n       finally:\n           del N\n\nThis means the exception must be assigned to a different name to be\nable to refer to it after the except clause.  Exceptions are cleared\nbecause with the traceback attached to them, they form a reference\ncycle with the stack frame, keeping all locals in that frame alive\nuntil the next garbage collection occurs.\n\nBefore an except clause\'s suite is executed, details about the\nexception are stored in the ``sys`` module and can be access via\n``sys.exc_info()``. ``sys.exc_info()`` returns a 3-tuple consisting of\nthe exception class, the exception instance and a traceback object\n(see section *The standard type hierarchy*) identifying the point in\nthe program where the exception occurred.  ``sys.exc_info()`` values\nare restored to their previous values (before the call) when returning\nfrom a function that handled an exception.\n\nThe optional ``else`` clause is executed if and when control flows off\nthe end of the ``try`` clause. [2] Exceptions in the ``else`` clause\nare not handled by the preceding ``except`` clauses.\n\nIf ``finally`` is present, it specifies a \'cleanup\' handler.  The\n``try`` clause is executed, including any ``except`` and ``else``\nclauses.  If an exception occurs in any of the clauses and is not\nhandled, the exception is temporarily saved. The ``finally`` clause is\nexecuted.  If there is a saved exception, it is re-raised at the end\nof the ``finally`` clause. If the ``finally`` clause raises another\nexception or executes a ``return`` or ``break`` statement, the saved\nexception is lost.  The exception information is not available to the\nprogram during execution of the ``finally`` clause.\n\nWhen a ``return``, ``break`` or ``continue`` statement is executed in\nthe ``try`` suite of a ``try``...``finally`` statement, the\n``finally`` clause is also executed \'on the way out.\' A ``continue``\nstatement is illegal in the ``finally`` clause. (The reason is a\nproblem with the current implementation --- this restriction may be\nlifted in the future).\n\nAdditional information on exceptions can be found in section\n*Exceptions*, and information on using the ``raise`` statement to\ngenerate exceptions may be found in section *The raise statement*.\n\n\nThe ``with`` statement\n======================\n\nThe ``with`` statement is used to wrap the execution of a block with\nmethods defined by a context manager (see section *With Statement\nContext Managers*). This allows common\n``try``...``except``...``finally`` usage patterns to be encapsulated\nfor convenient reuse.\n\n   with_stmt ::= "with" with_item ("," with_item)* ":" suite\n   with_item ::= expression ["as" target]\n\nThe execution of the ``with`` statement with one "item" proceeds as\nfollows:\n\n1. The context expression (the expression given in the ``with_item``)\n   is evaluated to obtain a context manager.\n\n2. The context manager\'s ``__exit__()`` is loaded for later use.\n\n3. The context manager\'s ``__enter__()`` method is invoked.\n\n4. If a target was included in the ``with`` statement, the return\n   value from ``__enter__()`` is assigned to it.\n\n   Note: The ``with`` statement guarantees that if the ``__enter__()``\n     method returns without an error, then ``__exit__()`` will always\n     be called. Thus, if an error occurs during the assignment to the\n     target list, it will be treated the same as an error occurring\n     within the suite would be. See step 6 below.\n\n5. The suite is executed.\n\n6. The context manager\'s ``__exit__()`` method is invoked.  If an\n   exception caused the suite to be exited, its type, value, and\n   traceback are passed as arguments to ``__exit__()``. Otherwise,\n   three ``None`` arguments are supplied.\n\n   If the suite was exited due to an exception, and the return value\n   from the ``__exit__()`` method was false, the exception is\n   reraised.  If the return value was true, the exception is\n   suppressed, and execution continues with the statement following\n   the ``with`` statement.\n\n   If the suite was exited for any reason other than an exception, the\n   return value from ``__exit__()`` is ignored, and execution proceeds\n   at the normal location for the kind of exit that was taken.\n\nWith more than one item, the context managers are processed as if\nmultiple ``with`` statements were nested:\n\n   with A() as a, B() as b:\n       suite\n\nis equivalent to\n\n   with A() as a:\n       with B() as b:\n           suite\n\nChanged in version 3.1: Support for multiple context expressions.\n\nSee also:\n\n   **PEP 0343** - The "with" statement\n      The specification, background, and examples for the Python\n      ``with`` statement.\n\n\nFunction definitions\n====================\n\nA function definition defines a user-defined function object (see\nsection *The standard type hierarchy*):\n\n   funcdef        ::= [decorators] "def" funcname "(" [parameter_list] ")" ["->" expression] ":" suite\n   decorators     ::= decorator+\n   decorator      ::= "@" dotted_name ["(" [argument_list [","]] ")"] NEWLINE\n   dotted_name    ::= identifier ("." identifier)*\n   parameter_list ::= (defparameter ",")*\n                      (  "*" [parameter] ("," defparameter)*\n                      [, "**" parameter]\n                      | "**" parameter\n                      | defparameter [","] )\n   parameter      ::= identifier [":" expression]\n   defparameter   ::= parameter ["=" expression]\n   funcname       ::= identifier\n\nA function definition is an executable statement.  Its execution binds\nthe function name in the current local namespace to a function object\n(a wrapper around the executable code for the function).  This\nfunction object contains a reference to the current global namespace\nas the global namespace to be used when the function is called.\n\nThe function definition does not execute the function body; this gets\nexecuted only when the function is called. [3]\n\nA function definition may be wrapped by one or more *decorator*\nexpressions. Decorator expressions are evaluated when the function is\ndefined, in the scope that contains the function definition.  The\nresult must be a callable, which is invoked with the function object\nas the only argument. The returned value is bound to the function name\ninstead of the function object.  Multiple decorators are applied in\nnested fashion. For example, the following code\n\n   @f1(arg)\n   @f2\n   def func(): pass\n\nis equivalent to\n\n   def func(): pass\n   func = f1(arg)(f2(func))\n\nWhen one or more parameters have the form *parameter* ``=``\n*expression*, the function is said to have "default parameter values."\nFor a parameter with a default value, the corresponding argument may\nbe omitted from a call, in which case the parameter\'s default value is\nsubstituted.  If a parameter has a default value, all following\nparameters up until the "``*``" must also have a default value ---\nthis is a syntactic restriction that is not expressed by the grammar.\n\n**Default parameter values are evaluated when the function definition\nis executed.** This means that the expression is evaluated once, when\nthe function is defined, and that that same "pre-computed" value is\nused for each call.  This is especially important to understand when a\ndefault parameter is a mutable object, such as a list or a dictionary:\nif the function modifies the object (e.g. by appending an item to a\nlist), the default value is in effect modified. This is generally not\nwhat was intended.  A way around this is to use ``None`` as the\ndefault, and explicitly test for it in the body of the function, e.g.:\n\n   def whats_on_the_telly(penguin=None):\n       if penguin is None:\n           penguin = []\n       penguin.append("property of the zoo")\n       return penguin\n\nFunction call semantics are described in more detail in section\n*Calls*. A function call always assigns values to all parameters\nmentioned in the parameter list, either from position arguments, from\nkeyword arguments, or from default values.  If the form\n"``*identifier``" is present, it is initialized to a tuple receiving\nany excess positional parameters, defaulting to the empty tuple.  If\nthe form "``**identifier``" is present, it is initialized to a new\ndictionary receiving any excess keyword arguments, defaulting to a new\nempty dictionary. Parameters after "``*``" or "``*identifier``" are\nkeyword-only parameters and may only be passed used keyword arguments.\n\nParameters may have annotations of the form "``: expression``"\nfollowing the parameter name.  Any parameter may have an annotation\neven those of the form ``*identifier`` or ``**identifier``.  Functions\nmay have "return" annotation of the form "``-> expression``" after the\nparameter list.  These annotations can be any valid Python expression\nand are evaluated when the function definition is executed.\nAnnotations may be evaluated in a different order than they appear in\nthe source code.  The presence of annotations does not change the\nsemantics of a function.  The annotation values are available as\nvalues of a dictionary keyed by the parameters\' names in the\n``__annotations__`` attribute of the function object.\n\nIt is also possible to create anonymous functions (functions not bound\nto a name), for immediate use in expressions.  This uses lambda forms,\ndescribed in section *Lambdas*.  Note that the lambda form is merely a\nshorthand for a simplified function definition; a function defined in\na "``def``" statement can be passed around or assigned to another name\njust like a function defined by a lambda form.  The "``def``" form is\nactually more powerful since it allows the execution of multiple\nstatements and annotations.\n\n**Programmer\'s note:** Functions are first-class objects.  A "``def``"\nform executed inside a function definition defines a local function\nthat can be returned or passed around.  Free variables used in the\nnested function can access the local variables of the function\ncontaining the def.  See section *Naming and binding* for details.\n\n\nClass definitions\n=================\n\nA class definition defines a class object (see section *The standard\ntype hierarchy*):\n\n   classdef    ::= [decorators] "class" classname [inheritance] ":" suite\n   inheritance ::= "(" [argument_list [","] | comprehension] ")"\n   classname   ::= identifier\n\nA class definition is an executable statement.  The inheritance list\nusually gives a list of base classes (see *Customizing class creation*\nfor more advanced uses), so each item in the list should evaluate to a\nclass object which allows subclassing.  Classes without an inheritance\nlist inherit, by default, from the base class ``object``; hence,\n\n   class Foo:\n       pass\n\nis equivalent to\n\n   class Foo(object):\n       pass\n\nThe class\'s suite is then executed in a new execution frame (see\n*Naming and binding*), using a newly created local namespace and the\noriginal global namespace. (Usually, the suite contains mostly\nfunction definitions.)  When the class\'s suite finishes execution, its\nexecution frame is discarded but its local namespace is saved. [4] A\nclass object is then created using the inheritance list for the base\nclasses and the saved local namespace for the attribute dictionary.\nThe class name is bound to this class object in the original local\nnamespace.\n\nClass creation can be customized heavily using *metaclasses*.\n\nClasses can also be decorated: just like when decorating functions,\n\n   @f1(arg)\n   @f2\n   class Foo: pass\n\nis equivalent to\n\n   class Foo: pass\n   Foo = f1(arg)(f2(Foo))\n\nThe evaluation rules for the decorator expressions are the same as for\nfunction decorators.  The result must be a class object, which is then\nbound to the class name.\n\n**Programmer\'s note:** Variables defined in the class definition are\nclass attributes; they are shared by instances.  Instance attributes\ncan be set in a method with ``self.name = value``.  Both class and\ninstance attributes are accessible through the notation\n"``self.name``", and an instance attribute hides a class attribute\nwith the same name when accessed in this way.  Class attributes can be\nused as defaults for instance attributes, but using mutable values\nthere can lead to unexpected results.  *Descriptors* can be used to\ncreate instance variables with different implementation details.\n\nSee also:\n\n   **PEP 3115** - Metaclasses in Python 3 **PEP 3129** - Class\n   Decorators\n\n-[ Footnotes ]-\n\n[1] The exception is propagated to the invocation stack unless there\n    is a ``finally`` clause which happens to raise another exception.\n    That new exception causes the old one to be lost.\n\n[2] Currently, control "flows off the end" except in the case of an\n    exception or the execution of a ``return``, ``continue``, or\n    ``break`` statement.\n\n[3] A string literal appearing as the first statement in the function\n    body is transformed into the function\'s ``__doc__`` attribute and\n    therefore the function\'s *docstring*.\n\n[4] A string literal appearing as the first statement in the class\n    body is transformed into the namespace\'s ``__doc__`` item and\n    therefore the class\'s *docstring*.\n',
@@ -67,10 +67,10 @@
  'try': '\nThe ``try`` statement\n*********************\n\nThe ``try`` statement specifies exception handlers and/or cleanup code\nfor a group of statements:\n\n   try_stmt  ::= try1_stmt | try2_stmt\n   try1_stmt ::= "try" ":" suite\n                 ("except" [expression ["as" target]] ":" suite)+\n                 ["else" ":" suite]\n                 ["finally" ":" suite]\n   try2_stmt ::= "try" ":" suite\n                 "finally" ":" suite\n\nThe ``except`` clause(s) specify one or more exception handlers. When\nno exception occurs in the ``try`` clause, no exception handler is\nexecuted. When an exception occurs in the ``try`` suite, a search for\nan exception handler is started.  This search inspects the except\nclauses in turn until one is found that matches the exception.  An\nexpression-less except clause, if present, must be last; it matches\nany exception.  For an except clause with an expression, that\nexpression is evaluated, and the clause matches the exception if the\nresulting object is "compatible" with the exception.  An object is\ncompatible with an exception if it is the class or a base class of the\nexception object or a tuple containing an item compatible with the\nexception.\n\nIf no except clause matches the exception, the search for an exception\nhandler continues in the surrounding code and on the invocation stack.\n[1]\n\nIf the evaluation of an expression in the header of an except clause\nraises an exception, the original search for a handler is canceled and\na search starts for the new exception in the surrounding code and on\nthe call stack (it is treated as if the entire ``try`` statement\nraised the exception).\n\nWhen a matching except clause is found, the exception is assigned to\nthe target specified after the ``as`` keyword in that except clause,\nif present, and the except clause\'s suite is executed.  All except\nclauses must have an executable block.  When the end of this block is\nreached, execution continues normally after the entire try statement.\n(This means that if two nested handlers exist for the same exception,\nand the exception occurs in the try clause of the inner handler, the\nouter handler will not handle the exception.)\n\nWhen an exception has been assigned using ``as target``, it is cleared\nat the end of the except clause.  This is as if\n\n   except E as N:\n       foo\n\nwas translated to\n\n   except E as N:\n       try:\n           foo\n       finally:\n           del N\n\nThis means the exception must be assigned to a different name to be\nable to refer to it after the except clause.  Exceptions are cleared\nbecause with the traceback attached to them, they form a reference\ncycle with the stack frame, keeping all locals in that frame alive\nuntil the next garbage collection occurs.\n\nBefore an except clause\'s suite is executed, details about the\nexception are stored in the ``sys`` module and can be access via\n``sys.exc_info()``. ``sys.exc_info()`` returns a 3-tuple consisting of\nthe exception class, the exception instance and a traceback object\n(see section *The standard type hierarchy*) identifying the point in\nthe program where the exception occurred.  ``sys.exc_info()`` values\nare restored to their previous values (before the call) when returning\nfrom a function that handled an exception.\n\nThe optional ``else`` clause is executed if and when control flows off\nthe end of the ``try`` clause. [2] Exceptions in the ``else`` clause\nare not handled by the preceding ``except`` clauses.\n\nIf ``finally`` is present, it specifies a \'cleanup\' handler.  The\n``try`` clause is executed, including any ``except`` and ``else``\nclauses.  If an exception occurs in any of the clauses and is not\nhandled, the exception is temporarily saved. The ``finally`` clause is\nexecuted.  If there is a saved exception, it is re-raised at the end\nof the ``finally`` clause. If the ``finally`` clause raises another\nexception or executes a ``return`` or ``break`` statement, the saved\nexception is lost.  The exception information is not available to the\nprogram during execution of the ``finally`` clause.\n\nWhen a ``return``, ``break`` or ``continue`` statement is executed in\nthe ``try`` suite of a ``try``...``finally`` statement, the\n``finally`` clause is also executed \'on the way out.\' A ``continue``\nstatement is illegal in the ``finally`` clause. (The reason is a\nproblem with the current implementation --- this restriction may be\nlifted in the future).\n\nAdditional information on exceptions can be found in section\n*Exceptions*, and information on using the ``raise`` statement to\ngenerate exceptions may be found in section *The raise statement*.\n',
  'types': '\nThe standard type hierarchy\n***************************\n\nBelow is a list of the types that are built into Python.  Extension\nmodules (written in C, Java, or other languages, depending on the\nimplementation) can define additional types.  Future versions of\nPython may add types to the type hierarchy (e.g., rational numbers,\nefficiently stored arrays of integers, etc.), although such additions\nwill often be provided via the standard library instead.\n\nSome of the type descriptions below contain a paragraph listing\n\'special attributes.\'  These are attributes that provide access to the\nimplementation and are not intended for general use.  Their definition\nmay change in the future.\n\nNone\n   This type has a single value.  There is a single object with this\n   value. This object is accessed through the built-in name ``None``.\n   It is used to signify the absence of a value in many situations,\n   e.g., it is returned from functions that don\'t explicitly return\n   anything. Its truth value is false.\n\nNotImplemented\n   This type has a single value.  There is a single object with this\n   value. This object is accessed through the built-in name\n   ``NotImplemented``. Numeric methods and rich comparison methods may\n   return this value if they do not implement the operation for the\n   operands provided.  (The interpreter will then try the reflected\n   operation, or some other fallback, depending on the operator.)  Its\n   truth value is true.\n\nEllipsis\n   This type has a single value.  There is a single object with this\n   value. This object is accessed through the literal ``...`` or the\n   built-in name ``Ellipsis``.  Its truth value is true.\n\n``numbers.Number``\n   These are created by numeric literals and returned as results by\n   arithmetic operators and arithmetic built-in functions.  Numeric\n   objects are immutable; once created their value never changes.\n   Python numbers are of course strongly related to mathematical\n   numbers, but subject to the limitations of numerical representation\n   in computers.\n\n   Python distinguishes between integers, floating point numbers, and\n   complex numbers:\n\n   ``numbers.Integral``\n      These represent elements from the mathematical set of integers\n      (positive and negative).\n\n      There are two types of integers:\n\n      Integers (``int``)\n\n         These represent numbers in an unlimited range, subject to\n         available (virtual) memory only.  For the purpose of shift\n         and mask operations, a binary representation is assumed, and\n         negative numbers are represented in a variant of 2\'s\n         complement which gives the illusion of an infinite string of\n         sign bits extending to the left.\n\n      Booleans (``bool``)\n         These represent the truth values False and True.  The two\n         objects representing the values False and True are the only\n         Boolean objects. The Boolean type is a subtype of the integer\n         type, and Boolean values behave like the values 0 and 1,\n         respectively, in almost all contexts, the exception being\n         that when converted to a string, the strings ``"False"`` or\n         ``"True"`` are returned, respectively.\n\n      The rules for integer representation are intended to give the\n      most meaningful interpretation of shift and mask operations\n      involving negative integers.\n\n   ``numbers.Real`` (``float``)\n      These represent machine-level double precision floating point\n      numbers. You are at the mercy of the underlying machine\n      architecture (and C or Java implementation) for the accepted\n      range and handling of overflow. Python does not support single-\n      precision floating point numbers; the savings in processor and\n      memory usage that are usually the reason for using these is\n      dwarfed by the overhead of using objects in Python, so there is\n      no reason to complicate the language with two kinds of floating\n      point numbers.\n\n   ``numbers.Complex`` (``complex``)\n      These represent complex numbers as a pair of machine-level\n      double precision floating point numbers.  The same caveats apply\n      as for floating point numbers. The real and imaginary parts of a\n      complex number ``z`` can be retrieved through the read-only\n      attributes ``z.real`` and ``z.imag``.\n\nSequences\n   These represent finite ordered sets indexed by non-negative\n   numbers. The built-in function ``len()`` returns the number of\n   items of a sequence. When the length of a sequence is *n*, the\n   index set contains the numbers 0, 1, ..., *n*-1.  Item *i* of\n   sequence *a* is selected by ``a[i]``.\n\n   Sequences also support slicing: ``a[i:j]`` selects all items with\n   index *k* such that *i* ``<=`` *k* ``<`` *j*.  When used as an\n   expression, a slice is a sequence of the same type.  This implies\n   that the index set is renumbered so that it starts at 0.\n\n   Some sequences also support "extended slicing" with a third "step"\n   parameter: ``a[i:j:k]`` selects all items of *a* with index *x*\n   where ``x = i + n*k``, *n* ``>=`` ``0`` and *i* ``<=`` *x* ``<``\n   *j*.\n\n   Sequences are distinguished according to their mutability:\n\n   Immutable sequences\n      An object of an immutable sequence type cannot change once it is\n      created.  (If the object contains references to other objects,\n      these other objects may be mutable and may be changed; however,\n      the collection of objects directly referenced by an immutable\n      object cannot change.)\n\n      The following types are immutable sequences:\n\n      Strings\n         The items of a string object are Unicode code units.  A\n         Unicode code unit is represented by a string object of one\n         item and can hold either a 16-bit or 32-bit value\n         representing a Unicode ordinal (the maximum value for the\n         ordinal is given in ``sys.maxunicode``, and depends on how\n         Python is configured at compile time).  Surrogate pairs may\n         be present in the Unicode object, and will be reported as two\n         separate items.  The built-in functions ``chr()`` and\n         ``ord()`` convert between code units and nonnegative integers\n         representing the Unicode ordinals as defined in the Unicode\n         Standard 3.0. Conversion from and to other encodings are\n         possible through the string method ``encode()``.\n\n      Tuples\n         The items of a tuple are arbitrary Python objects. Tuples of\n         two or more items are formed by comma-separated lists of\n         expressions.  A tuple of one item (a \'singleton\') can be\n         formed by affixing a comma to an expression (an expression by\n         itself does not create a tuple, since parentheses must be\n         usable for grouping of expressions).  An empty tuple can be\n         formed by an empty pair of parentheses.\n\n      Bytes\n         A bytes object is an immutable array.  The items are 8-bit\n         bytes, represented by integers in the range 0 <= x < 256.\n         Bytes literals (like ``b\'abc\'`` and the built-in function\n         ``bytes()`` can be used to construct bytes objects.  Also,\n         bytes objects can be decoded to strings via the ``decode()``\n         method.\n\n   Mutable sequences\n      Mutable sequences can be changed after they are created.  The\n      subscription and slicing notations can be used as the target of\n      assignment and ``del`` (delete) statements.\n\n      There are currently two intrinsic mutable sequence types:\n\n      Lists\n         The items of a list are arbitrary Python objects.  Lists are\n         formed by placing a comma-separated list of expressions in\n         square brackets. (Note that there are no special cases needed\n         to form lists of length 0 or 1.)\n\n      Byte Arrays\n         A bytearray object is a mutable array. They are created by\n         the built-in ``bytearray()`` constructor.  Aside from being\n         mutable (and hence unhashable), byte arrays otherwise provide\n         the same interface and functionality as immutable bytes\n         objects.\n\n      The extension module ``array`` provides an additional example of\n      a mutable sequence type, as does the ``collections`` module.\n\nSet types\n   These represent unordered, finite sets of unique, immutable\n   objects. As such, they cannot be indexed by any subscript. However,\n   they can be iterated over, and the built-in function ``len()``\n   returns the number of items in a set. Common uses for sets are fast\n   membership testing, removing duplicates from a sequence, and\n   computing mathematical operations such as intersection, union,\n   difference, and symmetric difference.\n\n   For set elements, the same immutability rules apply as for\n   dictionary keys. Note that numeric types obey the normal rules for\n   numeric comparison: if two numbers compare equal (e.g., ``1`` and\n   ``1.0``), only one of them can be contained in a set.\n\n   There are currently two intrinsic set types:\n\n   Sets\n      These represent a mutable set. They are created by the built-in\n      ``set()`` constructor and can be modified afterwards by several\n      methods, such as ``add()``.\n\n   Frozen sets\n      These represent an immutable set.  They are created by the\n      built-in ``frozenset()`` constructor.  As a frozenset is\n      immutable and *hashable*, it can be used again as an element of\n      another set, or as a dictionary key.\n\nMappings\n   These represent finite sets of objects indexed by arbitrary index\n   sets. The subscript notation ``a[k]`` selects the item indexed by\n   ``k`` from the mapping ``a``; this can be used in expressions and\n   as the target of assignments or ``del`` statements. The built-in\n   function ``len()`` returns the number of items in a mapping.\n\n   There is currently a single intrinsic mapping type:\n\n   Dictionaries\n      These represent finite sets of objects indexed by nearly\n      arbitrary values.  The only types of values not acceptable as\n      keys are values containing lists or dictionaries or other\n      mutable types that are compared by value rather than by object\n      identity, the reason being that the efficient implementation of\n      dictionaries requires a key\'s hash value to remain constant.\n      Numeric types used for keys obey the normal rules for numeric\n      comparison: if two numbers compare equal (e.g., ``1`` and\n      ``1.0``) then they can be used interchangeably to index the same\n      dictionary entry.\n\n      Dictionaries are mutable; they can be created by the ``{...}``\n      notation (see section *Dictionary displays*).\n\n      The extension modules ``dbm.ndbm`` and ``dbm.gnu`` provide\n      additional examples of mapping types, as does the\n      ``collections`` module.\n\nCallable types\n   These are the types to which the function call operation (see\n   section *Calls*) can be applied:\n\n   User-defined functions\n      A user-defined function object is created by a function\n      definition (see section *Function definitions*).  It should be\n      called with an argument list containing the same number of items\n      as the function\'s formal parameter list.\n\n      Special attributes:\n\n      +---------------------------+---------------------------------+-------------+\n      | Attribute                 | Meaning                         |             |\n      +===========================+=================================+=============+\n      | ``__doc__``               | The function\'s documentation    | Writable    |\n      |                           | string, or ``None`` if          |             |\n      |                           | unavailable                     |             |\n      +---------------------------+---------------------------------+-------------+\n      | ``__name__``              | The function\'s name             | Writable    |\n      +---------------------------+---------------------------------+-------------+\n      | ``__module__``            | The name of the module the      | Writable    |\n      |                           | function was defined in, or     |             |\n      |                           | ``None`` if unavailable.        |             |\n      +---------------------------+---------------------------------+-------------+\n      | ``__defaults__``          | A tuple containing default      | Writable    |\n      |                           | argument values for those       |             |\n      |                           | arguments that have defaults,   |             |\n      |                           | or ``None`` if no arguments     |             |\n      |                           | have a default value            |             |\n      +---------------------------+---------------------------------+-------------+\n      | ``__code__``              | The code object representing    | Writable    |\n      |                           | the compiled function body.     |             |\n      +---------------------------+---------------------------------+-------------+\n      | ``__globals__``           | A reference to the dictionary   | Read-only   |\n      |                           | that holds the function\'s       |             |\n      |                           | global variables --- the global |             |\n      |                           | namespace of the module in      |             |\n      |                           | which the function was defined. |             |\n      +---------------------------+---------------------------------+-------------+\n      | ``__dict__``              | The namespace supporting        | Writable    |\n      |                           | arbitrary function attributes.  |             |\n      +---------------------------+---------------------------------+-------------+\n      | ``__closure__``           | ``None`` or a tuple of cells    | Read-only   |\n      |                           | that contain bindings for the   |             |\n      |                           | function\'s free variables.      |             |\n      +---------------------------+---------------------------------+-------------+\n      | ``__annotations__``       | A dict containing annotations   | Writable    |\n      |                           | of parameters.  The keys of the |             |\n      |                           | dict are the parameter names,   |             |\n      |                           | or ``\'return\'`` for the return  |             |\n      |                           | annotation, if provided.        |             |\n      +---------------------------+---------------------------------+-------------+\n      | ``__kwdefaults__``        | A dict containing defaults for  | Writable    |\n      |                           | keyword-only parameters.        |             |\n      +---------------------------+---------------------------------+-------------+\n\n      Most of the attributes labelled "Writable" check the type of the\n      assigned value.\n\n      Function objects also support getting and setting arbitrary\n      attributes, which can be used, for example, to attach metadata\n      to functions.  Regular attribute dot-notation is used to get and\n      set such attributes. *Note that the current implementation only\n      supports function attributes on user-defined functions. Function\n      attributes on built-in functions may be supported in the\n      future.*\n\n      Additional information about a function\'s definition can be\n      retrieved from its code object; see the description of internal\n      types below.\n\n   Instance methods\n      An instance method object combines a class, a class instance and\n      any callable object (normally a user-defined function).\n\n      Special read-only attributes: ``__self__`` is the class instance\n      object, ``__func__`` is the function object; ``__doc__`` is the\n      method\'s documentation (same as ``__func__.__doc__``);\n      ``__name__`` is the method name (same as ``__func__.__name__``);\n      ``__module__`` is the name of the module the method was defined\n      in, or ``None`` if unavailable.\n\n      Methods also support accessing (but not setting) the arbitrary\n      function attributes on the underlying function object.\n\n      User-defined method objects may be created when getting an\n      attribute of a class (perhaps via an instance of that class), if\n      that attribute is a user-defined function object or a class\n      method object.\n\n      When an instance method object is created by retrieving a user-\n      defined function object from a class via one of its instances,\n      its ``__self__`` attribute is the instance, and the method\n      object is said to be bound.  The new method\'s ``__func__``\n      attribute is the original function object.\n\n      When a user-defined method object is created by retrieving\n      another method object from a class or instance, the behaviour is\n      the same as for a function object, except that the ``__func__``\n      attribute of the new instance is not the original method object\n      but its ``__func__`` attribute.\n\n      When an instance method object is created by retrieving a class\n      method object from a class or instance, its ``__self__``\n      attribute is the class itself, and its ``__func__`` attribute is\n      the function object underlying the class method.\n\n      When an instance method object is called, the underlying\n      function (``__func__``) is called, inserting the class instance\n      (``__self__``) in front of the argument list.  For instance,\n      when ``C`` is a class which contains a definition for a function\n      ``f()``, and ``x`` is an instance of ``C``, calling ``x.f(1)``\n      is equivalent to calling ``C.f(x, 1)``.\n\n      When an instance method object is derived from a class method\n      object, the "class instance" stored in ``__self__`` will\n      actually be the class itself, so that calling either ``x.f(1)``\n      or ``C.f(1)`` is equivalent to calling ``f(C,1)`` where ``f`` is\n      the underlying function.\n\n      Note that the transformation from function object to instance\n      method object happens each time the attribute is retrieved from\n      the instance.  In some cases, a fruitful optimization is to\n      assign the attribute to a local variable and call that local\n      variable. Also notice that this transformation only happens for\n      user-defined functions; other callable objects (and all non-\n      callable objects) are retrieved without transformation.  It is\n      also important to note that user-defined functions which are\n      attributes of a class instance are not converted to bound\n      methods; this *only* happens when the function is an attribute\n      of the class.\n\n   Generator functions\n      A function or method which uses the ``yield`` statement (see\n      section *The yield statement*) is called a *generator function*.\n      Such a function, when called, always returns an iterator object\n      which can be used to execute the body of the function:  calling\n      the iterator\'s ``__next__()`` method will cause the function to\n      execute until it provides a value using the ``yield`` statement.\n      When the function executes a ``return`` statement or falls off\n      the end, a ``StopIteration`` exception is raised and the\n      iterator will have reached the end of the set of values to be\n      returned.\n\n   Built-in functions\n      A built-in function object is a wrapper around a C function.\n      Examples of built-in functions are ``len()`` and ``math.sin()``\n      (``math`` is a standard built-in module). The number and type of\n      the arguments are determined by the C function. Special read-\n      only attributes: ``__doc__`` is the function\'s documentation\n      string, or ``None`` if unavailable; ``__name__`` is the\n      function\'s name; ``__self__`` is set to ``None`` (but see the\n      next item); ``__module__`` is the name of the module the\n      function was defined in or ``None`` if unavailable.\n\n   Built-in methods\n      This is really a different disguise of a built-in function, this\n      time containing an object passed to the C function as an\n      implicit extra argument.  An example of a built-in method is\n      ``alist.append()``, assuming *alist* is a list object. In this\n      case, the special read-only attribute ``__self__`` is set to the\n      object denoted by *alist*.\n\n   Classes\n      Classes are callable.  These objects normally act as factories\n      for new instances of themselves, but variations are possible for\n      class types that override ``__new__()``.  The arguments of the\n      call are passed to ``__new__()`` and, in the typical case, to\n      ``__init__()`` to initialize the new instance.\n\n   Class Instances\n      Instances of arbitrary classes can be made callable by defining\n      a ``__call__()`` method in their class.\n\nModules\n   Modules are imported by the ``import`` statement (see section *The\n   import statement*). A module object has a namespace implemented by\n   a dictionary object (this is the dictionary referenced by the\n   __globals__ attribute of functions defined in the module).\n   Attribute references are translated to lookups in this dictionary,\n   e.g., ``m.x`` is equivalent to ``m.__dict__["x"]``. A module object\n   does not contain the code object used to initialize the module\n   (since it isn\'t needed once the initialization is done).\n\n   Attribute assignment updates the module\'s namespace dictionary,\n   e.g., ``m.x = 1`` is equivalent to ``m.__dict__["x"] = 1``.\n\n   Special read-only attribute: ``__dict__`` is the module\'s namespace\n   as a dictionary object.\n\n   **CPython implementation detail:** Because of the way CPython\n   clears module dictionaries, the module dictionary will be cleared\n   when the module falls out of scope even if the dictionary still has\n   live references.  To avoid this, copy the dictionary or keep the\n   module around while using its dictionary directly.\n\n   Predefined (writable) attributes: ``__name__`` is the module\'s\n   name; ``__doc__`` is the module\'s documentation string, or ``None``\n   if unavailable; ``__file__`` is the pathname of the file from which\n   the module was loaded, if it was loaded from a file. The\n   ``__file__`` attribute is not present for C modules that are\n   statically linked into the interpreter; for extension modules\n   loaded dynamically from a shared library, it is the pathname of the\n   shared library file.\n\nCustom classes\n   Custom class types are typically created by class definitions (see\n   section *Class definitions*).  A class has a namespace implemented\n   by a dictionary object. Class attribute references are translated\n   to lookups in this dictionary, e.g., ``C.x`` is translated to\n   ``C.__dict__["x"]`` (although there are a number of hooks which\n   allow for other means of locating attributes). When the attribute\n   name is not found there, the attribute search continues in the base\n   classes. This search of the base classes uses the C3 method\n   resolution order which behaves correctly even in the presence of\n   \'diamond\' inheritance structures where there are multiple\n   inheritance paths leading back to a common ancestor. Additional\n   details on the C3 MRO used by Python can be found in the\n   documentation accompanying the 2.3 release at\n   http://www.python.org/download/releases/2.3/mro/.\n\n   When a class attribute reference (for class ``C``, say) would yield\n   a class method object, it is transformed into an instance method\n   object whose ``__self__`` attributes is ``C``.  When it would yield\n   a static method object, it is transformed into the object wrapped\n   by the static method object. See section *Implementing Descriptors*\n   for another way in which attributes retrieved from a class may\n   differ from those actually contained in its ``__dict__``.\n\n   Class attribute assignments update the class\'s dictionary, never\n   the dictionary of a base class.\n\n   A class object can be called (see above) to yield a class instance\n   (see below).\n\n   Special attributes: ``__name__`` is the class name; ``__module__``\n   is the module name in which the class was defined; ``__dict__`` is\n   the dictionary containing the class\'s namespace; ``__bases__`` is a\n   tuple (possibly empty or a singleton) containing the base classes,\n   in the order of their occurrence in the base class list;\n   ``__doc__`` is the class\'s documentation string, or None if\n   undefined.\n\nClass instances\n   A class instance is created by calling a class object (see above).\n   A class instance has a namespace implemented as a dictionary which\n   is the first place in which attribute references are searched.\n   When an attribute is not found there, and the instance\'s class has\n   an attribute by that name, the search continues with the class\n   attributes.  If a class attribute is found that is a user-defined\n   function object, it is transformed into an instance method object\n   whose ``__self__`` attribute is the instance.  Static method and\n   class method objects are also transformed; see above under\n   "Classes".  See section *Implementing Descriptors* for another way\n   in which attributes of a class retrieved via its instances may\n   differ from the objects actually stored in the class\'s\n   ``__dict__``.  If no class attribute is found, and the object\'s\n   class has a ``__getattr__()`` method, that is called to satisfy the\n   lookup.\n\n   Attribute assignments and deletions update the instance\'s\n   dictionary, never a class\'s dictionary.  If the class has a\n   ``__setattr__()`` or ``__delattr__()`` method, this is called\n   instead of updating the instance dictionary directly.\n\n   Class instances can pretend to be numbers, sequences, or mappings\n   if they have methods with certain special names.  See section\n   *Special method names*.\n\n   Special attributes: ``__dict__`` is the attribute dictionary;\n   ``__class__`` is the instance\'s class.\n\nI/O objects (also known as file objects)\n   A *file object* represents an open file.  Various shortcuts are\n   available to create file objects: the ``open()`` built-in function,\n   and also ``os.popen()``, ``os.fdopen()``, and the ``makefile()``\n   method of socket objects (and perhaps by other functions or methods\n   provided by extension modules).\n\n   The objects ``sys.stdin``, ``sys.stdout`` and ``sys.stderr`` are\n   initialized to file objects corresponding to the interpreter\'s\n   standard input, output and error streams; they are all open in text\n   mode and therefore follow the interface defined by the\n   ``io.TextIOBase`` abstract class.\n\nInternal types\n   A few types used internally by the interpreter are exposed to the\n   user. Their definitions may change with future versions of the\n   interpreter, but they are mentioned here for completeness.\n\n   Code objects\n      Code objects represent *byte-compiled* executable Python code,\n      or *bytecode*. The difference between a code object and a\n      function object is that the function object contains an explicit\n      reference to the function\'s globals (the module in which it was\n      defined), while a code object contains no context; also the\n      default argument values are stored in the function object, not\n      in the code object (because they represent values calculated at\n      run-time).  Unlike function objects, code objects are immutable\n      and contain no references (directly or indirectly) to mutable\n      objects.\n\n      Special read-only attributes: ``co_name`` gives the function\n      name; ``co_argcount`` is the number of positional arguments\n      (including arguments with default values); ``co_nlocals`` is the\n      number of local variables used by the function (including\n      arguments); ``co_varnames`` is a tuple containing the names of\n      the local variables (starting with the argument names);\n      ``co_cellvars`` is a tuple containing the names of local\n      variables that are referenced by nested functions;\n      ``co_freevars`` is a tuple containing the names of free\n      variables; ``co_code`` is a string representing the sequence of\n      bytecode instructions; ``co_consts`` is a tuple containing the\n      literals used by the bytecode; ``co_names`` is a tuple\n      containing the names used by the bytecode; ``co_filename`` is\n      the filename from which the code was compiled;\n      ``co_firstlineno`` is the first line number of the function;\n      ``co_lnotab`` is a string encoding the mapping from bytecode\n      offsets to line numbers (for details see the source code of the\n      interpreter); ``co_stacksize`` is the required stack size\n      (including local variables); ``co_flags`` is an integer encoding\n      a number of flags for the interpreter.\n\n      The following flag bits are defined for ``co_flags``: bit\n      ``0x04`` is set if the function uses the ``*arguments`` syntax\n      to accept an arbitrary number of positional arguments; bit\n      ``0x08`` is set if the function uses the ``**keywords`` syntax\n      to accept arbitrary keyword arguments; bit ``0x20`` is set if\n      the function is a generator.\n\n      Future feature declarations (``from __future__ import\n      division``) also use bits in ``co_flags`` to indicate whether a\n      code object was compiled with a particular feature enabled: bit\n      ``0x2000`` is set if the function was compiled with future\n      division enabled; bits ``0x10`` and ``0x1000`` were used in\n      earlier versions of Python.\n\n      Other bits in ``co_flags`` are reserved for internal use.\n\n      If a code object represents a function, the first item in\n      ``co_consts`` is the documentation string of the function, or\n      ``None`` if undefined.\n\n   Frame objects\n      Frame objects represent execution frames.  They may occur in\n      traceback objects (see below).\n\n      Special read-only attributes: ``f_back`` is to the previous\n      stack frame (towards the caller), or ``None`` if this is the\n      bottom stack frame; ``f_code`` is the code object being executed\n      in this frame; ``f_locals`` is the dictionary used to look up\n      local variables; ``f_globals`` is used for global variables;\n      ``f_builtins`` is used for built-in (intrinsic) names;\n      ``f_lasti`` gives the precise instruction (this is an index into\n      the bytecode string of the code object).\n\n      Special writable attributes: ``f_trace``, if not ``None``, is a\n      function called at the start of each source code line (this is\n      used by the debugger); ``f_lineno`` is the current line number\n      of the frame --- writing to this from within a trace function\n      jumps to the given line (only for the bottom-most frame).  A\n      debugger can implement a Jump command (aka Set Next Statement)\n      by writing to f_lineno.\n\n   Traceback objects\n      Traceback objects represent a stack trace of an exception.  A\n      traceback object is created when an exception occurs.  When the\n      search for an exception handler unwinds the execution stack, at\n      each unwound level a traceback object is inserted in front of\n      the current traceback.  When an exception handler is entered,\n      the stack trace is made available to the program. (See section\n      *The try statement*.) It is accessible as the third item of the\n      tuple returned by ``sys.exc_info()``. When the program contains\n      no suitable handler, the stack trace is written (nicely\n      formatted) to the standard error stream; if the interpreter is\n      interactive, it is also made available to the user as\n      ``sys.last_traceback``.\n\n      Special read-only attributes: ``tb_next`` is the next level in\n      the stack trace (towards the frame where the exception\n      occurred), or ``None`` if there is no next level; ``tb_frame``\n      points to the execution frame of the current level;\n      ``tb_lineno`` gives the line number where the exception\n      occurred; ``tb_lasti`` indicates the precise instruction.  The\n      line number and last instruction in the traceback may differ\n      from the line number of its frame object if the exception\n      occurred in a ``try`` statement with no matching except clause\n      or with a finally clause.\n\n   Slice objects\n      Slice objects are used to represent slices for ``__getitem__()``\n      methods.  They are also created by the built-in ``slice()``\n      function.\n\n      Special read-only attributes: ``start`` is the lower bound;\n      ``stop`` is the upper bound; ``step`` is the step value; each is\n      ``None`` if omitted. These attributes can have any type.\n\n      Slice objects support one method:\n\n      slice.indices(self, length)\n\n         This method takes a single integer argument *length* and\n         computes information about the slice that the slice object\n         would describe if applied to a sequence of *length* items.\n         It returns a tuple of three integers; respectively these are\n         the *start* and *stop* indices and the *step* or stride\n         length of the slice. Missing or out-of-bounds indices are\n         handled in a manner consistent with regular slices.\n\n   Static method objects\n      Static method objects provide a way of defeating the\n      transformation of function objects to method objects described\n      above. A static method object is a wrapper around any other\n      object, usually a user-defined method object. When a static\n      method object is retrieved from a class or a class instance, the\n      object actually returned is the wrapped object, which is not\n      subject to any further transformation. Static method objects are\n      not themselves callable, although the objects they wrap usually\n      are. Static method objects are created by the built-in\n      ``staticmethod()`` constructor.\n\n   Class method objects\n      A class method object, like a static method object, is a wrapper\n      around another object that alters the way in which that object\n      is retrieved from classes and class instances. The behaviour of\n      class method objects upon such retrieval is described above,\n      under "User-defined methods". Class method objects are created\n      by the built-in ``classmethod()`` constructor.\n',
  'typesfunctions': '\nFunctions\n*********\n\nFunction objects are created by function definitions.  The only\noperation on a function object is to call it: ``func(argument-list)``.\n\nThere are really two flavors of function objects: built-in functions\nand user-defined functions.  Both support the same operation (to call\nthe function), but the implementation is different, hence the\ndifferent object types.\n\nSee *Function definitions* for more information.\n',
- 'typesmapping': '\nMapping Types --- ``dict``\n**************************\n\nA *mapping* object maps *hashable* values to arbitrary objects.\nMappings are mutable objects.  There is currently only one standard\nmapping type, the *dictionary*.  (For other containers see the built\nin ``list``, ``set``, and ``tuple`` classes, and the ``collections``\nmodule.)\n\nA dictionary\'s keys are *almost* arbitrary values.  Values that are\nnot *hashable*, that is, values containing lists, dictionaries or\nother mutable types (that are compared by value rather than by object\nidentity) may not be used as keys.  Numeric types used for keys obey\nthe normal rules for numeric comparison: if two numbers compare equal\n(such as ``1`` and ``1.0``) then they can be used interchangeably to\nindex the same dictionary entry.  (Note however, that since computers\nstore floating-point numbers as approximations it is usually unwise to\nuse them as dictionary keys.)\n\nDictionaries can be created by placing a comma-separated list of\n``key: value`` pairs within braces, for example: ``{\'jack\': 4098,\n\'sjoerd\': 4127}`` or ``{4098: \'jack\', 4127: \'sjoerd\'}``, or by the\n``dict`` constructor.\n\nclass class dict([arg])\n\n   Return a new dictionary initialized from an optional positional\n   argument or from a set of keyword arguments.  If no arguments are\n   given, return a new empty dictionary.  If the positional argument\n   *arg* is a mapping object, return a dictionary mapping the same\n   keys to the same values as does the mapping object.  Otherwise the\n   positional argument must be a sequence, a container that supports\n   iteration, or an iterator object.  The elements of the argument\n   must each also be of one of those kinds, and each must in turn\n   contain exactly two objects.  The first is used as a key in the new\n   dictionary, and the second as the key\'s value.  If a given key is\n   seen more than once, the last value associated with it is retained\n   in the new dictionary.\n\n   If keyword arguments are given, the keywords themselves with their\n   associated values are added as items to the dictionary.  If a key\n   is specified both in the positional argument and as a keyword\n   argument, the value associated with the keyword is retained in the\n   dictionary.  For example, these all return a dictionary equal to\n   ``{"one": 1, "two": 2}``:\n\n   * ``dict(one=1, two=2)``\n\n   * ``dict({\'one\': 1, \'two\': 2})``\n\n   * ``dict(zip((\'one\', \'two\'), (1, 2)))``\n\n   * ``dict([[\'two\', 2], [\'one\', 1]])``\n\n   The first example only works for keys that are valid Python\n   identifiers; the others work with any valid keys.\n\n   These are the operations that dictionaries support (and therefore,\n   custom mapping types should support too):\n\n   len(d)\n\n      Return the number of items in the dictionary *d*.\n\n   d[key]\n\n      Return the item of *d* with key *key*.  Raises a ``KeyError`` if\n      *key* is not in the map.\n\n      If a subclass of dict defines a method ``__missing__()``, if the\n      key *key* is not present, the ``d[key]`` operation calls that\n      method with the key *key* as argument.  The ``d[key]`` operation\n      then returns or raises whatever is returned or raised by the\n      ``__missing__(key)`` call if the key is not present. No other\n      operations or methods invoke ``__missing__()``. If\n      ``__missing__()`` is not defined, ``KeyError`` is raised.\n      ``__missing__()`` must be a method; it cannot be an instance\n      variable:\n\n         >>> class Counter(dict):\n         ...     def __missing__(self, key):\n         ...         return 0\n         >>> c = Counter()\n         >>> c[\'red\']\n         0\n         >>> c[\'red\'] += 1\n         >>> c[\'red\']\n         1\n\n      See ``collections.Counter`` for a complete implementation\n      including other methods helpful for accumulating and managing\n      tallies.\n\n   d[key] = value\n\n      Set ``d[key]`` to *value*.\n\n   del d[key]\n\n      Remove ``d[key]`` from *d*.  Raises a ``KeyError`` if *key* is\n      not in the map.\n\n   key in d\n\n      Return ``True`` if *d* has a key *key*, else ``False``.\n\n   key not in d\n\n      Equivalent to ``not key in d``.\n\n   iter(d)\n\n      Return an iterator over the keys of the dictionary.  This is a\n      shortcut for ``iter(d.keys())``.\n\n   clear()\n\n      Remove all items from the dictionary.\n\n   copy()\n\n      Return a shallow copy of the dictionary.\n\n   classmethod fromkeys(seq[, value])\n\n      Create a new dictionary with keys from *seq* and values set to\n      *value*.\n\n      ``fromkeys()`` is a class method that returns a new dictionary.\n      *value* defaults to ``None``.\n\n   get(key[, default])\n\n      Return the value for *key* if *key* is in the dictionary, else\n      *default*. If *default* is not given, it defaults to ``None``,\n      so that this method never raises a ``KeyError``.\n\n   items()\n\n      Return a new view of the dictionary\'s items (``(key, value)``\n      pairs).  See below for documentation of view objects.\n\n   keys()\n\n      Return a new view of the dictionary\'s keys.  See below for\n      documentation of view objects.\n\n   pop(key[, default])\n\n      If *key* is in the dictionary, remove it and return its value,\n      else return *default*.  If *default* is not given and *key* is\n      not in the dictionary, a ``KeyError`` is raised.\n\n   popitem()\n\n      Remove and return an arbitrary ``(key, value)`` pair from the\n      dictionary.\n\n      ``popitem()`` is useful to destructively iterate over a\n      dictionary, as often used in set algorithms.  If the dictionary\n      is empty, calling ``popitem()`` raises a ``KeyError``.\n\n   setdefault(key[, default])\n\n      If *key* is in the dictionary, return its value.  If not, insert\n      *key* with a value of *default* and return *default*.  *default*\n      defaults to ``None``.\n\n   update([other])\n\n      Update the dictionary with the key/value pairs from *other*,\n      overwriting existing keys.  Return ``None``.\n\n      ``update()`` accepts either another dictionary object or an\n      iterable of key/value pairs (as tuples or other iterables of\n      length two).  If keyword arguments are specified, the dictionary\n      is then updated with those key/value pairs: ``d.update(red=1,\n      blue=2)``.\n\n   values()\n\n      Return a new view of the dictionary\'s values.  See below for\n      documentation of view objects.\n\n\nDictionary view objects\n=======================\n\nThe objects returned by ``dict.keys()``, ``dict.values()`` and\n``dict.items()`` are *view objects*.  They provide a dynamic view on\nthe dictionary\'s entries, which means that when the dictionary\nchanges, the view reflects these changes.\n\nDictionary views can be iterated over to yield their respective data,\nand support membership tests:\n\nlen(dictview)\n\n   Return the number of entries in the dictionary.\n\niter(dictview)\n\n   Return an iterator over the keys, values or items (represented as\n   tuples of ``(key, value)``) in the dictionary.\n\n   Keys and values are iterated over in an arbitrary order which is\n   non-random, varies across Python implementations, and depends on\n   the dictionary\'s history of insertions and deletions. If keys,\n   values and items views are iterated over with no intervening\n   modifications to the dictionary, the order of items will directly\n   correspond.  This allows the creation of ``(value, key)`` pairs\n   using ``zip()``: ``pairs = zip(d.values(), d.keys())``.  Another\n   way to create the same list is ``pairs = [(v, k) for (k, v) in\n   d.items()]``.\n\n   Iterating views while adding or deleting entries in the dictionary\n   may raise a ``RuntimeError`` or fail to iterate over all entries.\n\nx in dictview\n\n   Return ``True`` if *x* is in the underlying dictionary\'s keys,\n   values or items (in the latter case, *x* should be a ``(key,\n   value)`` tuple).\n\nKeys views are set-like since their entries are unique and hashable.\nIf all values are hashable, so that ``(key, value)`` pairs are unique\nand hashable, then the items view is also set-like.  (Values views are\nnot treated as set-like since the entries are generally not unique.)\nFor set-like views, all of the operations defined for the abstract\nbase class ``collections.Set`` are available (for example, ``==``,\n``<``, or ``^``).\n\nAn example of dictionary view usage:\n\n   >>> dishes = {\'eggs\': 2, \'sausage\': 1, \'bacon\': 1, \'spam\': 500}\n   >>> keys = dishes.keys()\n   >>> values = dishes.values()\n\n   >>> # iteration\n   >>> n = 0\n   >>> for val in values:\n   ...     n += val\n   >>> print(n)\n   504\n\n   >>> # keys and values are iterated over in the same order\n   >>> list(keys)\n   [\'eggs\', \'bacon\', \'sausage\', \'spam\']\n   >>> list(values)\n   [2, 1, 1, 500]\n\n   >>> # view objects are dynamic and reflect dict changes\n   >>> del dishes[\'eggs\']\n   >>> del dishes[\'sausage\']\n   >>> list(keys)\n   [\'spam\', \'bacon\']\n\n   >>> # set operations\n   >>> keys & {\'eggs\', \'bacon\', \'salad\'}\n   {\'bacon\'}\n   >>> keys ^ {\'sausage\', \'juice\'}\n   {\'juice\', \'eggs\', \'bacon\', \'spam\'}\n',
+ 'typesmapping': '\nMapping Types --- ``dict``\n**************************\n\nA *mapping* object maps *hashable* values to arbitrary objects.\nMappings are mutable objects.  There is currently only one standard\nmapping type, the *dictionary*.  (For other containers see the built\nin ``list``, ``set``, and ``tuple`` classes, and the ``collections``\nmodule.)\n\nA dictionary\'s keys are *almost* arbitrary values.  Values that are\nnot *hashable*, that is, values containing lists, dictionaries or\nother mutable types (that are compared by value rather than by object\nidentity) may not be used as keys.  Numeric types used for keys obey\nthe normal rules for numeric comparison: if two numbers compare equal\n(such as ``1`` and ``1.0``) then they can be used interchangeably to\nindex the same dictionary entry.  (Note however, that since computers\nstore floating-point numbers as approximations it is usually unwise to\nuse them as dictionary keys.)\n\nDictionaries can be created by placing a comma-separated list of\n``key: value`` pairs within braces, for example: ``{\'jack\': 4098,\n\'sjoerd\': 4127}`` or ``{4098: \'jack\', 4127: \'sjoerd\'}``, or by the\n``dict`` constructor.\n\nclass class dict([arg])\n\n   Return a new dictionary initialized from an optional positional\n   argument or from a set of keyword arguments.  If no arguments are\n   given, return a new empty dictionary.  If the positional argument\n   *arg* is a mapping object, return a dictionary mapping the same\n   keys to the same values as does the mapping object.  Otherwise the\n   positional argument must be a sequence, a container that supports\n   iteration, or an iterator object.  The elements of the argument\n   must each also be of one of those kinds, and each must in turn\n   contain exactly two objects.  The first is used as a key in the new\n   dictionary, and the second as the key\'s value.  If a given key is\n   seen more than once, the last value associated with it is retained\n   in the new dictionary.\n\n   If keyword arguments are given, the keywords themselves with their\n   associated values are added as items to the dictionary.  If a key\n   is specified both in the positional argument and as a keyword\n   argument, the value associated with the keyword is retained in the\n   dictionary.  For example, these all return a dictionary equal to\n   ``{"one": 1, "two": 2}``:\n\n   * ``dict(one=1, two=2)``\n\n   * ``dict({\'one\': 1, \'two\': 2})``\n\n   * ``dict(zip((\'one\', \'two\'), (1, 2)))``\n\n   * ``dict([[\'two\', 2], [\'one\', 1]])``\n\n   The first example only works for keys that are valid Python\n   identifiers; the others work with any valid keys.\n\n   These are the operations that dictionaries support (and therefore,\n   custom mapping types should support too):\n\n   len(d)\n\n      Return the number of items in the dictionary *d*.\n\n   d[key]\n\n      Return the item of *d* with key *key*.  Raises a ``KeyError`` if\n      *key* is not in the map.\n\n      If a subclass of dict defines a method ``__missing__()``, if the\n      key *key* is not present, the ``d[key]`` operation calls that\n      method with the key *key* as argument.  The ``d[key]`` operation\n      then returns or raises whatever is returned or raised by the\n      ``__missing__(key)`` call if the key is not present. No other\n      operations or methods invoke ``__missing__()``. If\n      ``__missing__()`` is not defined, ``KeyError`` is raised.\n      ``__missing__()`` must be a method; it cannot be an instance\n      variable:\n\n         >>> class Counter(dict):\n         ...     def __missing__(self, key):\n         ...         return 0\n         >>> c = Counter()\n         >>> c[\'red\']\n         0\n         >>> c[\'red\'] += 1\n         >>> c[\'red\']\n         1\n\n      See ``collections.Counter`` for a complete implementation\n      including other methods helpful for accumulating and managing\n      tallies.\n\n   d[key] = value\n\n      Set ``d[key]`` to *value*.\n\n   del d[key]\n\n      Remove ``d[key]`` from *d*.  Raises a ``KeyError`` if *key* is\n      not in the map.\n\n   key in d\n\n      Return ``True`` if *d* has a key *key*, else ``False``.\n\n   key not in d\n\n      Equivalent to ``not key in d``.\n\n   iter(d)\n\n      Return an iterator over the keys of the dictionary.  This is a\n      shortcut for ``iter(d.keys())``.\n\n   clear()\n\n      Remove all items from the dictionary.\n\n   copy()\n\n      Return a shallow copy of the dictionary.\n\n   classmethod fromkeys(seq[, value])\n\n      Create a new dictionary with keys from *seq* and values set to\n      *value*.\n\n      ``fromkeys()`` is a class method that returns a new dictionary.\n      *value* defaults to ``None``.\n\n   get(key[, default])\n\n      Return the value for *key* if *key* is in the dictionary, else\n      *default*. If *default* is not given, it defaults to ``None``,\n      so that this method never raises a ``KeyError``.\n\n   items()\n\n      Return a new view of the dictionary\'s items (``(key, value)``\n      pairs).  See below for documentation of view objects.\n\n   keys()\n\n      Return a new view of the dictionary\'s keys.  See below for\n      documentation of view objects.\n\n   pop(key[, default])\n\n      If *key* is in the dictionary, remove it and return its value,\n      else return *default*.  If *default* is not given and *key* is\n      not in the dictionary, a ``KeyError`` is raised.\n\n   popitem()\n\n      Remove and return an arbitrary ``(key, value)`` pair from the\n      dictionary.\n\n      ``popitem()`` is useful to destructively iterate over a\n      dictionary, as often used in set algorithms.  If the dictionary\n      is empty, calling ``popitem()`` raises a ``KeyError``.\n\n   setdefault(key[, default])\n\n      If *key* is in the dictionary, return its value.  If not, insert\n      *key* with a value of *default* and return *default*.  *default*\n      defaults to ``None``.\n\n   update([other])\n\n      Update the dictionary with the key/value pairs from *other*,\n      overwriting existing keys.  Return ``None``.\n\n      ``update()`` accepts either another dictionary object or an\n      iterable of key/value pairs (as tuples or other iterables of\n      length two).  If keyword arguments are specified, the dictionary\n      is then updated with those key/value pairs: ``d.update(red=1,\n      blue=2)``.\n\n   values()\n\n      Return a new view of the dictionary\'s values.  See below for\n      documentation of view objects.\n\n\nDictionary view objects\n=======================\n\nThe objects returned by ``dict.keys()``, ``dict.values()`` and\n``dict.items()`` are *view objects*.  They provide a dynamic view on\nthe dictionary\'s entries, which means that when the dictionary\nchanges, the view reflects these changes.\n\nDictionary views can be iterated over to yield their respective data,\nand support membership tests:\n\nlen(dictview)\n\n   Return the number of entries in the dictionary.\n\niter(dictview)\n\n   Return an iterator over the keys, values or items (represented as\n   tuples of ``(key, value)``) in the dictionary.\n\n   Keys and values are iterated over in an arbitrary order which is\n   non-random, varies across Python implementations, and depends on\n   the dictionary\'s history of insertions and deletions. If keys,\n   values and items views are iterated over with no intervening\n   modifications to the dictionary, the order of items will directly\n   correspond.  This allows the creation of ``(value, key)`` pairs\n   using ``zip()``: ``pairs = zip(d.values(), d.keys())``.  Another\n   way to create the same list is ``pairs = [(v, k) for (k, v) in\n   d.items()]``.\n\n   Iterating views while adding or deleting entries in the dictionary\n   may raise a ``RuntimeError`` or fail to iterate over all entries.\n\nx in dictview\n\n   Return ``True`` if *x* is in the underlying dictionary\'s keys,\n   values or items (in the latter case, *x* should be a ``(key,\n   value)`` tuple).\n\nKeys views are set-like since their entries are unique and hashable.\nIf all values are hashable, so that ``(key, value)`` pairs are unique\nand hashable, then the items view is also set-like.  (Values views are\nnot treated as set-like since the entries are generally not unique.)\nFor set-like views, all of the operations defined for the abstract\nbase class ``collections.Set`` are available (for example, ``==``,\n``<``, or ``^``).\n\nAn example of dictionary view usage:\n\n   >>> dishes = {\'eggs\': 2, \'sausage\': 1, \'bacon\': 1, \'spam\': 500}\n   >>> keys = dishes.keys()\n   >>> values = dishes.values()\n\n   >>> # iteration\n   >>> n = 0\n   >>> for val in values:\n   ...     n += val\n   >>> print(n)\n   504\n\n   >>> # keys and values are iterated over in the same order\n   >>> list(keys)\n   [\'eggs\', \'bacon\', \'sausage\', \'spam\']\n   >>> list(values)\n   [2, 1, 1, 500]\n\n   >>> # view objects are dynamic and reflect dict changes\n   >>> del dishes[\'eggs\']\n   >>> del dishes[\'sausage\']\n   >>> list(keys)\n   [\'spam\', \'bacon\']\n\n   >>> # set operations\n   >>> keys & {\'eggs\', \'bacon\', \'salad\'}\n   {\'bacon\'}\n   >>> keys ^ {\'sausage\', \'juice\'}\n   {\'juice\', \'sausage\', \'bacon\', \'spam\'}\n',
  'typesmethods': "\nMethods\n*******\n\nMethods are functions that are called using the attribute notation.\nThere are two flavors: built-in methods (such as ``append()`` on\nlists) and class instance methods.  Built-in methods are described\nwith the types that support them.\n\nIf you access a method (a function defined in a class namespace)\nthrough an instance, you get a special object: a *bound method* (also\ncalled *instance method*) object. When called, it will add the\n``self`` argument to the argument list.  Bound methods have two\nspecial read-only attributes: ``m.__self__`` is the object on which\nthe method operates, and ``m.__func__`` is the function implementing\nthe method.  Calling ``m(arg-1, arg-2, ..., arg-n)`` is completely\nequivalent to calling ``m.__func__(m.__self__, arg-1, arg-2, ...,\narg-n)``.\n\nLike function objects, bound method objects support getting arbitrary\nattributes.  However, since method attributes are actually stored on\nthe underlying function object (``meth.__func__``), setting method\nattributes on bound methods is disallowed.  Attempting to set a method\nattribute results in a ``TypeError`` being raised.  In order to set a\nmethod attribute, you need to explicitly set it on the underlying\nfunction object:\n\n   class C:\n       def method(self):\n           pass\n\n   c = C()\n   c.method.__func__.whoami = 'my name is c'\n\nSee *The standard type hierarchy* for more information.\n",
- 'typesmodules': "\nModules\n*******\n\nThe only special operation on a module is attribute access:\n``m.name``, where *m* is a module and *name* accesses a name defined\nin *m*'s symbol table. Module attributes can be assigned to.  (Note\nthat the ``import`` statement is not, strictly speaking, an operation\non a module object; ``import foo`` does not require a module object\nnamed *foo* to exist, rather it requires an (external) *definition*\nfor a module named *foo* somewhere.)\n\nA special member of every module is ``__dict__``. This is the\ndictionary containing the module's symbol table. Modifying this\ndictionary will actually change the module's symbol table, but direct\nassignment to the ``__dict__`` attribute is not possible (you can\nwrite ``m.__dict__['a'] = 1``, which defines ``m.a`` to be ``1``, but\nyou can't write ``m.__dict__ = {}``).  Modifying ``__dict__`` directly\nis not recommended.\n\nModules built into the interpreter are written like this: ``<module\n'sys' (built-in)>``.  If loaded from a file, they are written as\n``<module 'os' from '/usr/local/lib/pythonX.Y/os.pyc'>``.\n",
- 'typesseq': '\nSequence Types --- ``str``, ``bytes``, ``bytearray``, ``list``, ``tuple``, ``range``\n************************************************************************************\n\nThere are six sequence types: strings, byte sequences (``bytes``\nobjects), byte arrays (``bytearray`` objects), lists, tuples, and\nrange objects.  For other containers see the built in ``dict`` and\n``set`` classes, and the ``collections`` module.\n\nStrings contain Unicode characters.  Their literals are written in\nsingle or double quotes: ``\'xyzzy\'``, ``"frobozz"``.  See *String and\nBytes literals* for more about string literals.  In addition to the\nfunctionality described here, there are also string-specific methods\ndescribed in the *String Methods* section.\n\nBytes and bytearray objects contain single bytes -- the former is\nimmutable while the latter is a mutable sequence.  Bytes objects can\nbe constructed the constructor, ``bytes()``, and from literals; use a\n``b`` prefix with normal string syntax: ``b\'xyzzy\'``.  To construct\nbyte arrays, use the ``bytearray()`` function.\n\nWhile string objects are sequences of characters (represented by\nstrings of length 1), bytes and bytearray objects are sequences of\n*integers* (between 0 and 255), representing the ASCII value of single\nbytes.  That means that for a bytes or bytearray object *b*, ``b[0]``\nwill be an integer, while ``b[0:1]`` will be a bytes or bytearray\nobject of length 1.  The representation of bytes objects uses the\nliteral format (``b\'...\'``) since it is generally more useful than\ne.g. ``bytes([50, 19, 100])``.  You can always convert a bytes object\ninto a list of integers using ``list(b)``.\n\nAlso, while in previous Python versions, byte strings and Unicode\nstrings could be exchanged for each other rather freely (barring\nencoding issues), strings and bytes are now completely separate\nconcepts.  There\'s no implicit en-/decoding if you pass an object of\nthe wrong type.  A string always compares unequal to a bytes or\nbytearray object.\n\nLists are constructed with square brackets, separating items with\ncommas: ``[a, b, c]``.  Tuples are constructed by the comma operator\n(not within square brackets), with or without enclosing parentheses,\nbut an empty tuple must have the enclosing parentheses, such as ``a,\nb, c`` or ``()``.  A single item tuple must have a trailing comma,\nsuch as ``(d,)``.\n\nObjects of type range are created using the ``range()`` function.\nThey don\'t support concatenation or repetition, and using ``min()`` or\n``max()`` on them is inefficient.\n\nMost sequence types support the following operations.  The ``in`` and\n``not in`` operations have the same priorities as the comparison\noperations.  The ``+`` and ``*`` operations have the same priority as\nthe corresponding numeric operations. [3] Additional methods are\nprovided for *Mutable Sequence Types*.\n\nThis table lists the sequence operations sorted in ascending priority\n(operations in the same box have the same priority).  In the table,\n*s* and *t* are sequences of the same type; *n*, *i*, *j* and *k* are\nintegers.\n\n+--------------------+----------------------------------+------------+\n| Operation          | Result                           | Notes      |\n+====================+==================================+============+\n| ``x in s``         | ``True`` if an item of *s* is    | (1)        |\n|                    | equal to *x*, else ``False``     |            |\n+--------------------+----------------------------------+------------+\n| ``x not in s``     | ``False`` if an item of *s* is   | (1)        |\n|                    | equal to *x*, else ``True``      |            |\n+--------------------+----------------------------------+------------+\n| ``s + t``          | the concatenation of *s* and *t* | (6)        |\n+--------------------+----------------------------------+------------+\n| ``s * n, n * s``   | *n* shallow copies of *s*        | (2)        |\n|                    | concatenated                     |            |\n+--------------------+----------------------------------+------------+\n| ``s[i]``           | *i*\'th item of *s*, origin 0     | (3)        |\n+--------------------+----------------------------------+------------+\n| ``s[i:j]``         | slice of *s* from *i* to *j*     | (3)(4)     |\n+--------------------+----------------------------------+------------+\n| ``s[i:j:k]``       | slice of *s* from *i* to *j*     | (3)(5)     |\n|                    | with step *k*                    |            |\n+--------------------+----------------------------------+------------+\n| ``len(s)``         | length of *s*                    |            |\n+--------------------+----------------------------------+------------+\n| ``min(s)``         | smallest item of *s*             |            |\n+--------------------+----------------------------------+------------+\n| ``max(s)``         | largest item of *s*              |            |\n+--------------------+----------------------------------+------------+\n| ``s.index(i)``     | index of the first occurence of  |            |\n|                    | *i* in *s*                       |            |\n+--------------------+----------------------------------+------------+\n| ``s.count(i)``     | total number of occurences of    |            |\n|                    | *i* in *s*                       |            |\n+--------------------+----------------------------------+------------+\n\nSequence types also support comparisons.  In particular, tuples and\nlists are compared lexicographically by comparing corresponding\nelements.  This means that to compare equal, every element must\ncompare equal and the two sequences must be of the same type and have\nthe same length.  (For full details see *Comparisons* in the language\nreference.)\n\nNotes:\n\n1. When *s* is a string object, the ``in`` and ``not in`` operations\n   act like a substring test.\n\n2. Values of *n* less than ``0`` are treated as ``0`` (which yields an\n   empty sequence of the same type as *s*).  Note also that the copies\n   are shallow; nested structures are not copied.  This often haunts\n   new Python programmers; consider:\n\n   >>> lists = [[]] * 3\n   >>> lists\n   [[], [], []]\n   >>> lists[0].append(3)\n   >>> lists\n   [[3], [3], [3]]\n\n   What has happened is that ``[[]]`` is a one-element list containing\n   an empty list, so all three elements of ``[[]] * 3`` are (pointers\n   to) this single empty list.  Modifying any of the elements of\n   ``lists`` modifies this single list. You can create a list of\n   different lists this way:\n\n   >>> lists = [[] for i in range(3)]\n   >>> lists[0].append(3)\n   >>> lists[1].append(5)\n   >>> lists[2].append(7)\n   >>> lists\n   [[3], [5], [7]]\n\n3. If *i* or *j* is negative, the index is relative to the end of the\n   string: ``len(s) + i`` or ``len(s) + j`` is substituted.  But note\n   that ``-0`` is still ``0``.\n\n4. The slice of *s* from *i* to *j* is defined as the sequence of\n   items with index *k* such that ``i <= k < j``.  If *i* or *j* is\n   greater than ``len(s)``, use ``len(s)``.  If *i* is omitted or\n   ``None``, use ``0``.  If *j* is omitted or ``None``, use\n   ``len(s)``.  If *i* is greater than or equal to *j*, the slice is\n   empty.\n\n5. The slice of *s* from *i* to *j* with step *k* is defined as the\n   sequence of items with index  ``x = i + n*k`` such that ``0 <= n <\n   (j-i)/k``.  In other words, the indices are ``i``, ``i+k``,\n   ``i+2*k``, ``i+3*k`` and so on, stopping when *j* is reached (but\n   never including *j*).  If *i* or *j* is greater than ``len(s)``,\n   use ``len(s)``.  If *i* or *j* are omitted or ``None``, they become\n   "end" values (which end depends on the sign of *k*).  Note, *k*\n   cannot be zero. If *k* is ``None``, it is treated like ``1``.\n\n6. **CPython implementation detail:** If *s* and *t* are both strings,\n   some Python implementations such as CPython can usually perform an\n   in-place optimization for assignments of the form ``s = s + t`` or\n   ``s += t``.  When applicable, this optimization makes quadratic\n   run-time much less likely.  This optimization is both version and\n   implementation dependent.  For performance sensitive code, it is\n   preferable to use the ``str.join()`` method which assures\n   consistent linear concatenation performance across versions and\n   implementations.\n\n\nString Methods\n==============\n\nString objects support the methods listed below.\n\nIn addition, Python\'s strings support the sequence type methods\ndescribed in the *Sequence Types --- str, bytes, bytearray, list,\ntuple, range* section. To output formatted strings, see the *String\nFormatting* section. Also, see the ``re`` module for string functions\nbased on regular expressions.\n\nstr.capitalize()\n\n   Return a copy of the string with its first character capitalized\n   and the rest lowercased.\n\nstr.center(width[, fillchar])\n\n   Return centered in a string of length *width*. Padding is done\n   using the specified *fillchar* (default is a space).\n\nstr.count(sub[, start[, end]])\n\n   Return the number of non-overlapping occurrences of substring *sub*\n   in the range [*start*, *end*].  Optional arguments *start* and\n   *end* are interpreted as in slice notation.\n\nstr.encode(encoding="utf-8", errors="strict")\n\n   Return an encoded version of the string as a bytes object. Default\n   encoding is ``\'utf-8\'``. *errors* may be given to set a different\n   error handling scheme. The default for *errors* is ``\'strict\'``,\n   meaning that encoding errors raise a ``UnicodeError``. Other\n   possible values are ``\'ignore\'``, ``\'replace\'``,\n   ``\'xmlcharrefreplace\'``, ``\'backslashreplace\'`` and any other name\n   registered via ``codecs.register_error()``, see section *Codec Base\n   Classes*. For a list of possible encodings, see section *Standard\n   Encodings*.\n\n   Changed in version 3.1: Support for keyword arguments added.\n\nstr.endswith(suffix[, start[, end]])\n\n   Return ``True`` if the string ends with the specified *suffix*,\n   otherwise return ``False``.  *suffix* can also be a tuple of\n   suffixes to look for.  With optional *start*, test beginning at\n   that position.  With optional *end*, stop comparing at that\n   position.\n\nstr.expandtabs([tabsize])\n\n   Return a copy of the string where all tab characters are replaced\n   by one or more spaces, depending on the current column and the\n   given tab size.  The column number is reset to zero after each\n   newline occurring in the string. If *tabsize* is not given, a tab\n   size of ``8`` characters is assumed.  This doesn\'t understand other\n   non-printing characters or escape sequences.\n\nstr.find(sub[, start[, end]])\n\n   Return the lowest index in the string where substring *sub* is\n   found, such that *sub* is contained in the slice ``s[start:end]``.\n   Optional arguments *start* and *end* are interpreted as in slice\n   notation.  Return ``-1`` if *sub* is not found.\n\n   Note: The ``find()`` method should be used only if you need to know the\n     position of *sub*.  To check if *sub* is a substring or not, use\n     the ``in`` operator:\n\n        >>> \'Py\' in \'Python\'\n        True\n\nstr.format(*args, **kwargs)\n\n   Perform a string formatting operation.  The string on which this\n   method is called can contain literal text or replacement fields\n   delimited by braces ``{}``.  Each replacement field contains either\n   the numeric index of a positional argument, or the name of a\n   keyword argument.  Returns a copy of the string where each\n   replacement field is replaced with the string value of the\n   corresponding argument.\n\n   >>> "The sum of 1 + 2 is {0}".format(1+2)\n   \'The sum of 1 + 2 is 3\'\n\n   See *Format String Syntax* for a description of the various\n   formatting options that can be specified in format strings.\n\nstr.format_map(mapping)\n\n   Similar to ``str.format(**mapping)``, except that ``mapping`` is\n   used directly and not copied to a ``dict`` .  This is useful if for\n   example ``mapping`` is a dict subclass:\n\n   >>> class Default(dict):\n   ...     def __missing__(self, key):\n   ...         return key\n   ...\n   >>> \'{name} was born in {country}\'.format_map(Default(name=\'Guido\'))\n   \'Guido was born in country\'\n\n   New in version 3.2.\n\nstr.index(sub[, start[, end]])\n\n   Like ``find()``, but raise ``ValueError`` when the substring is not\n   found.\n\nstr.isalnum()\n\n   Return true if all characters in the string are alphanumeric and\n   there is at least one character, false otherwise.  A character\n   ``c`` is alphanumeric if one of the following returns ``True``:\n   ``c.isalpha()``, ``c.isdecimal()``, ``c.isdigit()``, or\n   ``c.isnumeric()``.\n\nstr.isalpha()\n\n   Return true if all characters in the string are alphabetic and\n   there is at least one character, false otherwise.  Alphabetic\n   characters are those characters defined in the Unicode character\n   database as "Letter", i.e., those with general category property\n   being one of "Lm", "Lt", "Lu", "Ll", or "Lo".  Note that this is\n   different from the "Alphabetic" property defined in the Unicode\n   Standard.\n\nstr.isdecimal()\n\n   Return true if all characters in the string are decimal characters\n   and there is at least one character, false otherwise. Decimal\n   characters are those from general category "Nd". This category\n   includes digit characters, and all characters that that can be used\n   to form decimal-radix numbers, e.g. U+0660, ARABIC-INDIC DIGIT\n   ZERO.\n\nstr.isdigit()\n\n   Return true if all characters in the string are digits and there is\n   at least one character, false otherwise.  Digits include decimal\n   characters and digits that need special handling, such as the\n   compatibility superscript digits.  Formally, a digit is a character\n   that has the property value Numeric_Type=Digit or\n   Numeric_Type=Decimal.\n\nstr.isidentifier()\n\n   Return true if the string is a valid identifier according to the\n   language definition, section *Identifiers and keywords*.\n\nstr.islower()\n\n   Return true if all cased characters in the string are lowercase and\n   there is at least one cased character, false otherwise.  Cased\n   characters are those with general category property being one of\n   "Lu", "Ll", or "Lt" and lowercase characters are those with general\n   category property "Ll".\n\nstr.isnumeric()\n\n   Return true if all characters in the string are numeric characters,\n   and there is at least one character, false otherwise. Numeric\n   characters include digit characters, and all characters that have\n   the Unicode numeric value property, e.g. U+2155, VULGAR FRACTION\n   ONE FIFTH.  Formally, numeric characters are those with the\n   property value Numeric_Type=Digit, Numeric_Type=Decimal or\n   Numeric_Type=Numeric.\n\nstr.isprintable()\n\n   Return true if all characters in the string are printable or the\n   string is empty, false otherwise.  Nonprintable characters are\n   those characters defined in the Unicode character database as\n   "Other" or "Separator", excepting the ASCII space (0x20) which is\n   considered printable.  (Note that printable characters in this\n   context are those which should not be escaped when ``repr()`` is\n   invoked on a string.  It has no bearing on the handling of strings\n   written to ``sys.stdout`` or ``sys.stderr``.)\n\nstr.isspace()\n\n   Return true if there are only whitespace characters in the string\n   and there is at least one character, false otherwise.  Whitespace\n   characters  are those characters defined in the Unicode character\n   database as "Other" or "Separator" and those with bidirectional\n   property being one of "WS", "B", or "S".\n\nstr.istitle()\n\n   Return true if the string is a titlecased string and there is at\n   least one character, for example uppercase characters may only\n   follow uncased characters and lowercase characters only cased ones.\n   Return false otherwise.\n\nstr.isupper()\n\n   Return true if all cased characters in the string are uppercase and\n   there is at least one cased character, false otherwise. Cased\n   characters are those with general category property being one of\n   "Lu", "Ll", or "Lt" and uppercase characters are those with general\n   category property "Lu".\n\nstr.join(iterable)\n\n   Return a string which is the concatenation of the strings in the\n   *iterable* *iterable*.  A ``TypeError`` will be raised if there are\n   any non-string values in *seq*, including ``bytes`` objects.  The\n   separator between elements is the string providing this method.\n\nstr.ljust(width[, fillchar])\n\n   Return the string left justified in a string of length *width*.\n   Padding is done using the specified *fillchar* (default is a\n   space).  The original string is returned if *width* is less than\n   ``len(s)``.\n\nstr.lower()\n\n   Return a copy of the string converted to lowercase.\n\nstr.lstrip([chars])\n\n   Return a copy of the string with leading characters removed.  The\n   *chars* argument is a string specifying the set of characters to be\n   removed.  If omitted or ``None``, the *chars* argument defaults to\n   removing whitespace.  The *chars* argument is not a prefix; rather,\n   all combinations of its values are stripped:\n\n   >>> \'   spacious   \'.lstrip()\n   \'spacious   \'\n   >>> \'www.example.com\'.lstrip(\'cmowz.\')\n   \'example.com\'\n\nstatic str.maketrans(x[, y[, z]])\n\n   This static method returns a translation table usable for\n   ``str.translate()``.\n\n   If there is only one argument, it must be a dictionary mapping\n   Unicode ordinals (integers) or characters (strings of length 1) to\n   Unicode ordinals, strings (of arbitrary lengths) or None.\n   Character keys will then be converted to ordinals.\n\n   If there are two arguments, they must be strings of equal length,\n   and in the resulting dictionary, each character in x will be mapped\n   to the character at the same position in y.  If there is a third\n   argument, it must be a string, whose characters will be mapped to\n   None in the result.\n\nstr.partition(sep)\n\n   Split the string at the first occurrence of *sep*, and return a\n   3-tuple containing the part before the separator, the separator\n   itself, and the part after the separator.  If the separator is not\n   found, return a 3-tuple containing the string itself, followed by\n   two empty strings.\n\nstr.replace(old, new[, count])\n\n   Return a copy of the string with all occurrences of substring *old*\n   replaced by *new*.  If the optional argument *count* is given, only\n   the first *count* occurrences are replaced.\n\nstr.rfind(sub[, start[, end]])\n\n   Return the highest index in the string where substring *sub* is\n   found, such that *sub* is contained within ``s[start:end]``.\n   Optional arguments *start* and *end* are interpreted as in slice\n   notation.  Return ``-1`` on failure.\n\nstr.rindex(sub[, start[, end]])\n\n   Like ``rfind()`` but raises ``ValueError`` when the substring *sub*\n   is not found.\n\nstr.rjust(width[, fillchar])\n\n   Return the string right justified in a string of length *width*.\n   Padding is done using the specified *fillchar* (default is a\n   space). The original string is returned if *width* is less than\n   ``len(s)``.\n\nstr.rpartition(sep)\n\n   Split the string at the last occurrence of *sep*, and return a\n   3-tuple containing the part before the separator, the separator\n   itself, and the part after the separator.  If the separator is not\n   found, return a 3-tuple containing two empty strings, followed by\n   the string itself.\n\nstr.rsplit([sep[, maxsplit]])\n\n   Return a list of the words in the string, using *sep* as the\n   delimiter string. If *maxsplit* is given, at most *maxsplit* splits\n   are done, the *rightmost* ones.  If *sep* is not specified or\n   ``None``, any whitespace string is a separator.  Except for\n   splitting from the right, ``rsplit()`` behaves like ``split()``\n   which is described in detail below.\n\nstr.rstrip([chars])\n\n   Return a copy of the string with trailing characters removed.  The\n   *chars* argument is a string specifying the set of characters to be\n   removed.  If omitted or ``None``, the *chars* argument defaults to\n   removing whitespace.  The *chars* argument is not a suffix; rather,\n   all combinations of its values are stripped:\n\n   >>> \'   spacious   \'.rstrip()\n   \'   spacious\'\n   >>> \'mississippi\'.rstrip(\'ipz\')\n   \'mississ\'\n\nstr.split([sep[, maxsplit]])\n\n   Return a list of the words in the string, using *sep* as the\n   delimiter string.  If *maxsplit* is given, at most *maxsplit*\n   splits are done (thus, the list will have at most ``maxsplit+1``\n   elements).  If *maxsplit* is not specified, then there is no limit\n   on the number of splits (all possible splits are made).\n\n   If *sep* is given, consecutive delimiters are not grouped together\n   and are deemed to delimit empty strings (for example,\n   ``\'1,,2\'.split(\',\')`` returns ``[\'1\', \'\', \'2\']``).  The *sep*\n   argument may consist of multiple characters (for example,\n   ``\'1<>2<>3\'.split(\'<>\')`` returns ``[\'1\', \'2\', \'3\']``). Splitting\n   an empty string with a specified separator returns ``[\'\']``.\n\n   If *sep* is not specified or is ``None``, a different splitting\n   algorithm is applied: runs of consecutive whitespace are regarded\n   as a single separator, and the result will contain no empty strings\n   at the start or end if the string has leading or trailing\n   whitespace.  Consequently, splitting an empty string or a string\n   consisting of just whitespace with a ``None`` separator returns\n   ``[]``.\n\n   For example, ``\' 1  2   3  \'.split()`` returns ``[\'1\', \'2\', \'3\']``,\n   and ``\'  1  2   3  \'.split(None, 1)`` returns ``[\'1\', \'2   3  \']``.\n\nstr.splitlines([keepends])\n\n   Return a list of the lines in the string, breaking at line\n   boundaries.  Line breaks are not included in the resulting list\n   unless *keepends* is given and true.\n\nstr.startswith(prefix[, start[, end]])\n\n   Return ``True`` if string starts with the *prefix*, otherwise\n   return ``False``. *prefix* can also be a tuple of prefixes to look\n   for.  With optional *start*, test string beginning at that\n   position.  With optional *end*, stop comparing string at that\n   position.\n\nstr.strip([chars])\n\n   Return a copy of the string with the leading and trailing\n   characters removed. The *chars* argument is a string specifying the\n   set of characters to be removed. If omitted or ``None``, the\n   *chars* argument defaults to removing whitespace. The *chars*\n   argument is not a prefix or suffix; rather, all combinations of its\n   values are stripped:\n\n   >>> \'   spacious   \'.strip()\n   \'spacious\'\n   >>> \'www.example.com\'.strip(\'cmowz.\')\n   \'example\'\n\nstr.swapcase()\n\n   Return a copy of the string with uppercase characters converted to\n   lowercase and vice versa.\n\nstr.title()\n\n   Return a titlecased version of the string where words start with an\n   uppercase character and the remaining characters are lowercase.\n\n   The algorithm uses a simple language-independent definition of a\n   word as groups of consecutive letters.  The definition works in\n   many contexts but it means that apostrophes in contractions and\n   possessives form word boundaries, which may not be the desired\n   result:\n\n      >>> "they\'re bill\'s friends from the UK".title()\n      "They\'Re Bill\'S Friends From The Uk"\n\n   A workaround for apostrophes can be constructed using regular\n   expressions:\n\n      >>> import re\n      >>> def titlecase(s):\n              return re.sub(r"[A-Za-z]+(\'[A-Za-z]+)?",\n                            lambda mo: mo.group(0)[0].upper() +\n                                       mo.group(0)[1:].lower(),\n                            s)\n\n      >>> titlecase("they\'re bill\'s friends.")\n      "They\'re Bill\'s Friends."\n\nstr.translate(map)\n\n   Return a copy of the *s* where all characters have been mapped\n   through the *map* which must be a dictionary of Unicode ordinals\n   (integers) to Unicode ordinals, strings or ``None``.  Unmapped\n   characters are left untouched. Characters mapped to ``None`` are\n   deleted.\n\n   You can use ``str.maketrans()`` to create a translation map from\n   character-to-character mappings in different formats.\n\n   Note: An even more flexible approach is to create a custom character\n     mapping codec using the ``codecs`` module (see\n     ``encodings.cp1251`` for an example).\n\nstr.upper()\n\n   Return a copy of the string converted to uppercase.\n\nstr.zfill(width)\n\n   Return the numeric string left filled with zeros in a string of\n   length *width*.  A sign prefix is handled correctly.  The original\n   string is returned if *width* is less than ``len(s)``.\n\n\nOld String Formatting Operations\n================================\n\nNote: The formatting operations described here are obsolete and may go\n  away in future versions of Python.  Use the new *String Formatting*\n  in new code.\n\nString objects have one unique built-in operation: the ``%`` operator\n(modulo). This is also known as the string *formatting* or\n*interpolation* operator. Given ``format % values`` (where *format* is\na string), ``%`` conversion specifications in *format* are replaced\nwith zero or more elements of *values*. The effect is similar to the\nusing ``sprintf()`` in the C language.\n\nIf *format* requires a single argument, *values* may be a single non-\ntuple object. [4]  Otherwise, *values* must be a tuple with exactly\nthe number of items specified by the format string, or a single\nmapping object (for example, a dictionary).\n\nA conversion specifier contains two or more characters and has the\nfollowing components, which must occur in this order:\n\n1. The ``\'%\'`` character, which marks the start of the specifier.\n\n2. Mapping key (optional), consisting of a parenthesised sequence of\n   characters (for example, ``(somename)``).\n\n3. Conversion flags (optional), which affect the result of some\n   conversion types.\n\n4. Minimum field width (optional).  If specified as an ``\'*\'``\n   (asterisk), the actual width is read from the next element of the\n   tuple in *values*, and the object to convert comes after the\n   minimum field width and optional precision.\n\n5. Precision (optional), given as a ``\'.\'`` (dot) followed by the\n   precision.  If specified as ``\'*\'`` (an asterisk), the actual width\n   is read from the next element of the tuple in *values*, and the\n   value to convert comes after the precision.\n\n6. Length modifier (optional).\n\n7. Conversion type.\n\nWhen the right argument is a dictionary (or other mapping type), then\nthe formats in the string *must* include a parenthesised mapping key\ninto that dictionary inserted immediately after the ``\'%\'`` character.\nThe mapping key selects the value to be formatted from the mapping.\nFor example:\n\n>>> print(\'%(language)s has %(number)03d quote types.\' %\n...       {\'language\': "Python", "number": 2})\nPython has 002 quote types.\n\nIn this case no ``*`` specifiers may occur in a format (since they\nrequire a sequential parameter list).\n\nThe conversion flag characters are:\n\n+-----------+-----------------------------------------------------------------------+\n| Flag      | Meaning                                                               |\n+===========+=======================================================================+\n| ``\'#\'``   | The value conversion will use the "alternate form" (where defined     |\n|           | below).                                                               |\n+-----------+-----------------------------------------------------------------------+\n| ``\'0\'``   | The conversion will be zero padded for numeric values.                |\n+-----------+-----------------------------------------------------------------------+\n| ``\'-\'``   | The converted value is left adjusted (overrides the ``\'0\'``           |\n|           | conversion if both are given).                                        |\n+-----------+-----------------------------------------------------------------------+\n| ``\' \'``   | (a space) A blank should be left before a positive number (or empty   |\n|           | string) produced by a signed conversion.                              |\n+-----------+-----------------------------------------------------------------------+\n| ``\'+\'``   | A sign character (``\'+\'`` or ``\'-\'``) will precede the conversion     |\n|           | (overrides a "space" flag).                                           |\n+-----------+-----------------------------------------------------------------------+\n\nA length modifier (``h``, ``l``, or ``L``) may be present, but is\nignored as it is not necessary for Python -- so e.g. ``%ld`` is\nidentical to ``%d``.\n\nThe conversion types are:\n\n+--------------+-------------------------------------------------------+---------+\n| Conversion   | Meaning                                               | Notes   |\n+==============+=======================================================+=========+\n| ``\'d\'``      | Signed integer decimal.                               |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'i\'``      | Signed integer decimal.                               |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'o\'``      | Signed octal value.                                   | (1)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'u\'``      | Obsolete type -- it is identical to ``\'d\'``.          | (7)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'x\'``      | Signed hexadecimal (lowercase).                       | (2)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'X\'``      | Signed hexadecimal (uppercase).                       | (2)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'e\'``      | Floating point exponential format (lowercase).        | (3)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'E\'``      | Floating point exponential format (uppercase).        | (3)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'f\'``      | Floating point decimal format.                        | (3)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'F\'``      | Floating point decimal format.                        | (3)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'g\'``      | Floating point format. Uses lowercase exponential     | (4)     |\n|              | format if exponent is less than -4 or not less than   |         |\n|              | precision, decimal format otherwise.                  |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'G\'``      | Floating point format. Uses uppercase exponential     | (4)     |\n|              | format if exponent is less than -4 or not less than   |         |\n|              | precision, decimal format otherwise.                  |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'c\'``      | Single character (accepts integer or single character |         |\n|              | string).                                              |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'r\'``      | String (converts any Python object using ``repr()``). | (5)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'s\'``      | String (converts any Python object using ``str()``).  |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'%\'``      | No argument is converted, results in a ``\'%\'``        |         |\n|              | character in the result.                              |         |\n+--------------+-------------------------------------------------------+---------+\n\nNotes:\n\n1. The alternate form causes a leading zero (``\'0\'``) to be inserted\n   between left-hand padding and the formatting of the number if the\n   leading character of the result is not already a zero.\n\n2. The alternate form causes a leading ``\'0x\'`` or ``\'0X\'`` (depending\n   on whether the ``\'x\'`` or ``\'X\'`` format was used) to be inserted\n   between left-hand padding and the formatting of the number if the\n   leading character of the result is not already a zero.\n\n3. The alternate form causes the result to always contain a decimal\n   point, even if no digits follow it.\n\n   The precision determines the number of digits after the decimal\n   point and defaults to 6.\n\n4. The alternate form causes the result to always contain a decimal\n   point, and trailing zeroes are not removed as they would otherwise\n   be.\n\n   The precision determines the number of significant digits before\n   and after the decimal point and defaults to 6.\n\n5. The precision determines the maximal number of characters used.\n\n1. See **PEP 237**.\n\nSince Python strings have an explicit length, ``%s`` conversions do\nnot assume that ``\'\\0\'`` is the end of the string.\n\nChanged in version 3.1: ``%f`` conversions for numbers whose absolute\nvalue is over 1e50 are no longer replaced by ``%g`` conversions.\n\nAdditional string operations are defined in standard modules\n``string`` and ``re``.\n\n\nRange Type\n==========\n\nThe ``range`` type is an immutable sequence which is commonly used for\nlooping.  The advantage of the ``range`` type is that an ``range``\nobject will always take the same amount of memory, no matter the size\nof the range it represents.\n\nRange objects have relatively little behavior: they support indexing,\ncontains, iteration, the ``len()`` function, and the following\nmethods:\n\nrange.count(x)\n\n   Return the number of *i*\'s for which ``s[i] == x``.\n\n      New in version 3.2.\n\nrange.index(x)\n\n   Return the smallest *i* such that ``s[i] == x``.  Raises\n   ``ValueError`` when *x* is not in the range.\n\n      New in version 3.2.\n\n\nMutable Sequence Types\n======================\n\nList and bytearray objects support additional operations that allow\nin-place modification of the object.  Other mutable sequence types\n(when added to the language) should also support these operations.\nStrings and tuples are immutable sequence types: such objects cannot\nbe modified once created. The following operations are defined on\nmutable sequence types (where *x* is an arbitrary object).\n\nNote that while lists allow their items to be of any type, bytearray\nobject "items" are all integers in the range 0 <= x < 256.\n\n+--------------------------------+----------------------------------+-----------------------+\n| Operation                      | Result                           | Notes                 |\n+================================+==================================+=======================+\n| ``s[i] = x``                   | item *i* of *s* is replaced by   |                       |\n|                                | *x*                              |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j] = t``                 | slice of *s* from *i* to *j* is  |                       |\n|                                | replaced by the contents of the  |                       |\n|                                | iterable *t*                     |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j]``                 | same as ``s[i:j] = []``          |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j:k] = t``               | the elements of ``s[i:j:k]`` are | (1)                   |\n|                                | replaced by those of *t*         |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j:k]``               | removes the elements of          |                       |\n|                                | ``s[i:j:k]`` from the list       |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.append(x)``                | same as ``s[len(s):len(s)] =     |                       |\n|                                | [x]``                            |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.extend(x)``                | same as ``s[len(s):len(s)] = x`` | (2)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.count(x)``                 | return number of *i*\'s for which |                       |\n|                                | ``s[i] == x``                    |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.index(x[, i[, j]])``       | return smallest *k* such that    | (3)                   |\n|                                | ``s[k] == x`` and ``i <= k < j`` |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.insert(i, x)``             | same as ``s[i:i] = [x]``         | (4)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.pop([i])``                 | same as ``x = s[i]; del s[i];    | (5)                   |\n|                                | return x``                       |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.remove(x)``                | same as ``del s[s.index(x)]``    | (3)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.reverse()``                | reverses the items of *s* in     | (6)                   |\n|                                | place                            |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.sort([key[, reverse]])``   | sort the items of *s* in place   | (6), (7), (8)         |\n+--------------------------------+----------------------------------+-----------------------+\n\nNotes:\n\n1. *t* must have the same length as the slice it is replacing.\n\n2. *x* can be any iterable object.\n\n3. Raises ``ValueError`` when *x* is not found in *s*. When a negative\n   index is passed as the second or third parameter to the ``index()``\n   method, the sequence length is added, as for slice indices.  If it\n   is still negative, it is truncated to zero, as for slice indices.\n\n4. When a negative index is passed as the first parameter to the\n   ``insert()`` method, the sequence length is added, as for slice\n   indices.  If it is still negative, it is truncated to zero, as for\n   slice indices.\n\n5. The optional argument *i* defaults to ``-1``, so that by default\n   the last item is removed and returned.\n\n6. The ``sort()`` and ``reverse()`` methods modify the sequence in\n   place for economy of space when sorting or reversing a large\n   sequence.  To remind you that they operate by side effect, they\n   don\'t return the sorted or reversed sequence.\n\n7. The ``sort()`` method takes optional arguments for controlling the\n   comparisons.  Each must be specified as a keyword argument.\n\n   *key* specifies a function of one argument that is used to extract\n   a comparison key from each list element: ``key=str.lower``.  The\n   default value is ``None``. Use ``functools.cmp_to_key()`` to\n   convert an old-style *cmp* function to a *key* function.\n\n   *reverse* is a boolean value.  If set to ``True``, then the list\n   elements are sorted as if each comparison were reversed.\n\n   The ``sort()`` method is guaranteed to be stable.  A sort is stable\n   if it guarantees not to change the relative order of elements that\n   compare equal --- this is helpful for sorting in multiple passes\n   (for example, sort by department, then by salary grade).\n\n   **CPython implementation detail:** While a list is being sorted,\n   the effect of attempting to mutate, or even inspect, the list is\n   undefined.  The C implementation of Python makes the list appear\n   empty for the duration, and raises ``ValueError`` if it can detect\n   that the list has been mutated during a sort.\n\n8. ``sort()`` is not supported by ``bytearray`` objects.\n\n\nBytes and Byte Array Methods\n============================\n\nBytes and bytearray objects, being "strings of bytes", have all\nmethods found on strings, with the exception of ``encode()``,\n``format()`` and ``isidentifier()``, which do not make sense with\nthese types.  For converting the objects to strings, they have a\n``decode()`` method.\n\nWherever one of these methods needs to interpret the bytes as\ncharacters (e.g. the ``is...()`` methods), the ASCII character set is\nassumed.\n\nNote: The methods on bytes and bytearray objects don\'t accept strings as\n  their arguments, just as the methods on strings don\'t accept bytes\n  as their arguments.  For example, you have to write\n\n     a = "abc"\n     b = a.replace("a", "f")\n\n  and\n\n     a = b"abc"\n     b = a.replace(b"a", b"f")\n\nbytes.decode(encoding="utf-8", errors="strict")\nbytearray.decode(encoding="utf-8", errors="strict")\n\n   Return a string decoded from the given bytes.  Default encoding is\n   ``\'utf-8\'``. *errors* may be given to set a different error\n   handling scheme.  The default for *errors* is ``\'strict\'``, meaning\n   that encoding errors raise a ``UnicodeError``.  Other possible\n   values are ``\'ignore\'``, ``\'replace\'`` and any other name\n   registered via ``codecs.register_error()``, see section *Codec Base\n   Classes*. For a list of possible encodings, see section *Standard\n   Encodings*.\n\n   Changed in version 3.1: Added support for keyword arguments.\n\nThe bytes and bytearray types have an additional class method:\n\nclassmethod bytes.fromhex(string)\nclassmethod bytearray.fromhex(string)\n\n   This ``bytes`` class method returns a bytes or bytearray object,\n   decoding the given string object.  The string must contain two\n   hexadecimal digits per byte, spaces are ignored.\n\n   >>> bytes.fromhex(\'f0 f1f2  \')\n   b\'\\xf0\\xf1\\xf2\'\n\nThe maketrans and translate methods differ in semantics from the\nversions available on strings:\n\nbytes.translate(table[, delete])\nbytearray.translate(table[, delete])\n\n   Return a copy of the bytes or bytearray object where all bytes\n   occurring in the optional argument *delete* are removed, and the\n   remaining bytes have been mapped through the given translation\n   table, which must be a bytes object of length 256.\n\n   You can use the ``bytes.maketrans()`` method to create a\n   translation table.\n\n   Set the *table* argument to ``None`` for translations that only\n   delete characters:\n\n      >>> b\'read this short text\'.translate(None, b\'aeiou\')\n      b\'rd ths shrt txt\'\n\nstatic bytes.maketrans(from, to)\nstatic bytearray.maketrans(from, to)\n\n   This static method returns a translation table usable for\n   ``bytes.translate()`` that will map each character in *from* into\n   the character at the same position in *to*; *from* and *to* must be\n   bytes objects and have the same length.\n\n   New in version 3.1.\n',
+ 'typesmodules': "\nModules\n*******\n\nThe only special operation on a module is attribute access:\n``m.name``, where *m* is a module and *name* accesses a name defined\nin *m*'s symbol table. Module attributes can be assigned to.  (Note\nthat the ``import`` statement is not, strictly speaking, an operation\non a module object; ``import foo`` does not require a module object\nnamed *foo* to exist, rather it requires an (external) *definition*\nfor a module named *foo* somewhere.)\n\nA special attribute of every module is ``__dict__``. This is the\ndictionary containing the module's symbol table. Modifying this\ndictionary will actually change the module's symbol table, but direct\nassignment to the ``__dict__`` attribute is not possible (you can\nwrite ``m.__dict__['a'] = 1``, which defines ``m.a`` to be ``1``, but\nyou can't write ``m.__dict__ = {}``).  Modifying ``__dict__`` directly\nis not recommended.\n\nModules built into the interpreter are written like this: ``<module\n'sys' (built-in)>``.  If loaded from a file, they are written as\n``<module 'os' from '/usr/local/lib/pythonX.Y/os.pyc'>``.\n",
+ 'typesseq': '\nSequence Types --- ``str``, ``bytes``, ``bytearray``, ``list``, ``tuple``, ``range``\n************************************************************************************\n\nThere are six sequence types: strings, byte sequences (``bytes``\nobjects), byte arrays (``bytearray`` objects), lists, tuples, and\nrange objects.  For other containers see the built in ``dict`` and\n``set`` classes, and the ``collections`` module.\n\nStrings contain Unicode characters.  Their literals are written in\nsingle or double quotes: ``\'xyzzy\'``, ``"frobozz"``.  See *String and\nBytes literals* for more about string literals.  In addition to the\nfunctionality described here, there are also string-specific methods\ndescribed in the *String Methods* section.\n\nBytes and bytearray objects contain single bytes -- the former is\nimmutable while the latter is a mutable sequence.  Bytes objects can\nbe constructed the constructor, ``bytes()``, and from literals; use a\n``b`` prefix with normal string syntax: ``b\'xyzzy\'``.  To construct\nbyte arrays, use the ``bytearray()`` function.\n\nWhile string objects are sequences of characters (represented by\nstrings of length 1), bytes and bytearray objects are sequences of\n*integers* (between 0 and 255), representing the ASCII value of single\nbytes.  That means that for a bytes or bytearray object *b*, ``b[0]``\nwill be an integer, while ``b[0:1]`` will be a bytes or bytearray\nobject of length 1.  The representation of bytes objects uses the\nliteral format (``b\'...\'``) since it is generally more useful than\ne.g. ``bytes([50, 19, 100])``.  You can always convert a bytes object\ninto a list of integers using ``list(b)``.\n\nAlso, while in previous Python versions, byte strings and Unicode\nstrings could be exchanged for each other rather freely (barring\nencoding issues), strings and bytes are now completely separate\nconcepts.  There\'s no implicit en-/decoding if you pass an object of\nthe wrong type.  A string always compares unequal to a bytes or\nbytearray object.\n\nLists are constructed with square brackets, separating items with\ncommas: ``[a, b, c]``.  Tuples are constructed by the comma operator\n(not within square brackets), with or without enclosing parentheses,\nbut an empty tuple must have the enclosing parentheses, such as ``a,\nb, c`` or ``()``.  A single item tuple must have a trailing comma,\nsuch as ``(d,)``.\n\nObjects of type range are created using the ``range()`` function.\nThey don\'t support concatenation or repetition, and using ``min()`` or\n``max()`` on them is inefficient.\n\nMost sequence types support the following operations.  The ``in`` and\n``not in`` operations have the same priorities as the comparison\noperations.  The ``+`` and ``*`` operations have the same priority as\nthe corresponding numeric operations. [3] Additional methods are\nprovided for *Mutable Sequence Types*.\n\nThis table lists the sequence operations sorted in ascending priority\n(operations in the same box have the same priority).  In the table,\n*s* and *t* are sequences of the same type; *n*, *i*, *j* and *k* are\nintegers.\n\n+--------------------+----------------------------------+------------+\n| Operation          | Result                           | Notes      |\n+====================+==================================+============+\n| ``x in s``         | ``True`` if an item of *s* is    | (1)        |\n|                    | equal to *x*, else ``False``     |            |\n+--------------------+----------------------------------+------------+\n| ``x not in s``     | ``False`` if an item of *s* is   | (1)        |\n|                    | equal to *x*, else ``True``      |            |\n+--------------------+----------------------------------+------------+\n| ``s + t``          | the concatenation of *s* and *t* | (6)        |\n+--------------------+----------------------------------+------------+\n| ``s * n, n * s``   | *n* shallow copies of *s*        | (2)        |\n|                    | concatenated                     |            |\n+--------------------+----------------------------------+------------+\n| ``s[i]``           | *i*\'th item of *s*, origin 0     | (3)        |\n+--------------------+----------------------------------+------------+\n| ``s[i:j]``         | slice of *s* from *i* to *j*     | (3)(4)     |\n+--------------------+----------------------------------+------------+\n| ``s[i:j:k]``       | slice of *s* from *i* to *j*     | (3)(5)     |\n|                    | with step *k*                    |            |\n+--------------------+----------------------------------+------------+\n| ``len(s)``         | length of *s*                    |            |\n+--------------------+----------------------------------+------------+\n| ``min(s)``         | smallest item of *s*             |            |\n+--------------------+----------------------------------+------------+\n| ``max(s)``         | largest item of *s*              |            |\n+--------------------+----------------------------------+------------+\n| ``s.index(i)``     | index of the first occurence of  |            |\n|                    | *i* in *s*                       |            |\n+--------------------+----------------------------------+------------+\n| ``s.count(i)``     | total number of occurences of    |            |\n|                    | *i* in *s*                       |            |\n+--------------------+----------------------------------+------------+\n\nSequence types also support comparisons.  In particular, tuples and\nlists are compared lexicographically by comparing corresponding\nelements.  This means that to compare equal, every element must\ncompare equal and the two sequences must be of the same type and have\nthe same length.  (For full details see *Comparisons* in the language\nreference.)\n\nNotes:\n\n1. When *s* is a string object, the ``in`` and ``not in`` operations\n   act like a substring test.\n\n2. Values of *n* less than ``0`` are treated as ``0`` (which yields an\n   empty sequence of the same type as *s*).  Note also that the copies\n   are shallow; nested structures are not copied.  This often haunts\n   new Python programmers; consider:\n\n   >>> lists = [[]] * 3\n   >>> lists\n   [[], [], []]\n   >>> lists[0].append(3)\n   >>> lists\n   [[3], [3], [3]]\n\n   What has happened is that ``[[]]`` is a one-element list containing\n   an empty list, so all three elements of ``[[]] * 3`` are (pointers\n   to) this single empty list.  Modifying any of the elements of\n   ``lists`` modifies this single list. You can create a list of\n   different lists this way:\n\n   >>> lists = [[] for i in range(3)]\n   >>> lists[0].append(3)\n   >>> lists[1].append(5)\n   >>> lists[2].append(7)\n   >>> lists\n   [[3], [5], [7]]\n\n3. If *i* or *j* is negative, the index is relative to the end of the\n   string: ``len(s) + i`` or ``len(s) + j`` is substituted.  But note\n   that ``-0`` is still ``0``.\n\n4. The slice of *s* from *i* to *j* is defined as the sequence of\n   items with index *k* such that ``i <= k < j``.  If *i* or *j* is\n   greater than ``len(s)``, use ``len(s)``.  If *i* is omitted or\n   ``None``, use ``0``.  If *j* is omitted or ``None``, use\n   ``len(s)``.  If *i* is greater than or equal to *j*, the slice is\n   empty.\n\n5. The slice of *s* from *i* to *j* with step *k* is defined as the\n   sequence of items with index  ``x = i + n*k`` such that ``0 <= n <\n   (j-i)/k``.  In other words, the indices are ``i``, ``i+k``,\n   ``i+2*k``, ``i+3*k`` and so on, stopping when *j* is reached (but\n   never including *j*).  If *i* or *j* is greater than ``len(s)``,\n   use ``len(s)``.  If *i* or *j* are omitted or ``None``, they become\n   "end" values (which end depends on the sign of *k*).  Note, *k*\n   cannot be zero. If *k* is ``None``, it is treated like ``1``.\n\n6. **CPython implementation detail:** If *s* and *t* are both strings,\n   some Python implementations such as CPython can usually perform an\n   in-place optimization for assignments of the form ``s = s + t`` or\n   ``s += t``.  When applicable, this optimization makes quadratic\n   run-time much less likely.  This optimization is both version and\n   implementation dependent.  For performance sensitive code, it is\n   preferable to use the ``str.join()`` method which assures\n   consistent linear concatenation performance across versions and\n   implementations.\n\n\nString Methods\n==============\n\nString objects support the methods listed below.\n\nIn addition, Python\'s strings support the sequence type methods\ndescribed in the *Sequence Types --- str, bytes, bytearray, list,\ntuple, range* section. To output formatted strings, see the *String\nFormatting* section. Also, see the ``re`` module for string functions\nbased on regular expressions.\n\nstr.capitalize()\n\n   Return a copy of the string with its first character capitalized\n   and the rest lowercased.\n\nstr.center(width[, fillchar])\n\n   Return centered in a string of length *width*. Padding is done\n   using the specified *fillchar* (default is a space).\n\nstr.count(sub[, start[, end]])\n\n   Return the number of non-overlapping occurrences of substring *sub*\n   in the range [*start*, *end*].  Optional arguments *start* and\n   *end* are interpreted as in slice notation.\n\nstr.encode(encoding="utf-8", errors="strict")\n\n   Return an encoded version of the string as a bytes object. Default\n   encoding is ``\'utf-8\'``. *errors* may be given to set a different\n   error handling scheme. The default for *errors* is ``\'strict\'``,\n   meaning that encoding errors raise a ``UnicodeError``. Other\n   possible values are ``\'ignore\'``, ``\'replace\'``,\n   ``\'xmlcharrefreplace\'``, ``\'backslashreplace\'`` and any other name\n   registered via ``codecs.register_error()``, see section *Codec Base\n   Classes*. For a list of possible encodings, see section *Standard\n   Encodings*.\n\n   Changed in version 3.1: Support for keyword arguments added.\n\nstr.endswith(suffix[, start[, end]])\n\n   Return ``True`` if the string ends with the specified *suffix*,\n   otherwise return ``False``.  *suffix* can also be a tuple of\n   suffixes to look for.  With optional *start*, test beginning at\n   that position.  With optional *end*, stop comparing at that\n   position.\n\nstr.expandtabs([tabsize])\n\n   Return a copy of the string where all tab characters are replaced\n   by one or more spaces, depending on the current column and the\n   given tab size.  The column number is reset to zero after each\n   newline occurring in the string. If *tabsize* is not given, a tab\n   size of ``8`` characters is assumed.  This doesn\'t understand other\n   non-printing characters or escape sequences.\n\nstr.find(sub[, start[, end]])\n\n   Return the lowest index in the string where substring *sub* is\n   found, such that *sub* is contained in the slice ``s[start:end]``.\n   Optional arguments *start* and *end* are interpreted as in slice\n   notation.  Return ``-1`` if *sub* is not found.\n\n   Note: The ``find()`` method should be used only if you need to know the\n     position of *sub*.  To check if *sub* is a substring or not, use\n     the ``in`` operator:\n\n        >>> \'Py\' in \'Python\'\n        True\n\nstr.format(*args, **kwargs)\n\n   Perform a string formatting operation.  The string on which this\n   method is called can contain literal text or replacement fields\n   delimited by braces ``{}``.  Each replacement field contains either\n   the numeric index of a positional argument, or the name of a\n   keyword argument.  Returns a copy of the string where each\n   replacement field is replaced with the string value of the\n   corresponding argument.\n\n   >>> "The sum of 1 + 2 is {0}".format(1+2)\n   \'The sum of 1 + 2 is 3\'\n\n   See *Format String Syntax* for a description of the various\n   formatting options that can be specified in format strings.\n\nstr.format_map(mapping)\n\n   Similar to ``str.format(**mapping)``, except that ``mapping`` is\n   used directly and not copied to a ``dict`` .  This is useful if for\n   example ``mapping`` is a dict subclass:\n\n   >>> class Default(dict):\n   ...     def __missing__(self, key):\n   ...         return key\n   ...\n   >>> \'{name} was born in {country}\'.format_map(Default(name=\'Guido\'))\n   \'Guido was born in country\'\n\n   New in version 3.2.\n\nstr.index(sub[, start[, end]])\n\n   Like ``find()``, but raise ``ValueError`` when the substring is not\n   found.\n\nstr.isalnum()\n\n   Return true if all characters in the string are alphanumeric and\n   there is at least one character, false otherwise.  A character\n   ``c`` is alphanumeric if one of the following returns ``True``:\n   ``c.isalpha()``, ``c.isdecimal()``, ``c.isdigit()``, or\n   ``c.isnumeric()``.\n\nstr.isalpha()\n\n   Return true if all characters in the string are alphabetic and\n   there is at least one character, false otherwise.  Alphabetic\n   characters are those characters defined in the Unicode character\n   database as "Letter", i.e., those with general category property\n   being one of "Lm", "Lt", "Lu", "Ll", or "Lo".  Note that this is\n   different from the "Alphabetic" property defined in the Unicode\n   Standard.\n\nstr.isdecimal()\n\n   Return true if all characters in the string are decimal characters\n   and there is at least one character, false otherwise. Decimal\n   characters are those from general category "Nd". This category\n   includes digit characters, and all characters that that can be used\n   to form decimal-radix numbers, e.g. U+0660, ARABIC-INDIC DIGIT\n   ZERO.\n\nstr.isdigit()\n\n   Return true if all characters in the string are digits and there is\n   at least one character, false otherwise.  Digits include decimal\n   characters and digits that need special handling, such as the\n   compatibility superscript digits.  Formally, a digit is a character\n   that has the property value Numeric_Type=Digit or\n   Numeric_Type=Decimal.\n\nstr.isidentifier()\n\n   Return true if the string is a valid identifier according to the\n   language definition, section *Identifiers and keywords*.\n\nstr.islower()\n\n   Return true if all cased characters in the string are lowercase and\n   there is at least one cased character, false otherwise.  Cased\n   characters are those with general category property being one of\n   "Lu", "Ll", or "Lt" and lowercase characters are those with general\n   category property "Ll".\n\nstr.isnumeric()\n\n   Return true if all characters in the string are numeric characters,\n   and there is at least one character, false otherwise. Numeric\n   characters include digit characters, and all characters that have\n   the Unicode numeric value property, e.g. U+2155, VULGAR FRACTION\n   ONE FIFTH.  Formally, numeric characters are those with the\n   property value Numeric_Type=Digit, Numeric_Type=Decimal or\n   Numeric_Type=Numeric.\n\nstr.isprintable()\n\n   Return true if all characters in the string are printable or the\n   string is empty, false otherwise.  Nonprintable characters are\n   those characters defined in the Unicode character database as\n   "Other" or "Separator", excepting the ASCII space (0x20) which is\n   considered printable.  (Note that printable characters in this\n   context are those which should not be escaped when ``repr()`` is\n   invoked on a string.  It has no bearing on the handling of strings\n   written to ``sys.stdout`` or ``sys.stderr``.)\n\nstr.isspace()\n\n   Return true if there are only whitespace characters in the string\n   and there is at least one character, false otherwise.  Whitespace\n   characters  are those characters defined in the Unicode character\n   database as "Other" or "Separator" and those with bidirectional\n   property being one of "WS", "B", or "S".\n\nstr.istitle()\n\n   Return true if the string is a titlecased string and there is at\n   least one character, for example uppercase characters may only\n   follow uncased characters and lowercase characters only cased ones.\n   Return false otherwise.\n\nstr.isupper()\n\n   Return true if all cased characters in the string are uppercase and\n   there is at least one cased character, false otherwise. Cased\n   characters are those with general category property being one of\n   "Lu", "Ll", or "Lt" and uppercase characters are those with general\n   category property "Lu".\n\nstr.join(iterable)\n\n   Return a string which is the concatenation of the strings in the\n   *iterable* *iterable*.  A ``TypeError`` will be raised if there are\n   any non-string values in *seq*, including ``bytes`` objects.  The\n   separator between elements is the string providing this method.\n\nstr.ljust(width[, fillchar])\n\n   Return the string left justified in a string of length *width*.\n   Padding is done using the specified *fillchar* (default is a\n   space).  The original string is returned if *width* is less than\n   ``len(s)``.\n\nstr.lower()\n\n   Return a copy of the string converted to lowercase.\n\nstr.lstrip([chars])\n\n   Return a copy of the string with leading characters removed.  The\n   *chars* argument is a string specifying the set of characters to be\n   removed.  If omitted or ``None``, the *chars* argument defaults to\n   removing whitespace.  The *chars* argument is not a prefix; rather,\n   all combinations of its values are stripped:\n\n   >>> \'   spacious   \'.lstrip()\n   \'spacious   \'\n   >>> \'www.example.com\'.lstrip(\'cmowz.\')\n   \'example.com\'\n\nstatic str.maketrans(x[, y[, z]])\n\n   This static method returns a translation table usable for\n   ``str.translate()``.\n\n   If there is only one argument, it must be a dictionary mapping\n   Unicode ordinals (integers) or characters (strings of length 1) to\n   Unicode ordinals, strings (of arbitrary lengths) or None.\n   Character keys will then be converted to ordinals.\n\n   If there are two arguments, they must be strings of equal length,\n   and in the resulting dictionary, each character in x will be mapped\n   to the character at the same position in y.  If there is a third\n   argument, it must be a string, whose characters will be mapped to\n   None in the result.\n\nstr.partition(sep)\n\n   Split the string at the first occurrence of *sep*, and return a\n   3-tuple containing the part before the separator, the separator\n   itself, and the part after the separator.  If the separator is not\n   found, return a 3-tuple containing the string itself, followed by\n   two empty strings.\n\nstr.replace(old, new[, count])\n\n   Return a copy of the string with all occurrences of substring *old*\n   replaced by *new*.  If the optional argument *count* is given, only\n   the first *count* occurrences are replaced.\n\nstr.rfind(sub[, start[, end]])\n\n   Return the highest index in the string where substring *sub* is\n   found, such that *sub* is contained within ``s[start:end]``.\n   Optional arguments *start* and *end* are interpreted as in slice\n   notation.  Return ``-1`` on failure.\n\nstr.rindex(sub[, start[, end]])\n\n   Like ``rfind()`` but raises ``ValueError`` when the substring *sub*\n   is not found.\n\nstr.rjust(width[, fillchar])\n\n   Return the string right justified in a string of length *width*.\n   Padding is done using the specified *fillchar* (default is a\n   space). The original string is returned if *width* is less than\n   ``len(s)``.\n\nstr.rpartition(sep)\n\n   Split the string at the last occurrence of *sep*, and return a\n   3-tuple containing the part before the separator, the separator\n   itself, and the part after the separator.  If the separator is not\n   found, return a 3-tuple containing two empty strings, followed by\n   the string itself.\n\nstr.rsplit([sep[, maxsplit]])\n\n   Return a list of the words in the string, using *sep* as the\n   delimiter string. If *maxsplit* is given, at most *maxsplit* splits\n   are done, the *rightmost* ones.  If *sep* is not specified or\n   ``None``, any whitespace string is a separator.  Except for\n   splitting from the right, ``rsplit()`` behaves like ``split()``\n   which is described in detail below.\n\nstr.rstrip([chars])\n\n   Return a copy of the string with trailing characters removed.  The\n   *chars* argument is a string specifying the set of characters to be\n   removed.  If omitted or ``None``, the *chars* argument defaults to\n   removing whitespace.  The *chars* argument is not a suffix; rather,\n   all combinations of its values are stripped:\n\n   >>> \'   spacious   \'.rstrip()\n   \'   spacious\'\n   >>> \'mississippi\'.rstrip(\'ipz\')\n   \'mississ\'\n\nstr.split([sep[, maxsplit]])\n\n   Return a list of the words in the string, using *sep* as the\n   delimiter string.  If *maxsplit* is given, at most *maxsplit*\n   splits are done (thus, the list will have at most ``maxsplit+1``\n   elements).  If *maxsplit* is not specified, then there is no limit\n   on the number of splits (all possible splits are made).\n\n   If *sep* is given, consecutive delimiters are not grouped together\n   and are deemed to delimit empty strings (for example,\n   ``\'1,,2\'.split(\',\')`` returns ``[\'1\', \'\', \'2\']``).  The *sep*\n   argument may consist of multiple characters (for example,\n   ``\'1<>2<>3\'.split(\'<>\')`` returns ``[\'1\', \'2\', \'3\']``). Splitting\n   an empty string with a specified separator returns ``[\'\']``.\n\n   If *sep* is not specified or is ``None``, a different splitting\n   algorithm is applied: runs of consecutive whitespace are regarded\n   as a single separator, and the result will contain no empty strings\n   at the start or end if the string has leading or trailing\n   whitespace.  Consequently, splitting an empty string or a string\n   consisting of just whitespace with a ``None`` separator returns\n   ``[]``.\n\n   For example, ``\' 1  2   3  \'.split()`` returns ``[\'1\', \'2\', \'3\']``,\n   and ``\'  1  2   3  \'.split(None, 1)`` returns ``[\'1\', \'2   3  \']``.\n\nstr.splitlines([keepends])\n\n   Return a list of the lines in the string, breaking at line\n   boundaries.  Line breaks are not included in the resulting list\n   unless *keepends* is given and true.\n\nstr.startswith(prefix[, start[, end]])\n\n   Return ``True`` if string starts with the *prefix*, otherwise\n   return ``False``. *prefix* can also be a tuple of prefixes to look\n   for.  With optional *start*, test string beginning at that\n   position.  With optional *end*, stop comparing string at that\n   position.\n\nstr.strip([chars])\n\n   Return a copy of the string with the leading and trailing\n   characters removed. The *chars* argument is a string specifying the\n   set of characters to be removed. If omitted or ``None``, the\n   *chars* argument defaults to removing whitespace. The *chars*\n   argument is not a prefix or suffix; rather, all combinations of its\n   values are stripped:\n\n   >>> \'   spacious   \'.strip()\n   \'spacious\'\n   >>> \'www.example.com\'.strip(\'cmowz.\')\n   \'example\'\n\nstr.swapcase()\n\n   Return a copy of the string with uppercase characters converted to\n   lowercase and vice versa.\n\nstr.title()\n\n   Return a titlecased version of the string where words start with an\n   uppercase character and the remaining characters are lowercase.\n\n   The algorithm uses a simple language-independent definition of a\n   word as groups of consecutive letters.  The definition works in\n   many contexts but it means that apostrophes in contractions and\n   possessives form word boundaries, which may not be the desired\n   result:\n\n      >>> "they\'re bill\'s friends from the UK".title()\n      "They\'Re Bill\'S Friends From The Uk"\n\n   A workaround for apostrophes can be constructed using regular\n   expressions:\n\n      >>> import re\n      >>> def titlecase(s):\n              return re.sub(r"[A-Za-z]+(\'[A-Za-z]+)?",\n                            lambda mo: mo.group(0)[0].upper() +\n                                       mo.group(0)[1:].lower(),\n                            s)\n\n      >>> titlecase("they\'re bill\'s friends.")\n      "They\'re Bill\'s Friends."\n\nstr.translate(map)\n\n   Return a copy of the *s* where all characters have been mapped\n   through the *map* which must be a dictionary of Unicode ordinals\n   (integers) to Unicode ordinals, strings or ``None``.  Unmapped\n   characters are left untouched. Characters mapped to ``None`` are\n   deleted.\n\n   You can use ``str.maketrans()`` to create a translation map from\n   character-to-character mappings in different formats.\n\n   Note: An even more flexible approach is to create a custom character\n     mapping codec using the ``codecs`` module (see\n     ``encodings.cp1251`` for an example).\n\nstr.upper()\n\n   Return a copy of the string converted to uppercase.\n\nstr.zfill(width)\n\n   Return the numeric string left filled with zeros in a string of\n   length *width*.  A sign prefix is handled correctly.  The original\n   string is returned if *width* is less than ``len(s)``.\n\n\nOld String Formatting Operations\n================================\n\nNote: The formatting operations described here are obsolete and may go\n  away in future versions of Python.  Use the new *String Formatting*\n  in new code.\n\nString objects have one unique built-in operation: the ``%`` operator\n(modulo). This is also known as the string *formatting* or\n*interpolation* operator. Given ``format % values`` (where *format* is\na string), ``%`` conversion specifications in *format* are replaced\nwith zero or more elements of *values*. The effect is similar to the\nusing ``sprintf()`` in the C language.\n\nIf *format* requires a single argument, *values* may be a single non-\ntuple object. [4]  Otherwise, *values* must be a tuple with exactly\nthe number of items specified by the format string, or a single\nmapping object (for example, a dictionary).\n\nA conversion specifier contains two or more characters and has the\nfollowing components, which must occur in this order:\n\n1. The ``\'%\'`` character, which marks the start of the specifier.\n\n2. Mapping key (optional), consisting of a parenthesised sequence of\n   characters (for example, ``(somename)``).\n\n3. Conversion flags (optional), which affect the result of some\n   conversion types.\n\n4. Minimum field width (optional).  If specified as an ``\'*\'``\n   (asterisk), the actual width is read from the next element of the\n   tuple in *values*, and the object to convert comes after the\n   minimum field width and optional precision.\n\n5. Precision (optional), given as a ``\'.\'`` (dot) followed by the\n   precision.  If specified as ``\'*\'`` (an asterisk), the actual\n   precision is read from the next element of the tuple in *values*,\n   and the value to convert comes after the precision.\n\n6. Length modifier (optional).\n\n7. Conversion type.\n\nWhen the right argument is a dictionary (or other mapping type), then\nthe formats in the string *must* include a parenthesised mapping key\ninto that dictionary inserted immediately after the ``\'%\'`` character.\nThe mapping key selects the value to be formatted from the mapping.\nFor example:\n\n>>> print(\'%(language)s has %(number)03d quote types.\' %\n...       {\'language\': "Python", "number": 2})\nPython has 002 quote types.\n\nIn this case no ``*`` specifiers may occur in a format (since they\nrequire a sequential parameter list).\n\nThe conversion flag characters are:\n\n+-----------+-----------------------------------------------------------------------+\n| Flag      | Meaning                                                               |\n+===========+=======================================================================+\n| ``\'#\'``   | The value conversion will use the "alternate form" (where defined     |\n|           | below).                                                               |\n+-----------+-----------------------------------------------------------------------+\n| ``\'0\'``   | The conversion will be zero padded for numeric values.                |\n+-----------+-----------------------------------------------------------------------+\n| ``\'-\'``   | The converted value is left adjusted (overrides the ``\'0\'``           |\n|           | conversion if both are given).                                        |\n+-----------+-----------------------------------------------------------------------+\n| ``\' \'``   | (a space) A blank should be left before a positive number (or empty   |\n|           | string) produced by a signed conversion.                              |\n+-----------+-----------------------------------------------------------------------+\n| ``\'+\'``   | A sign character (``\'+\'`` or ``\'-\'``) will precede the conversion     |\n|           | (overrides a "space" flag).                                           |\n+-----------+-----------------------------------------------------------------------+\n\nA length modifier (``h``, ``l``, or ``L``) may be present, but is\nignored as it is not necessary for Python -- so e.g. ``%ld`` is\nidentical to ``%d``.\n\nThe conversion types are:\n\n+--------------+-------------------------------------------------------+---------+\n| Conversion   | Meaning                                               | Notes   |\n+==============+=======================================================+=========+\n| ``\'d\'``      | Signed integer decimal.                               |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'i\'``      | Signed integer decimal.                               |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'o\'``      | Signed octal value.                                   | (1)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'u\'``      | Obsolete type -- it is identical to ``\'d\'``.          | (7)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'x\'``      | Signed hexadecimal (lowercase).                       | (2)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'X\'``      | Signed hexadecimal (uppercase).                       | (2)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'e\'``      | Floating point exponential format (lowercase).        | (3)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'E\'``      | Floating point exponential format (uppercase).        | (3)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'f\'``      | Floating point decimal format.                        | (3)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'F\'``      | Floating point decimal format.                        | (3)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'g\'``      | Floating point format. Uses lowercase exponential     | (4)     |\n|              | format if exponent is less than -4 or not less than   |         |\n|              | precision, decimal format otherwise.                  |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'G\'``      | Floating point format. Uses uppercase exponential     | (4)     |\n|              | format if exponent is less than -4 or not less than   |         |\n|              | precision, decimal format otherwise.                  |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'c\'``      | Single character (accepts integer or single character |         |\n|              | string).                                              |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'r\'``      | String (converts any Python object using ``repr()``). | (5)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'s\'``      | String (converts any Python object using ``str()``).  | (5)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'a\'``      | String (converts any Python object using              | (5)     |\n|              | ``ascii()``).                                         |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'%\'``      | No argument is converted, results in a ``\'%\'``        |         |\n|              | character in the result.                              |         |\n+--------------+-------------------------------------------------------+---------+\n\nNotes:\n\n1. The alternate form causes a leading zero (``\'0\'``) to be inserted\n   between left-hand padding and the formatting of the number if the\n   leading character of the result is not already a zero.\n\n2. The alternate form causes a leading ``\'0x\'`` or ``\'0X\'`` (depending\n   on whether the ``\'x\'`` or ``\'X\'`` format was used) to be inserted\n   between left-hand padding and the formatting of the number if the\n   leading character of the result is not already a zero.\n\n3. The alternate form causes the result to always contain a decimal\n   point, even if no digits follow it.\n\n   The precision determines the number of digits after the decimal\n   point and defaults to 6.\n\n4. The alternate form causes the result to always contain a decimal\n   point, and trailing zeroes are not removed as they would otherwise\n   be.\n\n   The precision determines the number of significant digits before\n   and after the decimal point and defaults to 6.\n\n5. If precision is ``N``, the output is truncated to ``N`` characters.\n\n1. See **PEP 237**.\n\nSince Python strings have an explicit length, ``%s`` conversions do\nnot assume that ``\'\\0\'`` is the end of the string.\n\nChanged in version 3.1: ``%f`` conversions for numbers whose absolute\nvalue is over 1e50 are no longer replaced by ``%g`` conversions.\n\nAdditional string operations are defined in standard modules\n``string`` and ``re``.\n\n\nRange Type\n==========\n\nThe ``range`` type is an immutable sequence which is commonly used for\nlooping.  The advantage of the ``range`` type is that an ``range``\nobject will always take the same amount of memory, no matter the size\nof the range it represents.\n\nRange objects have relatively little behavior: they support indexing,\ncontains, iteration, the ``len()`` function, and the following\nmethods:\n\nrange.count(x)\n\n   Return the number of *i*\'s for which ``s[i] == x``.\n\n      New in version 3.2.\n\nrange.index(x)\n\n   Return the smallest *i* such that ``s[i] == x``.  Raises\n   ``ValueError`` when *x* is not in the range.\n\n      New in version 3.2.\n\n\nMutable Sequence Types\n======================\n\nList and bytearray objects support additional operations that allow\nin-place modification of the object.  Other mutable sequence types\n(when added to the language) should also support these operations.\nStrings and tuples are immutable sequence types: such objects cannot\nbe modified once created. The following operations are defined on\nmutable sequence types (where *x* is an arbitrary object).\n\nNote that while lists allow their items to be of any type, bytearray\nobject "items" are all integers in the range 0 <= x < 256.\n\n+--------------------------------+----------------------------------+-----------------------+\n| Operation                      | Result                           | Notes                 |\n+================================+==================================+=======================+\n| ``s[i] = x``                   | item *i* of *s* is replaced by   |                       |\n|                                | *x*                              |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j] = t``                 | slice of *s* from *i* to *j* is  |                       |\n|                                | replaced by the contents of the  |                       |\n|                                | iterable *t*                     |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j]``                 | same as ``s[i:j] = []``          |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j:k] = t``               | the elements of ``s[i:j:k]`` are | (1)                   |\n|                                | replaced by those of *t*         |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j:k]``               | removes the elements of          |                       |\n|                                | ``s[i:j:k]`` from the list       |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.append(x)``                | same as ``s[len(s):len(s)] =     |                       |\n|                                | [x]``                            |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.extend(x)``                | same as ``s[len(s):len(s)] = x`` | (2)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.count(x)``                 | return number of *i*\'s for which |                       |\n|                                | ``s[i] == x``                    |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.index(x[, i[, j]])``       | return smallest *k* such that    | (3)                   |\n|                                | ``s[k] == x`` and ``i <= k < j`` |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.insert(i, x)``             | same as ``s[i:i] = [x]``         | (4)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.pop([i])``                 | same as ``x = s[i]; del s[i];    | (5)                   |\n|                                | return x``                       |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.remove(x)``                | same as ``del s[s.index(x)]``    | (3)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.reverse()``                | reverses the items of *s* in     | (6)                   |\n|                                | place                            |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.sort([key[, reverse]])``   | sort the items of *s* in place   | (6), (7), (8)         |\n+--------------------------------+----------------------------------+-----------------------+\n\nNotes:\n\n1. *t* must have the same length as the slice it is replacing.\n\n2. *x* can be any iterable object.\n\n3. Raises ``ValueError`` when *x* is not found in *s*. When a negative\n   index is passed as the second or third parameter to the ``index()``\n   method, the sequence length is added, as for slice indices.  If it\n   is still negative, it is truncated to zero, as for slice indices.\n\n4. When a negative index is passed as the first parameter to the\n   ``insert()`` method, the sequence length is added, as for slice\n   indices.  If it is still negative, it is truncated to zero, as for\n   slice indices.\n\n5. The optional argument *i* defaults to ``-1``, so that by default\n   the last item is removed and returned.\n\n6. The ``sort()`` and ``reverse()`` methods modify the sequence in\n   place for economy of space when sorting or reversing a large\n   sequence.  To remind you that they operate by side effect, they\n   don\'t return the sorted or reversed sequence.\n\n7. The ``sort()`` method takes optional arguments for controlling the\n   comparisons.  Each must be specified as a keyword argument.\n\n   *key* specifies a function of one argument that is used to extract\n   a comparison key from each list element: ``key=str.lower``.  The\n   default value is ``None``. Use ``functools.cmp_to_key()`` to\n   convert an old-style *cmp* function to a *key* function.\n\n   *reverse* is a boolean value.  If set to ``True``, then the list\n   elements are sorted as if each comparison were reversed.\n\n   The ``sort()`` method is guaranteed to be stable.  A sort is stable\n   if it guarantees not to change the relative order of elements that\n   compare equal --- this is helpful for sorting in multiple passes\n   (for example, sort by department, then by salary grade).\n\n   **CPython implementation detail:** While a list is being sorted,\n   the effect of attempting to mutate, or even inspect, the list is\n   undefined.  The C implementation of Python makes the list appear\n   empty for the duration, and raises ``ValueError`` if it can detect\n   that the list has been mutated during a sort.\n\n8. ``sort()`` is not supported by ``bytearray`` objects.\n\n\nBytes and Byte Array Methods\n============================\n\nBytes and bytearray objects, being "strings of bytes", have all\nmethods found on strings, with the exception of ``encode()``,\n``format()`` and ``isidentifier()``, which do not make sense with\nthese types.  For converting the objects to strings, they have a\n``decode()`` method.\n\nWherever one of these methods needs to interpret the bytes as\ncharacters (e.g. the ``is...()`` methods), the ASCII character set is\nassumed.\n\nNote: The methods on bytes and bytearray objects don\'t accept strings as\n  their arguments, just as the methods on strings don\'t accept bytes\n  as their arguments.  For example, you have to write\n\n     a = "abc"\n     b = a.replace("a", "f")\n\n  and\n\n     a = b"abc"\n     b = a.replace(b"a", b"f")\n\nbytes.decode(encoding="utf-8", errors="strict")\nbytearray.decode(encoding="utf-8", errors="strict")\n\n   Return a string decoded from the given bytes.  Default encoding is\n   ``\'utf-8\'``. *errors* may be given to set a different error\n   handling scheme.  The default for *errors* is ``\'strict\'``, meaning\n   that encoding errors raise a ``UnicodeError``.  Other possible\n   values are ``\'ignore\'``, ``\'replace\'`` and any other name\n   registered via ``codecs.register_error()``, see section *Codec Base\n   Classes*. For a list of possible encodings, see section *Standard\n   Encodings*.\n\n   Changed in version 3.1: Added support for keyword arguments.\n\nThe bytes and bytearray types have an additional class method:\n\nclassmethod bytes.fromhex(string)\nclassmethod bytearray.fromhex(string)\n\n   This ``bytes`` class method returns a bytes or bytearray object,\n   decoding the given string object.  The string must contain two\n   hexadecimal digits per byte, spaces are ignored.\n\n   >>> bytes.fromhex(\'f0 f1f2  \')\n   b\'\\xf0\\xf1\\xf2\'\n\nThe maketrans and translate methods differ in semantics from the\nversions available on strings:\n\nbytes.translate(table[, delete])\nbytearray.translate(table[, delete])\n\n   Return a copy of the bytes or bytearray object where all bytes\n   occurring in the optional argument *delete* are removed, and the\n   remaining bytes have been mapped through the given translation\n   table, which must be a bytes object of length 256.\n\n   You can use the ``bytes.maketrans()`` method to create a\n   translation table.\n\n   Set the *table* argument to ``None`` for translations that only\n   delete characters:\n\n      >>> b\'read this short text\'.translate(None, b\'aeiou\')\n      b\'rd ths shrt txt\'\n\nstatic bytes.maketrans(from, to)\nstatic bytearray.maketrans(from, to)\n\n   This static method returns a translation table usable for\n   ``bytes.translate()`` that will map each character in *from* into\n   the character at the same position in *to*; *from* and *to* must be\n   bytes objects and have the same length.\n\n   New in version 3.1.\n',
  'typesseq-mutable': '\nMutable Sequence Types\n**********************\n\nList and bytearray objects support additional operations that allow\nin-place modification of the object.  Other mutable sequence types\n(when added to the language) should also support these operations.\nStrings and tuples are immutable sequence types: such objects cannot\nbe modified once created. The following operations are defined on\nmutable sequence types (where *x* is an arbitrary object).\n\nNote that while lists allow their items to be of any type, bytearray\nobject "items" are all integers in the range 0 <= x < 256.\n\n+--------------------------------+----------------------------------+-----------------------+\n| Operation                      | Result                           | Notes                 |\n+================================+==================================+=======================+\n| ``s[i] = x``                   | item *i* of *s* is replaced by   |                       |\n|                                | *x*                              |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j] = t``                 | slice of *s* from *i* to *j* is  |                       |\n|                                | replaced by the contents of the  |                       |\n|                                | iterable *t*                     |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j]``                 | same as ``s[i:j] = []``          |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j:k] = t``               | the elements of ``s[i:j:k]`` are | (1)                   |\n|                                | replaced by those of *t*         |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j:k]``               | removes the elements of          |                       |\n|                                | ``s[i:j:k]`` from the list       |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.append(x)``                | same as ``s[len(s):len(s)] =     |                       |\n|                                | [x]``                            |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.extend(x)``                | same as ``s[len(s):len(s)] = x`` | (2)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.count(x)``                 | return number of *i*\'s for which |                       |\n|                                | ``s[i] == x``                    |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.index(x[, i[, j]])``       | return smallest *k* such that    | (3)                   |\n|                                | ``s[k] == x`` and ``i <= k < j`` |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.insert(i, x)``             | same as ``s[i:i] = [x]``         | (4)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.pop([i])``                 | same as ``x = s[i]; del s[i];    | (5)                   |\n|                                | return x``                       |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.remove(x)``                | same as ``del s[s.index(x)]``    | (3)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.reverse()``                | reverses the items of *s* in     | (6)                   |\n|                                | place                            |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.sort([key[, reverse]])``   | sort the items of *s* in place   | (6), (7), (8)         |\n+--------------------------------+----------------------------------+-----------------------+\n\nNotes:\n\n1. *t* must have the same length as the slice it is replacing.\n\n2. *x* can be any iterable object.\n\n3. Raises ``ValueError`` when *x* is not found in *s*. When a negative\n   index is passed as the second or third parameter to the ``index()``\n   method, the sequence length is added, as for slice indices.  If it\n   is still negative, it is truncated to zero, as for slice indices.\n\n4. When a negative index is passed as the first parameter to the\n   ``insert()`` method, the sequence length is added, as for slice\n   indices.  If it is still negative, it is truncated to zero, as for\n   slice indices.\n\n5. The optional argument *i* defaults to ``-1``, so that by default\n   the last item is removed and returned.\n\n6. The ``sort()`` and ``reverse()`` methods modify the sequence in\n   place for economy of space when sorting or reversing a large\n   sequence.  To remind you that they operate by side effect, they\n   don\'t return the sorted or reversed sequence.\n\n7. The ``sort()`` method takes optional arguments for controlling the\n   comparisons.  Each must be specified as a keyword argument.\n\n   *key* specifies a function of one argument that is used to extract\n   a comparison key from each list element: ``key=str.lower``.  The\n   default value is ``None``. Use ``functools.cmp_to_key()`` to\n   convert an old-style *cmp* function to a *key* function.\n\n   *reverse* is a boolean value.  If set to ``True``, then the list\n   elements are sorted as if each comparison were reversed.\n\n   The ``sort()`` method is guaranteed to be stable.  A sort is stable\n   if it guarantees not to change the relative order of elements that\n   compare equal --- this is helpful for sorting in multiple passes\n   (for example, sort by department, then by salary grade).\n\n   **CPython implementation detail:** While a list is being sorted,\n   the effect of attempting to mutate, or even inspect, the list is\n   undefined.  The C implementation of Python makes the list appear\n   empty for the duration, and raises ``ValueError`` if it can detect\n   that the list has been mutated during a sort.\n\n8. ``sort()`` is not supported by ``bytearray`` objects.\n',
  'unary': '\nUnary arithmetic and bitwise operations\n***************************************\n\nAll unary arithmetic and bitwise operations have the same priority:\n\n   u_expr ::= power | "-" u_expr | "+" u_expr | "~" u_expr\n\nThe unary ``-`` (minus) operator yields the negation of its numeric\nargument.\n\nThe unary ``+`` (plus) operator yields its numeric argument unchanged.\n\nThe unary ``~`` (invert) operator yields the bitwise inversion of its\ninteger argument.  The bitwise inversion of ``x`` is defined as\n``-(x+1)``.  It only applies to integral numbers.\n\nIn all three cases, if the argument does not have the proper type, a\n``TypeError`` exception is raised.\n',
  'while': '\nThe ``while`` statement\n***********************\n\nThe ``while`` statement is used for repeated execution as long as an\nexpression is true:\n\n   while_stmt ::= "while" expression ":" suite\n                  ["else" ":" suite]\n\nThis repeatedly tests the expression and, if it is true, executes the\nfirst suite; if the expression is false (which may be the first time\nit is tested) the suite of the ``else`` clause, if present, is\nexecuted and the loop terminates.\n\nA ``break`` statement executed in the first suite terminates the loop\nwithout executing the ``else`` clause\'s suite.  A ``continue``\nstatement executed in the first suite skips the rest of the suite and\ngoes back to testing the expression.\n',

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:41:29 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:41:29 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Bump_version_to?=
	=?utf8?q?_3=2E2=2E2rc1=2E?=
Message-ID: <E1R06OX-0002XG-UL@dinsdale.python.org>

http://hg.python.org/cpython/rev/33392e7372c8
changeset:   72221:33392e7372c8
branch:      3.2
user:        Georg Brandl <georg at python.org>
date:        Sat Aug 13 11:34:58 2011 +0200
summary:
  Bump version to 3.2.2rc1.

files:
  Include/patchlevel.h      |  8 ++++----
  Lib/distutils/__init__.py |  2 +-
  Lib/idlelib/idlever.py    |  2 +-
  Misc/NEWS                 |  8 ++++----
  Misc/RPM/python-3.2.spec  |  2 +-
  README                    |  4 ++--
  6 files changed, 13 insertions(+), 13 deletions(-)


diff --git a/Include/patchlevel.h b/Include/patchlevel.h
--- a/Include/patchlevel.h
+++ b/Include/patchlevel.h
@@ -18,12 +18,12 @@
 /*--start constants--*/
 #define PY_MAJOR_VERSION	3
 #define PY_MINOR_VERSION	2
-#define PY_MICRO_VERSION	1
-#define PY_RELEASE_LEVEL	PY_RELEASE_LEVEL_FINAL
-#define PY_RELEASE_SERIAL	0
+#define PY_MICRO_VERSION	2
+#define PY_RELEASE_LEVEL	PY_RELEASE_LEVEL_GAMMA
+#define PY_RELEASE_SERIAL	1
 
 /* Version as a string */
-#define PY_VERSION      	"3.2.1+"
+#define PY_VERSION      	"3.2.2rc1"
 /*--end constants--*/
 
 /* Subversion Revision number of this file (not of the repository). Empty
diff --git a/Lib/distutils/__init__.py b/Lib/distutils/__init__.py
--- a/Lib/distutils/__init__.py
+++ b/Lib/distutils/__init__.py
@@ -13,5 +13,5 @@
 # Updated automatically by the Python release process.
 #
 #--start constants--
-__version__ = "3.2.1"
+__version__ = "3.2.2rc1"
 #--end constants--
diff --git a/Lib/idlelib/idlever.py b/Lib/idlelib/idlever.py
--- a/Lib/idlelib/idlever.py
+++ b/Lib/idlelib/idlever.py
@@ -1,1 +1,1 @@
-IDLE_VERSION = "3.2.1"
+IDLE_VERSION = "3.2.2rc1"
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -2,10 +2,10 @@
 Python News
 +++++++++++
 
-What's New in Python 3.2.2?
-===========================
-
-*Release date: XX-XXX-2011*
+What's New in Python 3.2.2 release candidate 1?
+===============================================
+
+*Release date: 14-Aug-2011*
 
 Core and Builtins
 -----------------
diff --git a/Misc/RPM/python-3.2.spec b/Misc/RPM/python-3.2.spec
--- a/Misc/RPM/python-3.2.spec
+++ b/Misc/RPM/python-3.2.spec
@@ -39,7 +39,7 @@
 
 %define name python
 #--start constants--
-%define version 3.2.1
+%define version 3.2.2rc1
 %define libvers 3.2
 #--end constants--
 %define release 1pydotorg
diff --git a/README b/README
--- a/README
+++ b/README
@@ -1,5 +1,5 @@
-This is Python version 3.2.1
-============================
+This is Python version 3.2.2 release candidate 1
+================================================
 
 Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011
 Python Software Foundation.  All rights reserved.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:41:51 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:41:51 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMik6?=
	=?utf8?q?_Merge_with_cpython=2E?=
Message-ID: <E1R06Ot-0002XM-1Y@dinsdale.python.org>

http://hg.python.org/cpython/rev/a600d84d815b
changeset:   72222:a600d84d815b
branch:      3.2
parent:      72221:33392e7372c8
parent:      71855:843cd43206b4
user:        Georg Brandl <georg at python.org>
date:        Sat Aug 13 11:54:33 2011 +0200
summary:
  Merge with cpython.

files:
  Lib/tarfile.py           |  6 ++++--
  Lib/test/test_tarfile.py |  8 ++++++++
  Misc/NEWS                |  3 +++
  3 files changed, 15 insertions(+), 2 deletions(-)


diff --git a/Lib/tarfile.py b/Lib/tarfile.py
--- a/Lib/tarfile.py
+++ b/Lib/tarfile.py
@@ -1804,11 +1804,13 @@
             fileobj = gzip.GzipFile(name, mode + "b", compresslevel, fileobj)
             t = cls.taropen(name, mode, fileobj, **kwargs)
         except IOError:
-            if not extfileobj:
+            if not extfileobj and fileobj is not None:
                 fileobj.close()
+            if fileobj is None:
+                raise
             raise ReadError("not a gzip file")
         except:
-            if not extfileobj:
+            if not extfileobj and fileobj is not None:
                 fileobj.close()
             raise
         t._extfileobj = extfileobj
diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py
--- a/Lib/test/test_tarfile.py
+++ b/Lib/test/test_tarfile.py
@@ -1682,6 +1682,14 @@
 class GzipMiscReadTest(MiscReadTest):
     tarname = gzipname
     mode = "r:gz"
+
+    def test_non_existent_targz_file(self):
+        # Test for issue11513: prevent non-existent gzipped tarfiles raising
+        # multiple exceptions.
+        with self.assertRaisesRegex(IOError, "xxx") as ex:
+            tarfile.open("xxx", self.mode)
+        self.assertEqual(ex.exception.errno, errno.ENOENT)
+
 class GzipUstarReadTest(UstarReadTest):
     tarname = gzipname
     mode = "r:gz"
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -44,6 +44,9 @@
 Library
 -------
 
+- Issue #11513: Fix exception handling ``tarfile.TarFile.gzopen()`` when
+  the file cannot be opened.
+
 - Issue #12687: Fix a possible buffering bug when unpickling text mode
   (protocol 0, mostly) pickles.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:41:55 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:41:55 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Update_versions?=
	=?utf8?q?_in_LICENSE_files=2E?=
Message-ID: <E1R06Ox-0002XQ-4r@dinsdale.python.org>

http://hg.python.org/cpython/rev/c860feaa348d
changeset:   72223:c860feaa348d
branch:      3.2
tag:         v3.2.2rc1
user:        Georg Brandl <georg at python.org>
date:        Sat Aug 13 11:59:12 2011 +0200
summary:
  Update versions in LICENSE files.

files:
  Doc/license.rst |  10 +++++++++-
  LICENSE         |   6 +++++-
  2 files changed, 14 insertions(+), 2 deletions(-)


diff --git a/Doc/license.rst b/Doc/license.rst
--- a/Doc/license.rst
+++ b/Doc/license.rst
@@ -106,10 +106,18 @@
 +----------------+--------------+------------+------------+-----------------+
 | 3.1.1          | 3.1          | 2009       | PSF        | yes             |
 +----------------+--------------+------------+------------+-----------------+
-| 3.1.2          | 3.1          | 2010       | PSF        | yes             |
+| 3.1.2          | 3.1.1        | 2010       | PSF        | yes             |
++----------------+--------------+------------+------------+-----------------+
+| 3.1.3          | 3.1.2        | 2010       | PSF        | yes             |
++----------------+--------------+------------+------------+-----------------+
+| 3.1.4          | 3.1.3        | 2011       | PSF        | yes             |
 +----------------+--------------+------------+------------+-----------------+
 | 3.2            | 3.1          | 2011       | PSF        | yes             |
 +----------------+--------------+------------+------------+-----------------+
+| 3.2.1          | 3.2          | 2011       | PSF        | yes             |
++----------------+--------------+------------+------------+-----------------+
+| 3.2.2          | 3.2.1        | 2011       | PSF        | yes             |
++----------------+--------------+------------+------------+-----------------+
 
 .. note::
 
diff --git a/LICENSE b/LICENSE
--- a/LICENSE
+++ b/LICENSE
@@ -67,8 +67,12 @@
     3.0.1           3.0         2009        PSF         yes
     3.1             3.0.1       2009        PSF         yes
     3.1.1           3.1         2009        PSF         yes
-    3.1.2           3.1         2010        PSF         yes
+    3.1.2           3.1.1       2010        PSF         yes
+    3.1.3           3.1.2       2010        PSF         yes
+    3.1.4           3.1.3       2011        PSF         yes
     3.2             3.1         2011        PSF         yes
+    3.2.1           3.2         2011        PSF         yes
+    3.2.2           3.2.1       2011        PSF         yes
 
 Footnotes:
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:41:58 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:41:58 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Added_tag_v3=2E?=
 =?utf8?q?2=2E2rc1_for_changeset_c860feaa348d?=
Message-ID: <E1R06P0-0002XW-Cw@dinsdale.python.org>

http://hg.python.org/cpython/rev/7558edc1fcb9
changeset:   72224:7558edc1fcb9
branch:      3.2
user:        Georg Brandl <georg at python.org>
date:        Sat Aug 13 19:06:56 2011 +0200
summary:
  Added tag v3.2.2rc1 for changeset c860feaa348d

files:
  .hgtags |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/.hgtags b/.hgtags
--- a/.hgtags
+++ b/.hgtags
@@ -91,3 +91,4 @@
 cfa9364997c7f2e67b9cbb45c3a5fa3bba4e4999 v3.2.1rc1
 5df549718fb4841ff521fe051f6b54f290fad5d8 v3.2.1rc2
 ac1f7e5c05104d557d5acd922e95625ba5d1fe10 v3.2.1
+c860feaa348d663e598986894ee4680480577e15 v3.2.2rc1

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:00 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:00 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Post-release_st?=
	=?utf8?q?eps=2E?=
Message-ID: <E1R06P2-0002Xa-Ix@dinsdale.python.org>

http://hg.python.org/cpython/rev/3fc4bdf445a6
changeset:   72225:3fc4bdf445a6
branch:      3.2
user:        Georg Brandl <georg at python.org>
date:        Sat Aug 13 20:19:09 2011 +0200
summary:
  Post-release steps.

files:
  Include/patchlevel.h |   2 +-
  Misc/NEWS            |  12 ++++++++++++
  2 files changed, 13 insertions(+), 1 deletions(-)


diff --git a/Include/patchlevel.h b/Include/patchlevel.h
--- a/Include/patchlevel.h
+++ b/Include/patchlevel.h
@@ -23,7 +23,7 @@
 #define PY_RELEASE_SERIAL	1
 
 /* Version as a string */
-#define PY_VERSION      	"3.2.2rc1"
+#define PY_VERSION      	"3.2.2rc1+"
 /*--end constants--*/
 
 /* Subversion Revision number of this file (not of the repository). Empty
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -2,6 +2,18 @@
 Python News
 +++++++++++
 
+What's New in Python 3.2.2?
+===========================
+
+*Release date: XXXX-XX-XX*
+
+Core and Builtins
+-----------------
+
+Library
+-------
+
+
 What's New in Python 3.2.2 release candidate 1?
 ===============================================
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:01 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:01 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzEyNzI1OiBmaXgg?=
 =?utf8?q?working=2E_Patch_by_Ben_Hayden=2E?=
Message-ID: <E1R06P3-0002Xf-UK@dinsdale.python.org>

http://hg.python.org/cpython/rev/8730fdc9349f
changeset:   72226:8730fdc9349f
branch:      3.2
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Sun Aug 14 08:28:57 2011 +0300
summary:
  #12725: fix working. Patch by Ben Hayden.

files:
  Doc/ACKS.txt           |  1 +
  Doc/library/socket.rst |  6 +++---
  Modules/socketmodule.c |  6 +++---
  3 files changed, 7 insertions(+), 6 deletions(-)


diff --git a/Doc/ACKS.txt b/Doc/ACKS.txt
--- a/Doc/ACKS.txt
+++ b/Doc/ACKS.txt
@@ -79,6 +79,7 @@
    * Travis B. Hartwell
    * Tim Hatch
    * Janko Hauser
+   * Ben Hayden
    * Thomas Heller
    * Bernhard Herzog
    * Magnus L. Hetland
diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst
--- a/Doc/library/socket.rst
+++ b/Doc/library/socket.rst
@@ -513,14 +513,14 @@
 
 .. function:: getdefaulttimeout()
 
-   Return the default timeout in floating seconds for new socket objects. A value
+   Return the default timeout in seconds (float) for new socket objects. A value
    of ``None`` indicates that new socket objects have no timeout. When the socket
    module is first imported, the default is ``None``.
 
 
 .. function:: setdefaulttimeout(timeout)
 
-   Set the default timeout in floating seconds for new socket objects.  When
+   Set the default timeout in seconds (float) for new socket objects.  When
    the socket module is first imported, the default is ``None``.  See
    :meth:`~socket.settimeout` for possible values and their respective
    meanings.
@@ -632,7 +632,7 @@
 
 .. method:: socket.gettimeout()
 
-   Return the timeout in floating seconds associated with socket operations,
+   Return the timeout in seconds (float) associated with socket operations,
    or ``None`` if no timeout is set.  This reflects the last call to
    :meth:`setblocking` or :meth:`settimeout`.
 
diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c
--- a/Modules/socketmodule.c
+++ b/Modules/socketmodule.c
@@ -1808,7 +1808,7 @@
 PyDoc_STRVAR(gettimeout_doc,
 "gettimeout() -> timeout\n\
 \n\
-Returns the timeout in floating seconds associated with socket \n\
+Returns the timeout in seconds (float) associated with socket \n\
 operations. A timeout of None indicates that timeouts on socket \n\
 operations are disabled.");
 
@@ -4201,7 +4201,7 @@
 PyDoc_STRVAR(getdefaulttimeout_doc,
 "getdefaulttimeout() -> timeout\n\
 \n\
-Returns the default timeout in floating seconds for new socket objects.\n\
+Returns the default timeout in seconds (float) for new socket objects.\n\
 A value of None indicates that new socket objects have no timeout.\n\
 When the socket module is first imported, the default is None.");
 
@@ -4231,7 +4231,7 @@
 PyDoc_STRVAR(setdefaulttimeout_doc,
 "setdefaulttimeout(timeout)\n\
 \n\
-Set the default timeout in floating seconds for new socket objects.\n\
+Set the default timeout in seconds (float) for new socket objects.\n\
 A value of None indicates that new socket objects have no timeout.\n\
 When the socket module is first imported, the default is None.");
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:03 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:03 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Revert_change_t?=
 =?utf8?q?hat_was_not_a_syntax_fix_but_actually_a_behavior_change?=
Message-ID: <E1R06P5-0002Xm-0L@dinsdale.python.org>

http://hg.python.org/cpython/rev/b457c840b54f
changeset:   72227:b457c840b54f
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Aug 16 19:05:56 2011 +0200
summary:
  Revert change that was not a syntax fix but actually a behavior change

files:
  Makefile.pre.in |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Makefile.pre.in b/Makefile.pre.in
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -1313,7 +1313,7 @@
 		-o -name .hgignore \
 		-o -name .bzrignore \
 		-o -name MANIFEST \
-		-print
+		-o -print
 
 # Perform some verification checks on any modified files.
 patchcheck:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:04 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:04 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzEyNzYxOiBmaXgg?=
 =?utf8?q?wording_of_zlib_license_section?=
Message-ID: <E1R06P6-0002Xq-4X@dinsdale.python.org>

http://hg.python.org/cpython/rev/5b21312c3800
changeset:   72228:5b21312c3800
branch:      3.2
user:        Sandro Tosi <sandro.tosi at gmail.com>
date:        Tue Aug 16 20:03:11 2011 +0200
summary:
  #12761: fix wording of zlib license section

files:
  Doc/license.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/license.rst b/Doc/license.rst
--- a/Doc/license.rst
+++ b/Doc/license.rst
@@ -881,7 +881,7 @@
 ----
 
 The :mod:`zlib` extension is built using an included copy of the zlib
-sources unless the zlib version found on the system is too old to be
+sources if the zlib version found on the system is too old to be
 used for the build::
 
   Copyright (C) 1995-2011 Jean-loup Gailly and Mark Adler

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:09 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:09 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyNjcy?=
 =?utf8?q?=3A_remove_confusing_part_of_sentence_in_documentation?=
Message-ID: <E1R06PB-0002Xu-4J@dinsdale.python.org>

http://hg.python.org/cpython/rev/8083e51522ee
changeset:   72229:8083e51522ee
branch:      3.2
user:        Eli Bendersky <eliben at gmail.com>
date:        Fri Aug 19 06:29:51 2011 +0300
summary:
  Issue #12672: remove confusing part of sentence in documentation

files:
  Doc/extending/newtypes.rst |  3 +--
  1 files changed, 1 insertions(+), 2 deletions(-)


diff --git a/Doc/extending/newtypes.rst b/Doc/extending/newtypes.rst
--- a/Doc/extending/newtypes.rst
+++ b/Doc/extending/newtypes.rst
@@ -30,8 +30,7 @@
 just contains the refcount and a pointer to the object's "type object".  This is
 where the action is; the type object determines which (C) functions get called
 when, for instance, an attribute gets looked up on an object or it is multiplied
-by another object.  These C functions are called "type methods" to distinguish
-them from things like ``[].append`` (which we call "object methods").
+by another object.  These C functions are called "type methods".
 
 So, if you want to define a new object type, you need to create a new type
 object.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:12 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:12 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_mention_RFC1123?=
 =?utf8?q?_as_origin_of_4-year_digit=3B_thanks_to_John_Haxby_from_docs=40?=
Message-ID: <E1R06PE-0002Xy-F9@dinsdale.python.org>

http://hg.python.org/cpython/rev/8d92f08adf11
changeset:   72230:8d92f08adf11
branch:      3.2
user:        Sandro Tosi <sandro.tosi at gmail.com>
date:        Fri Aug 19 18:40:21 2011 +0200
summary:
  mention RFC1123 as origin of 4-year digit; thanks to John Haxby from docs@

files:
  Doc/library/time.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/library/time.rst b/Doc/library/time.rst
--- a/Doc/library/time.rst
+++ b/Doc/library/time.rst
@@ -553,6 +553,6 @@
    preferred  hour/minute offset is not supported by all ANSI C libraries. Also, a
    strict reading of the original 1982 :rfc:`822` standard calls for a two-digit
    year (%y rather than %Y), but practice moved to 4-digit years long before the
-   year 2000.  The 4-digit year has been mandated by :rfc:`2822`, which obsoletes
-   :rfc:`822`.
+   year 2000.  After that, :rfc:`822` became obsolete and the 4-digit year has
+   been first recommended by :rfc:`1123` and then mandated by :rfc:`2822`.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:14 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:14 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_fix_description?=
 =?utf8?q?_of_=5Cr=3B_thanks_to_Thomas_Waldmann_from_docs=40?=
Message-ID: <E1R06PG-0002Y2-3L@dinsdale.python.org>

http://hg.python.org/cpython/rev/6d3e82c3dc6e
changeset:   72231:6d3e82c3dc6e
branch:      3.2
user:        Sandro Tosi <sandro.tosi at gmail.com>
date:        Fri Aug 19 22:54:50 2011 +0200
summary:
  fix description of \r; thanks to Thomas Waldmann from docs@

files:
  Doc/library/re.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/re.rst b/Doc/library/re.rst
--- a/Doc/library/re.rst
+++ b/Doc/library/re.rst
@@ -635,7 +635,7 @@
    of *pattern* in *string* by the replacement *repl*.  If the pattern isn't found,
    *string* is returned unchanged.  *repl* can be a string or a function; if it is
    a string, any backslash escapes in it are processed.  That is, ``\n`` is
-   converted to a single newline character, ``\r`` is converted to a linefeed, and
+   converted to a single newline character, ``\r`` is converted to a carriage return, and
    so forth.  Unknown escapes such as ``\j`` are left alone.  Backreferences, such
    as ``\6``, are replaced with the substring matched by group 6 in the pattern.
    For example:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:17 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:17 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyMzI2?=
 =?utf8?q?=3A_sys=2Eplatform_is_now_always_=27linux2=27_on_Linux?=
Message-ID: <E1R06PJ-0002Y6-MU@dinsdale.python.org>

http://hg.python.org/cpython/rev/265da863d017
changeset:   72232:265da863d017
branch:      3.2
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sat Aug 20 14:01:05 2011 +0200
summary:
  Issue #12326: sys.platform is now always 'linux2' on Linux

Even if Python is compiled on Linux 3.

files:
  Misc/NEWS    |  4 ++++
  configure    |  1 +
  configure.in |  1 +
  3 files changed, 6 insertions(+), 0 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,10 @@
 Core and Builtins
 -----------------
 
+- Issue #12326: sys.platform is now always 'linux2' on Linux, even if Python
+  is compiled on Linux 3.
+
+
 Library
 -------
 
diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -2997,6 +2997,7 @@
 	MACHDEP="$ac_md_system$ac_md_release"
 
 	case $MACHDEP in
+	linux*) MACHDEP="linux2";;
 	cygwin*) MACHDEP="cygwin";;
 	darwin*) MACHDEP="darwin";;
         irix646) MACHDEP="irix6";;
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -290,6 +290,7 @@
 	MACHDEP="$ac_md_system$ac_md_release"
 
 	case $MACHDEP in
+	linux*) MACHDEP="linux2";;
 	cygwin*) MACHDEP="cygwin";;
 	darwin*) MACHDEP="darwin";;
         irix646) MACHDEP="irix6";;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:19 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:19 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyMjEz?=
 =?utf8?q?=3A_make_it_clear_that_BufferedRWPair_shouldn=27t_be_called_with?=
 =?utf8?q?_the?=
Message-ID: <E1R06PL-0002YA-4I@dinsdale.python.org>

http://hg.python.org/cpython/rev/8de8945ebfc3
changeset:   72233:8de8945ebfc3
branch:      3.2
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Aug 20 19:48:43 2011 +0200
summary:
  Issue #12213: make it clear that BufferedRWPair shouldn't be called with the
same object as reader and writer, and deemphasize it in document order.

files:
  Doc/library/io.rst |  42 ++++++++++++++++++---------------
  1 files changed, 23 insertions(+), 19 deletions(-)


diff --git a/Doc/library/io.rst b/Doc/library/io.rst
--- a/Doc/library/io.rst
+++ b/Doc/library/io.rst
@@ -607,25 +607,6 @@
       if the buffer needs to be written out but the raw stream blocks.
 
 
-.. class:: BufferedRWPair(reader, writer, buffer_size=DEFAULT_BUFFER_SIZE)
-
-   A buffered I/O object giving a combined, higher-level access to two
-   sequential :class:`RawIOBase` objects: one readable, the other writeable.
-   It is useful for pairs of unidirectional communication channels
-   (pipes, for instance).  It inherits :class:`BufferedIOBase`.
-
-   *reader* and *writer* are :class:`RawIOBase` objects that are readable and
-   writeable respectively.  If the *buffer_size* is omitted it defaults to
-   :data:`DEFAULT_BUFFER_SIZE`.
-
-   A fourth argument, *max_buffer_size*, is supported, but unused and
-   deprecated.
-
-   :class:`BufferedRWPair` implements all of :class:`BufferedIOBase`\'s methods
-   except for :meth:`~BufferedIOBase.detach`, which raises
-   :exc:`UnsupportedOperation`.
-
-
 .. class:: BufferedRandom(raw, buffer_size=DEFAULT_BUFFER_SIZE)
 
    A buffered interface to random access streams.  It inherits
@@ -642,6 +623,29 @@
    :class:`BufferedWriter` can do.
 
 
+.. class:: BufferedRWPair(reader, writer, buffer_size=DEFAULT_BUFFER_SIZE)
+
+   A buffered I/O object combining two unidirectional :class:`RawIOBase`
+   objects -- one readable, the other writeable -- into a single bidirectional
+   endpoint.  It inherits :class:`BufferedIOBase`.
+
+   *reader* and *writer* are :class:`RawIOBase` objects that are readable and
+   writeable respectively.  If the *buffer_size* is omitted it defaults to
+   :data:`DEFAULT_BUFFER_SIZE`.
+
+   A fourth argument, *max_buffer_size*, is supported, but unused and
+   deprecated.
+
+   :class:`BufferedRWPair` implements all of :class:`BufferedIOBase`\'s methods
+   except for :meth:`~BufferedIOBase.detach`, which raises
+   :exc:`UnsupportedOperation`.
+
+   .. warning::
+      :class:`BufferedRWPair` does not attempt to synchronize accesses to
+      its underlying raw streams.  You should not pass it the same object
+      as reader and writer; use :class:`BufferedRandom` instead.
+
+
 Text I/O
 ^^^^^^^^
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:20 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:20 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzUzMDE6IGFkZCBp?=
 =?utf8?q?mage/vnd=2Emicrosoft=2Eicon_=28=2Eico=29_MIME_type?=
Message-ID: <E1R06PM-0002YE-Cs@dinsdale.python.org>

http://hg.python.org/cpython/rev/cba2558b73ee
changeset:   72234:cba2558b73ee
branch:      3.2
user:        Sandro Tosi <sandro.tosi at gmail.com>
date:        Sun Aug 21 00:16:18 2011 +0200
summary:
  #5301: add image/vnd.microsoft.icon (.ico) MIME type

files:
  Lib/mimetypes.py |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Lib/mimetypes.py b/Lib/mimetypes.py
--- a/Lib/mimetypes.py
+++ b/Lib/mimetypes.py
@@ -425,6 +425,7 @@
         '.hdf'    : 'application/x-hdf',
         '.htm'    : 'text/html',
         '.html'   : 'text/html',
+        '.ico'    : 'image/vnd.microsoft.icon',
         '.ief'    : 'image/ief',
         '.jpe'    : 'image/jpeg',
         '.jpeg'   : 'image/jpeg',

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:21 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:21 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Add_missing_nam?=
	=?utf8?q?e_in_shutil?=
Message-ID: <E1R06PN-0002YI-N8@dinsdale.python.org>

http://hg.python.org/cpython/rev/2f549ddd5bdc
changeset:   72235:2f549ddd5bdc
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Aug 21 14:29:18 2011 +0200
summary:
  Add missing name in shutil

files:
  Lib/shutil.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/shutil.py b/Lib/shutil.py
--- a/Lib/shutil.py
+++ b/Lib/shutil.py
@@ -34,7 +34,7 @@
            "ExecError", "make_archive", "get_archive_formats",
            "register_archive_format", "unregister_archive_format",
            "get_unpack_formats", "register_unpack_format",
-           "unregister_unpack_format", "unpack_archive"]
+           "unregister_unpack_format", "unpack_archive", "ignore_patterns"]
 
 class Error(EnvironmentError):
     pass

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:22 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:22 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_A_warning_doesn?=
	=?utf8?q?=27t_equate_a_failed_test?=
Message-ID: <E1R06PO-0002Yo-Vm@dinsdale.python.org>

http://hg.python.org/cpython/rev/cbf4024d3202
changeset:   72236:cbf4024d3202
branch:      3.2
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue Aug 23 19:32:26 2011 +0200
summary:
  A warning doesn't equate a failed test
(this broken -F with e.g. test_multiprocessing)

files:
  Lib/test/regrtest.py |  1 -
  1 files changed, 0 insertions(+), 1 deletions(-)


diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -515,7 +515,6 @@
         elif ok == FAILED:
             bad.append(test)
         elif ok == ENV_CHANGED:
-            bad.append(test)
             environment_changed.append(test)
         elif ok == SKIPPED:
             skipped.append(test)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:24 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:24 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogQ2xvc2UgIzEyODM4?=
	=?utf8?q?=3A_fix_range=28=29_call=2E?=
Message-ID: <E1R06PQ-0002ZN-3j@dinsdale.python.org>

http://hg.python.org/cpython/rev/6e156a6c4fd9
changeset:   72237:6e156a6c4fd9
branch:      3.2
user:        Georg Brandl <georg at python.org>
date:        Thu Aug 25 11:52:26 2011 +0200
summary:
  Close #12838: fix range() call.

files:
  Doc/faq/programming.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst
--- a/Doc/faq/programming.rst
+++ b/Doc/faq/programming.rst
@@ -171,7 +171,7 @@
 Thus to get the same effect as::
 
    L2 = []
-   for i in range[3]:
+   for i in range(3):
        L2.append(L1[i])
 
 it is much shorter and far faster to use ::

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:25 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:25 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Document_the_?=
 =?utf8?q?=22optional=22_argument_of_distutils=E2=80=99_Extension_class?=
Message-ID: <E1R06PR-0002ZR-B6@dinsdale.python.org>

http://hg.python.org/cpython/rev/948691b28ede
changeset:   72238:948691b28ede
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Aug 26 00:45:18 2011 +0200
summary:
  Document the "optional" argument of distutils? Extension class

files:
  Doc/distutils/apiref.rst |  5 +++++
  1 files changed, 5 insertions(+), 0 deletions(-)


diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst
--- a/Doc/distutils/apiref.rst
+++ b/Doc/distutils/apiref.rst
@@ -261,6 +261,11 @@
    |                        | from the source extensions if  |                           |
    |                        | not provided.                  |                           |
    +------------------------+--------------------------------+---------------------------+
+   | *optional*             | specifies that a build failure | a boolean                 |
+   |                        | in the extension should not    |                           |
+   |                        | abort the build process, but   |                           |
+   |                        | simply skip the extension.     |                           |
+   +------------------------+--------------------------------+---------------------------+
 
 
 .. class:: Distribution

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:26 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:26 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Remove_outdated?=
 =?utf8?q?_pointer_to_optparse_=28fixes_=2311360=29=2E?=
Message-ID: <E1R06PS-0002ZV-DH@dinsdale.python.org>

http://hg.python.org/cpython/rev/dff0f4db67da
changeset:   72239:dff0f4db67da
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Aug 26 16:38:40 2011 +0200
summary:
  Remove outdated pointer to optparse (fixes #11360).

The doc already points to argparse.

files:
  Doc/library/getopt.rst |  3 ---
  1 files changed, 0 insertions(+), 3 deletions(-)


diff --git a/Doc/library/getopt.rst b/Doc/library/getopt.rst
--- a/Doc/library/getopt.rst
+++ b/Doc/library/getopt.rst
@@ -22,9 +22,6 @@
 options similar to those supported by GNU software may be used as well via an
 optional third argument.
 
-A more convenient, flexible, and powerful alternative is the
-:mod:`optparse` module.
-
 This module provides two functions and an
 exception:
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:27 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:27 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Make_regrtest_c?=
 =?utf8?q?omplain_when_-M_and_-j_are_used_together=2E?=
Message-ID: <E1R06PT-0002ZZ-U6@dinsdale.python.org>

http://hg.python.org/cpython/rev/34bafe09dc2b
changeset:   72240:34bafe09dc2b
branch:      3.2
user:        Nadeem Vawda <nadeem.vawda at gmail.com>
date:        Sat Aug 27 15:22:05 2011 +0200
summary:
  Make regrtest complain when -M and -j are used together.

-j doesn't pass the memlimit on to child processes, so this doesn't work at
present, and even if it did, running multiple bigmem tests at once would
usually not be desirable (since you generally want to devote as much of the
available RAM as possible to each test).

files:
  Lib/test/regrtest.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -412,6 +412,8 @@
         usage("-T and -j don't go together!")
     if use_mp and findleaks:
         usage("-l and -j don't go together!")
+    if use_mp and support.max_memuse:
+        usage("-M and -j don't go together!")
     if failfast and not (verbose or verbose3):
         usage("-G/--failfast needs either -v or -W")
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:29 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:29 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyODM5?=
 =?utf8?q?=3A_Fix_crash_in_zlib_module_due_to_version_mismatch=2E?=
Message-ID: <E1R06PV-0002Zd-9c@dinsdale.python.org>

http://hg.python.org/cpython/rev/2fb00b6ed17c
changeset:   72241:2fb00b6ed17c
branch:      3.2
user:        Nadeem Vawda <nadeem.vawda at gmail.com>
date:        Sun Aug 28 11:26:46 2011 +0200
summary:
  Issue #12839: Fix crash in zlib module due to version mismatch.

If the version of zlib used to compile the zlib module is incompatible
with the one that is actually linked in, then calls into zlib will fail.
This can leave attributes of the z_stream uninitialized, so we must take
care to avoid segfaulting by trying to use an invalid pointer.

Fix by Richard M. Tew.

files:
  Misc/ACKS            |  1 +
  Misc/NEWS            |  4 ++++
  Modules/zlibmodule.c |  8 +++++++-
  3 files changed, 12 insertions(+), 1 deletions(-)


diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -873,6 +873,7 @@
 Amy Taylor
 Anatoly Techtonik
 Mikhail Terekhov
+Richard M. Tew
 Tobias Thelen
 James Thomas
 Robin Thomas
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -17,6 +17,10 @@
 Library
 -------
 
+- Issue #12839: Fix crash in zlib module due to version mismatch.
+  Fix by Richard M. Tew.
+
+
 
 What's New in Python 3.2.2 release candidate 1?
 ===============================================
diff --git a/Modules/zlibmodule.c b/Modules/zlibmodule.c
--- a/Modules/zlibmodule.c
+++ b/Modules/zlibmodule.c
@@ -52,7 +52,13 @@
 static void
 zlib_error(z_stream zst, int err, char *msg)
 {
-    const char *zmsg = zst.msg;
+    const char *zmsg = Z_NULL;
+    /* In case of a version mismatch, zst.msg won't be initialized.
+       Check for this case first, before looking at zst.msg. */
+    if (err == Z_VERSION_ERROR)
+        zmsg = "library version mismatch";
+    if (zmsg == Z_NULL)
+        zmsg = zst.msg;
     if (zmsg == Z_NULL) {
         switch (err) {
         case Z_BUF_ERROR:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:30 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:30 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzk2NTE6?=
 =?utf8?q?_Fix_a_crash_when_ctypes=2Ecreate=5Fstring=5Fbuffer=280=29_was_p?=
 =?utf8?q?assed_to?=
Message-ID: <E1R06PW-0002Zh-KR@dinsdale.python.org>

http://hg.python.org/cpython/rev/f19d00b5220a
changeset:   72242:f19d00b5220a
branch:      3.2
user:        Amaury Forgeot d'Arc <amauryfa at gmail.com>
date:        Tue Aug 30 21:40:20 2011 +0200
summary:
  Issue #9651: Fix a crash when ctypes.create_string_buffer(0) was passed to
some functions like file.write().

files:
  Lib/ctypes/test/test_buffers.py |  4 ++++
  Misc/NEWS                       |  3 +++
  Modules/_ctypes/_ctypes.c       |  6 ++++--
  3 files changed, 11 insertions(+), 2 deletions(-)


diff --git a/Lib/ctypes/test/test_buffers.py b/Lib/ctypes/test/test_buffers.py
--- a/Lib/ctypes/test/test_buffers.py
+++ b/Lib/ctypes/test/test_buffers.py
@@ -20,6 +20,10 @@
         self.assertEqual(b[::2], b"ac")
         self.assertEqual(b[::5], b"a")
 
+    def test_buffer_interface(self):
+        self.assertEqual(len(bytearray(create_string_buffer(0))), 0)
+        self.assertEqual(len(bytearray(create_string_buffer(1))), 1)
+
     try:
         c_wchar
     except NameError:
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -152,6 +152,9 @@
 Extension Modules
 -----------------
 
+- Issue #9651: Fix a crash when ctypes.create_string_buffer(0) was passed to
+  some functions like file.write().
+
 - Issue #10309: Define _GNU_SOURCE so that mremap() gets the proper
   signature.  Without this, architectures where sizeof void* != sizeof int are
   broken.  Patch given by Hallvard B Furuseth.
diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c
--- a/Modules/_ctypes/_ctypes.c
+++ b/Modules/_ctypes/_ctypes.c
@@ -2488,8 +2488,10 @@
     view->ndim = dict->ndim;
     view->shape = dict->shape;
     view->itemsize = self->b_size;
-    for (i = 0; i < view->ndim; ++i) {
-        view->itemsize /= dict->shape[i];
+    if (view->itemsize) {
+        for (i = 0; i < view->ndim; ++i) {
+            view->itemsize /= dict->shape[i];
+        }
     }
     view->strides = NULL;
     view->suboffsets = NULL;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:31 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:31 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix-up_NEWS_mer?=
	=?utf8?b?Z2Uu?=
Message-ID: <E1R06PX-0002Zl-P4@dinsdale.python.org>

http://hg.python.org/cpython/rev/c49474f7f986
changeset:   72243:c49474f7f986
branch:      3.2
user:        Georg Brandl <georg at python.org>
date:        Sat Sep 03 09:08:49 2011 +0200
summary:
  Fix-up NEWS merge.

files:
  Misc/NEWS |  10 ++++++----
  1 files changed, 6 insertions(+), 4 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -13,13 +13,18 @@
 - Issue #12326: sys.platform is now always 'linux2' on Linux, even if Python
   is compiled on Linux 3.
 
-
 Library
 -------
 
 - Issue #12839: Fix crash in zlib module due to version mismatch.
   Fix by Richard M. Tew.
 
+Extension Modules
+-----------------
+
+- Issue #9651: Fix a crash when ctypes.create_string_buffer(0) was passed to
+  some functions like file.write().
+
 
 
 What's New in Python 3.2.2 release candidate 1?
@@ -152,9 +157,6 @@
 Extension Modules
 -----------------
 
-- Issue #9651: Fix a crash when ctypes.create_string_buffer(0) was passed to
-  some functions like file.write().
-
 - Issue #10309: Define _GNU_SOURCE so that mremap() gets the proper
   signature.  Without this, architectures where sizeof void* != sizeof int are
   broken.  Patch given by Hallvard B Furuseth.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:33 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:33 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_accept_bytes_fo?=
 =?utf8?q?r_the_AST_=27string=27_type?=
Message-ID: <E1R06PZ-0002Zq-5w@dinsdale.python.org>

http://hg.python.org/cpython/rev/1ad7e71ebd5f
changeset:   72244:1ad7e71ebd5f
branch:      3.2
user:        Benjamin Peterson <benjamin at python.org>
date:        Wed Aug 31 22:13:03 2011 -0400
summary:
  accept bytes for the AST 'string' type

This is a temporary kludge and all is well in 3.3.

files:
  Misc/NEWS           |  3 +++
  Parser/asdl_c.py    |  2 +-
  Python/Python-ast.c |  2 +-
  3 files changed, 5 insertions(+), 2 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -13,6 +13,9 @@
 - Issue #12326: sys.platform is now always 'linux2' on Linux, even if Python
   is compiled on Linux 3.
 
+- Accept bytes for the AST string type. This is temporary until a proper fix in
+  3.3.
+
 Library
 -------
 
diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py
--- a/Parser/asdl_c.py
+++ b/Parser/asdl_c.py
@@ -805,7 +805,7 @@
 
 static int obj2ast_string(PyObject* obj, PyObject** out, PyArena* arena)
 {
-    if (!PyUnicode_CheckExact(obj)) {
+    if (!PyUnicode_CheckExact(obj) && !PyBytes_CheckExact(obj)) {
         PyErr_SetString(PyExc_TypeError, "AST string must be of type str");
         return 1;
     }
diff --git a/Python/Python-ast.c b/Python/Python-ast.c
--- a/Python/Python-ast.c
+++ b/Python/Python-ast.c
@@ -611,7 +611,7 @@
 
 static int obj2ast_string(PyObject* obj, PyObject** out, PyArena* arena)
 {
-    if (!PyUnicode_CheckExact(obj)) {
+    if (!PyUnicode_CheckExact(obj) && !PyBytes_CheckExact(obj)) {
         PyErr_SetString(PyExc_TypeError, "AST string must be of type str");
         return 1;
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:34 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:34 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_some_misuse?=
 =?utf8?q?s_of_Sphinx_roles_and_one_typo?=
Message-ID: <E1R06Pa-0002Zu-D5@dinsdale.python.org>

http://hg.python.org/cpython/rev/8d06fbff1187
changeset:   72245:8d06fbff1187
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 03:19:30 2011 +0200
summary:
  Fix some misuses of Sphinx roles and one typo

files:
  Doc/c-api/init.rst       |  4 ++--
  Doc/faq/design.rst       |  2 +-
  Doc/faq/windows.rst      |  2 +-
  Doc/howto/logging.rst    |  6 +++---
  Doc/library/argparse.rst |  2 +-
  5 files changed, 8 insertions(+), 8 deletions(-)


diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -122,7 +122,7 @@
    program name is ``'/usr/local/bin/python'``, the prefix is ``'/usr/local'``. The
    returned string points into static storage; the caller should not modify its
    value.  This corresponds to the :makevar:`prefix` variable in the top-level
-   :file:`Makefile` and the :option:`--prefix` argument to the :program:`configure`
+   :file:`Makefile` and the ``--prefix`` argument to the :program:`configure`
    script at build time.  The value is available to Python code as ``sys.prefix``.
    It is only useful on Unix.  See also the next function.
 
@@ -135,7 +135,7 @@
    program name is ``'/usr/local/bin/python'``, the exec-prefix is
    ``'/usr/local'``.  The returned string points into static storage; the caller
    should not modify its value.  This corresponds to the :makevar:`exec_prefix`
-   variable in the top-level :file:`Makefile` and the :option:`--exec-prefix`
+   variable in the top-level :file:`Makefile` and the ``--exec-prefix``
    argument to the :program:`configure` script at build  time.  The value is
    available to Python code as ``sys.exec_prefix``.  It is only useful on Unix.
 
diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst
--- a/Doc/faq/design.rst
+++ b/Doc/faq/design.rst
@@ -667,7 +667,7 @@
 Python 2.6 adds an :mod:`abc` module that lets you define Abstract Base Classes
 (ABCs).  You can then use :func:`isinstance` and :func:`issubclass` to check
 whether an instance or a class implements a particular ABC.  The
-:mod:`collections` modules defines a set of useful ABCs such as
+:mod:`collections` module defines a set of useful ABCs such as
 :class:`Iterable`, :class:`Container`, and :class:`MutableMapping`.
 
 For Python, many of the advantages of interface specifications can be obtained
diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst
--- a/Doc/faq/windows.rst
+++ b/Doc/faq/windows.rst
@@ -546,7 +546,7 @@
 :func:`execfile` with the name of your file as argument.
 
 Also note that you can not mix-and-match Debug and Release versions.  If you
-wish to use the Debug Multithreaded DLL, then your module *must* have an "_d"
+wish to use the Debug Multithreaded DLL, then your module *must* have ``_d``
 appended to the base name.
 
 
diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst
--- a/Doc/howto/logging.rst
+++ b/Doc/howto/logging.rst
@@ -412,10 +412,10 @@
   :meth:`Logger.error`, and :meth:`Logger.critical` all create log records with
   a message and a level that corresponds to their respective method names. The
   message is actually a format string, which may contain the standard string
-  substitution syntax of :const:`%s`, :const:`%d`, :const:`%f`, and so on.  The
+  substitution syntax of ``%s``, ``%d``, ``%f``, and so on.  The
   rest of their arguments is a list of objects that correspond with the
-  substitution fields in the message.  With regard to :const:`**kwargs`, the
-  logging methods care only about a keyword of :const:`exc_info` and use it to
+  substitution fields in the message.  With regard to ``**kwargs``, the
+  logging methods care only about a keyword of ``exc_info`` and use it to
   determine whether to log exception information.
 
 * :meth:`Logger.exception` creates a log message similar to
diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst
--- a/Doc/library/argparse.rst
+++ b/Doc/library/argparse.rst
@@ -155,7 +155,7 @@
      conflicting optionals.
 
    * prog_ - The name of the program (default:
-     :data:`sys.argv[0]`)
+     ``sys.argv[0]``)
 
    * usage_ - The string describing the program usage (default: generated)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:35 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:35 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Adapt/remove_me?=
 =?utf8?q?ntions_of_functions_gone_in_3=2Ex?=
Message-ID: <E1R06Pb-0002Zy-J5@dinsdale.python.org>

http://hg.python.org/cpython/rev/6972e2a45a57
changeset:   72246:6972e2a45a57
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 03:20:13 2011 +0200
summary:
  Adapt/remove mentions of functions gone in 3.x

files:
  Doc/faq/programming.rst |  9 ---------
  Doc/faq/windows.rst     |  2 +-
  Doc/glossary.rst        |  2 +-
  3 files changed, 2 insertions(+), 11 deletions(-)


diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst
--- a/Doc/faq/programming.rst
+++ b/Doc/faq/programming.rst
@@ -473,15 +473,6 @@
        ...
        g(x, *args, **kwargs)
 
-In the unlikely case that you care about Python versions older than 2.0, use
-:func:`apply`::
-
-   def f(x, *args, **kwargs):
-       ...
-       kwargs['width'] = '14.3c'
-       ...
-       apply(g, (x,)+args, kwargs)
-
 
 How do I write a function with output parameters (call by reference)?
 ---------------------------------------------------------------------
diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst
--- a/Doc/faq/windows.rst
+++ b/Doc/faq/windows.rst
@@ -543,7 +543,7 @@
 
 If you can't change compilers or flags, try using :c:func:`Py_RunSimpleString`.
 A trick to get it to run an arbitrary file is to construct a call to
-:func:`execfile` with the name of your file as argument.
+:func:`exec` and :func:`open` with the name of your file as argument.
 
 Also note that you can not mix-and-match Debug and Release versions.  If you
 wish to use the Debug Multithreaded DLL, then your module *must* have ``_d``
diff --git a/Doc/glossary.rst b/Doc/glossary.rst
--- a/Doc/glossary.rst
+++ b/Doc/glossary.rst
@@ -489,7 +489,7 @@
       :func:`builtins.open` and :func:`os.open` are distinguished by their
       namespaces.  Namespaces also aid readability and maintainability by making
       it clear which module implements a function.  For instance, writing
-      :func:`random.seed` or :func:`itertools.izip` makes it clear that those
+      :func:`random.seed` or :func:`itertools.islice` makes it clear that those
       functions are implemented by the :mod:`random` and :mod:`itertools`
       modules, respectively.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:36 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:36 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Add_version_num?=
 =?utf8?q?ber_for_versionchanged_directive_=28backport_from_3=2E3=29?=
Message-ID: <E1R06Pc-0002a2-LZ@dinsdale.python.org>

http://hg.python.org/cpython/rev/062a3305a606
changeset:   72247:062a3305a606
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 05:55:26 2011 +0200
summary:
  Add version number for versionchanged directive (backport from 3.3)

files:
  Doc/library/unittest.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -723,8 +723,8 @@
    Here, we create two instances of :class:`WidgetTestCase`, each of which runs a
    single test.
 
-   .. versionchanged::
-      `TestCase` can be instantiated successfully without providing a method
+   .. versionchanged:: 3.2
+      :class:`TestCase` can be instantiated successfully without providing a method
       name. This makes it easier to experiment with `TestCase` from the
       interactive interpreter.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:37 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:37 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Document_that_f?=
 =?utf8?q?ormat_string_don=E2=80=99t_support_arbitrary_dictonary_keys=2E?=
Message-ID: <E1R06Pd-0002a6-Fm@dinsdale.python.org>

http://hg.python.org/cpython/rev/2c27eb4033f5
changeset:   72248:2c27eb4033f5
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 01 18:59:06 2011 +0200
summary:
  Document that format string don?t support arbitrary dictonary keys.

Text adapted from the PEP.  Addition requested by Terry J. Reedy on
2011-02-23 on python-dev.

files:
  Doc/library/string.rst |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Doc/library/string.rst b/Doc/library/string.rst
--- a/Doc/library/string.rst
+++ b/Doc/library/string.rst
@@ -217,6 +217,8 @@
 it refers to a named keyword argument.  If the numerical arg_names in a format string
 are 0, 1, 2, ... in sequence, they can all be omitted (not just some)
 and the numbers 0, 1, 2, ... will be automatically inserted in that order.
+Because *arg_name* is not quote-delimited, it is not possible to specify arbitrary
+dictionary keys (e.g., the strings ``'10'`` or ``':-]'``) within a format string.
 The *arg_name* can be followed by any number of index or
 attribute expressions. An expression of the form ``'.name'`` selects the named
 attribute using :func:`getattr`, while an expression of the form ``'[index]'``

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:38 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:38 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzEyNzgxOiBNZW50?=
 =?utf8?q?ion_SO=5FREUSEADDR_flag_near_socket_examples?=
Message-ID: <E1R06Pe-0002aA-7S@dinsdale.python.org>

http://hg.python.org/cpython/rev/126069a5ecf6
changeset:   72249:126069a5ecf6
branch:      3.2
user:        Sandro Tosi <sandro.tosi at gmail.com>
date:        Fri Sep 02 20:06:31 2011 +0200
summary:
  #12781: Mention SO_REUSEADDR flag near socket examples

files:
  Doc/library/socket.rst |  19 +++++++++++++++++++
  1 files changed, 19 insertions(+), 0 deletions(-)


diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst
--- a/Doc/library/socket.rst
+++ b/Doc/library/socket.rst
@@ -1014,6 +1014,25 @@
    s.ioctl(socket.SIO_RCVALL, socket.RCVALL_OFF)
 
 
+Running an example several times with too small delay between executions, could
+lead to this error::
+
+   socket.error: [Errno 98] Address already in use
+
+This is because the previous execution has left the socket in a ``TIME_WAIT``
+state, and can't be immediately reused.
+
+There is a :mod:`socket` flag to set, in order to prevent this,
+:data:`socket.SO_REUSEADDR`::
+
+   s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+   s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+   s.bind((HOST, PORT))
+
+the :data:`SO_REUSEADDR` flag tells the kernel to reuse a local socket in
+``TIME_WAIT`` state, without waiting for its natural timeout to expire.
+
+
 .. seealso::
 
    For an introduction to socket programming (in C), see the following papers:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:38 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:38 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_removed_mislead?=
	=?utf8?q?ing_editing_leftovers?=
Message-ID: <E1R06Pe-0002aE-R9@dinsdale.python.org>

http://hg.python.org/cpython/rev/dba0bb3f67b8
changeset:   72250:dba0bb3f67b8
branch:      3.2
user:        ?ukasz Langa <lukasz at langa.pl>
date:        Fri Sep 02 23:17:39 2011 +0200
summary:
  removed misleading editing leftovers

files:
  Doc/library/configparser.rst |  4 ----
  1 files changed, 0 insertions(+), 4 deletions(-)


diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst
--- a/Doc/library/configparser.rst
+++ b/Doc/library/configparser.rst
@@ -865,10 +865,6 @@
    Comments can be indented. When *inline_comment_prefixes* is given, it will be
    used as the set of substrings that prefix comments in non-empty lines.
 
-   line and inline comments.  For backwards compatibility, the default value for
-   *comment_prefixes* is a special value that indicates that ``;`` and ``#`` can
-   start whole line comments while only ``;`` can start inline comments.
-
    When *strict* is ``True`` (the default), the parser won't allow for
    any section or option duplicates while reading from a single source (file,
    string or dictionary), raising :exc:`DuplicateSectionError` or

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:39 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:39 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Update_sys=2Epl?=
	=?utf8?q?atform_doc_for_=2312326=2E?=
Message-ID: <E1R06Pf-0002aI-Di@dinsdale.python.org>

http://hg.python.org/cpython/rev/e11b4c945f7e
changeset:   72251:e11b4c945f7e
branch:      3.2
user:        Georg Brandl <georg at python.org>
date:        Sat Sep 03 09:26:09 2011 +0200
summary:
  Update sys.platform doc for #12326.

files:
  Doc/library/sys.rst |  40 ++++++++++++++++++++------------
  1 files changed, 25 insertions(+), 15 deletions(-)


diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -699,26 +699,36 @@
    This string contains a platform identifier that can be used to append
    platform-specific components to :data:`sys.path`, for instance.
 
-   For Unix systems, this is the lowercased OS name as returned by ``uname -s``
-   with the first part of the version as returned by ``uname -r`` appended,
-   e.g. ``'sunos5'`` or ``'linux2'``, *at the time when Python was built*.
-   Unless you want to test for a specific system version, it is therefore
-   recommended to use the following idiom::
+   For most Unix systems, this is the lowercased OS name as returned by ``uname
+   -s`` with the first part of the version as returned by ``uname -r`` appended,
+   e.g. ``'sunos5'``, *at the time when Python was built*.  Unless you want to
+   test for a specific system version, it is therefore recommended to use the
+   following idiom::
 
-      if sys.platform.startswith('linux'):
+      if sys.platform.startswith('freebsd'):
+          # FreeBSD-specific code here...
+      elif sys.platform.startswith('linux'):
           # Linux-specific code here...
 
+   .. versionchanged:: 3.2.2
+      Since lots of code check for ``sys.platform == 'linux2'``, and there is
+      no essential change between Linux 2.x and 3.x, ``sys.platform`` is always
+      set to ``'linux2'``, even on Linux 3.x.  In Python 3.3 and later, the
+      value will always be set to ``'linux'``, so it is recommended to always
+      use the ``startswith`` idiom presented above.
+
    For other systems, the values are:
 
-   ================ ===========================
-   System           :data:`platform` value
-   ================ ===========================
-   Windows          ``'win32'``
-   Windows/Cygwin   ``'cygwin'``
-   Mac OS X         ``'darwin'``
-   OS/2             ``'os2'``
-   OS/2 EMX         ``'os2emx'``
-   ================ ===========================
+   ====================== ===========================
+   System                 :data:`platform` value
+   ====================== ===========================
+   Linux (2.x *and* 3.x)  ``'linux2'``
+   Windows                ``'win32'``
+   Windows/Cygwin         ``'cygwin'``
+   Mac OS X               ``'darwin'``
+   OS/2                   ``'os2'``
+   OS/2 EMX               ``'os2emx'``
+   ====================== ===========================
 
    .. seealso::
       :attr:`os.name` has a coarser granularity.  :func:`os.uname` gives

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:40 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:40 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Update_suspicio?=
	=?utf8?q?us_ignore_file=2E?=
Message-ID: <E1R06Pg-0002aM-1i@dinsdale.python.org>

http://hg.python.org/cpython/rev/7085746fb73f
changeset:   72252:7085746fb73f
branch:      3.2
user:        Georg Brandl <georg at python.org>
date:        Sat Sep 03 09:28:05 2011 +0200
summary:
  Update suspicious ignore file.

files:
  Doc/tools/sphinxext/susp-ignored.csv |  8 ++++----
  1 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/Doc/tools/sphinxext/susp-ignored.csv b/Doc/tools/sphinxext/susp-ignored.csv
--- a/Doc/tools/sphinxext/susp-ignored.csv
+++ b/Doc/tools/sphinxext/susp-ignored.csv
@@ -193,10 +193,10 @@
 documenting/rest,187,.. function:,.. function:: foo(x)
 documenting/rest,187,:bar,:bar: no
 documenting/rest,208,.. rubric:,.. rubric:: Footnotes
-faq/programming,762,:reduce,"print((lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+y,map(lambda y,"
-faq/programming,762,:reduce,"Sx=Sx,Sy=Sy:reduce(lambda x,y:x+y,map(lambda x,xc=Ru,yc=yc,Ru=Ru,Ro=Ro,"
-faq/programming,762,:chr,">=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr("
-faq/programming,1047,::,for x in sequence[::-1]:
+faq/programming,,:reduce,"print((lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+y,map(lambda y,"
+faq/programming,,:reduce,"Sx=Sx,Sy=Sy:reduce(lambda x,y:x+y,map(lambda x,xc=Ru,yc=yc,Ru=Ru,Ro=Ro,"
+faq/programming,,:chr,">=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr("
+faq/programming,,::,for x in sequence[::-1]:
 faq/windows,229,:EOF, at setlocal enableextensions & python -x %~f0 %* & goto :EOF
 faq/windows,393,:REG,.py :REG_SZ: c:\<path to python>\python.exe -u %s %s
 library/bisect,32,:hi,all(val >= x for val in a[i:hi])

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:40 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:40 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Refactor_the_co?=
 =?utf8?q?pying_of_xxmodule=2Ec_in_distutils_tests_=28=2312141=29=2E?=
Message-ID: <E1R06Pg-0002aQ-KO@dinsdale.python.org>

http://hg.python.org/cpython/rev/402c4bbedf9c
changeset:   72253:402c4bbedf9c
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Aug 20 06:27:18 2011 +0200
summary:
  Refactor the copying of xxmodule.c in distutils tests (#12141).

I need to copy this file in another test too, so I moved the support
code to distutils.tests.support and improved it:

- don?t skip when run from the Lib/distutils/tests directory
- use proper skip machinery instead of custom print/return/test suite
  fiddling.

files:
  Lib/distutils/tests/support.py        |  42 +++++++++++++++
  Lib/distutils/tests/test_build_ext.py |  28 +--------
  2 files changed, 46 insertions(+), 24 deletions(-)


diff --git a/Lib/distutils/tests/support.py b/Lib/distutils/tests/support.py
--- a/Lib/distutils/tests/support.py
+++ b/Lib/distutils/tests/support.py
@@ -2,12 +2,15 @@
 import os
 import shutil
 import tempfile
+import unittest
+import sysconfig
 from copy import deepcopy
 
 from distutils import log
 from distutils.log import DEBUG, INFO, WARN, ERROR, FATAL
 from distutils.core import Distribution
 
+
 class LoggingSilencer(object):
 
     def setUp(self):
@@ -41,6 +44,7 @@
     def clear_logs(self):
         self.logs = []
 
+
 class TempdirManager(object):
     """Mix-in class that handles temporary directories for test cases.
 
@@ -97,6 +101,7 @@
 
         return pkg_dir, dist
 
+
 class DummyCommand:
     """Class to store options for retrieval via set_undefined_options()."""
 
@@ -107,6 +112,7 @@
     def ensure_finalized(self):
         pass
 
+
 class EnvironGuard(object):
 
     def setUp(self):
@@ -123,3 +129,39 @@
                 del os.environ[key]
 
         super(EnvironGuard, self).tearDown()
+
+
+def copy_xxmodule_c(directory):
+    """Helper for tests that need the xxmodule.c source file.
+
+    Example use:
+
+            def test_compile(self):
+                copy_xxmodule_c(self.tmpdir)
+                self.assertIn('xxmodule.c', os.listdir(self.tmpdir)
+
+    If the source file can be found, it will be copied to *directory*.  If not,
+    the test will be skipped.  Errors during copy are not caught.
+    """
+    filename = _get_xxmodule_path()
+    if filename is None:
+        raise unittest.SkipTest('cannot find xxmodule.c (test must run in '
+                                'the python build dir)')
+    shutil.copy(filename, directory)
+
+
+def _get_xxmodule_path():
+    srcdir = sysconfig.get_config_var('srcdir')
+    candidates = [
+        # use installed copy if available
+        os.path.join(os.path.dirname(__file__), 'xxmodule.c'),
+        # otherwise try using copy from build directory
+        os.path.join(srcdir, 'Modules', 'xxmodule.c'),
+        # srcdir mysteriously can be $srcdir/Lib/distutils/tests when
+        # this file is run from its parent directory, so walk up the
+        # tree to find the real srcdir
+        os.path.join(srcdir, '..', '..', '..', 'Modules', 'xxmodule.c'),
+    ]
+    for path in candidates:
+        if os.path.exists(path):
+            return path
diff --git a/Lib/distutils/tests/test_build_ext.py b/Lib/distutils/tests/test_build_ext.py
--- a/Lib/distutils/tests/test_build_ext.py
+++ b/Lib/distutils/tests/test_build_ext.py
@@ -1,14 +1,13 @@
 import sys
 import os
-import shutil
 from io import StringIO
 import textwrap
 
 from distutils.core import Distribution
 from distutils.command.build_ext import build_ext
 from distutils import sysconfig
-from distutils.tests.support import TempdirManager
-from distutils.tests.support import LoggingSilencer
+from distutils.tests.support import (TempdirManager, LoggingSilencer,
+                                     copy_xxmodule_c)
 from distutils.extension import Extension
 from distutils.errors import (
     CompileError, DistutilsPlatformError, DistutilsSetupError,
@@ -16,20 +15,11 @@
 
 import unittest
 from test import support
-from test.support import run_unittest
 
 # http://bugs.python.org/issue4373
 # Don't load the xx module more than once.
 ALREADY_TESTED = False
 
-def _get_source_filename():
-    # use installed copy if available
-    tests_f = os.path.join(os.path.dirname(__file__), 'xxmodule.c')
-    if os.path.exists(tests_f):
-        return tests_f
-    # otherwise try using copy from build directory
-    srcdir = sysconfig.get_config_var('srcdir')
-    return os.path.join(srcdir, 'Modules', 'xxmodule.c')
 
 class BuildExtTestCase(TempdirManager,
                        LoggingSilencer,
@@ -41,9 +31,6 @@
         self.tmp_dir = self.mkdtemp()
         self.sys_path = sys.path, sys.path[:]
         sys.path.append(self.tmp_dir)
-        filename = _get_source_filename()
-        if os.path.exists(filename):
-            shutil.copy(filename, self.tmp_dir)
         if sys.version > "2.6":
             import site
             self.old_user_base = site.USER_BASE
@@ -72,9 +59,8 @@
 
     def test_build_ext(self):
         global ALREADY_TESTED
+        copy_xxmodule_c(self.tmp_dir)
         xx_c = os.path.join(self.tmp_dir, 'xxmodule.c')
-        if not os.path.exists(xx_c):
-            return
         xx_ext = Extension('xx', [xx_c])
         dist = Distribution({'name': 'xx', 'ext_modules': [xx_ext]})
         dist.package_dir = self.tmp_dir
@@ -518,13 +504,7 @@
 
 
 def test_suite():
-    src = _get_source_filename()
-    if not os.path.exists(src):
-        if support.verbose:
-            print('test_build_ext: Cannot find source code (test'
-                  ' must run in python build dir)')
-        return unittest.TestSuite()
-    else: return unittest.makeSuite(BuildExtTestCase)
+    return unittest.makeSuite(BuildExtTestCase)
 
 if __name__ == '__main__':
     support.run_unittest(test_suite())

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:41 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:41 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Factor_out_the_?=
 =?utf8?q?build=5Fext_fixup_for_shared_Python_builds=2E?=
Message-ID: <E1R06Ph-0002aU-6S@dinsdale.python.org>

http://hg.python.org/cpython/rev/d9692e47f405
changeset:   72254:d9692e47f405
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Aug 21 17:02:07 2011 +0200
summary:
  Factor out the build_ext fixup for shared Python builds.

I need this to fix the failing test_install.

files:
  Lib/distutils/tests/support.py        |  27 +++++++++++++
  Lib/distutils/tests/test_build_ext.py |  31 ++------------
  2 files changed, 33 insertions(+), 25 deletions(-)


diff --git a/Lib/distutils/tests/support.py b/Lib/distutils/tests/support.py
--- a/Lib/distutils/tests/support.py
+++ b/Lib/distutils/tests/support.py
@@ -1,5 +1,6 @@
 """Support code for distutils test cases."""
 import os
+import sys
 import shutil
 import tempfile
 import unittest
@@ -165,3 +166,29 @@
     for path in candidates:
         if os.path.exists(path):
             return path
+
+
+def fixup_build_ext(cmd):
+    """Function needed to make build_ext tests pass on shared builds.
+
+    When Python was build with --enable-shared, -L. is not good enough to find
+    the libpython<blah>.so.  This is because regrtest runs it under a tempdir,
+    not in the top level where the .so lives.  By the time we've gotten here,
+    Python's already been chdir'd to the tempdir.  This function work arounds
+    that.  Example use:
+
+        cmd = build_ext(dist)
+        support.fixup_build_ext(cmd)
+        cmd.ensure_finalized()
+    """
+    # To further add to the fun, we can't just add library_dirs to the
+    # Extension() instance because that doesn't get plumbed through to the
+    # final compiler command.
+    if (sysconfig.get_config_var('Py_ENABLE_SHARED') and
+        not sys.platform.startswith('win')):
+        runshared = sysconfig.get_config_var('RUNSHARED')
+        if runshared is None:
+            cmd.library_dirs = ['.']
+        else:
+            name, equals, value = runshared.partition('=')
+            cmd.library_dirs = value.split(os.pathsep)
diff --git a/Lib/distutils/tests/test_build_ext.py b/Lib/distutils/tests/test_build_ext.py
--- a/Lib/distutils/tests/test_build_ext.py
+++ b/Lib/distutils/tests/test_build_ext.py
@@ -7,7 +7,7 @@
 from distutils.command.build_ext import build_ext
 from distutils import sysconfig
 from distutils.tests.support import (TempdirManager, LoggingSilencer,
-                                     copy_xxmodule_c)
+                                     copy_xxmodule_c, fixup_build_ext)
 from distutils.extension import Extension
 from distutils.errors import (
     CompileError, DistutilsPlatformError, DistutilsSetupError,
@@ -38,25 +38,6 @@
             from distutils.command import build_ext
             build_ext.USER_BASE = site.USER_BASE
 
-    def _fixup_command(self, cmd):
-        # When Python was build with --enable-shared, -L. is not good enough
-        # to find the libpython<blah>.so.  This is because regrtest runs it
-        # under a tempdir, not in the top level where the .so lives.  By the
-        # time we've gotten here, Python's already been chdir'd to the
-        # tempdir.
-        #
-        # To further add to the fun, we can't just add library_dirs to the
-        # Extension() instance because that doesn't get plumbed through to the
-        # final compiler command.
-        if (sysconfig.get_config_var('Py_ENABLE_SHARED') and
-            not sys.platform.startswith('win')):
-            runshared = sysconfig.get_config_var('RUNSHARED')
-            if runshared is None:
-                cmd.library_dirs = ['.']
-            else:
-                name, equals, value = runshared.partition('=')
-                cmd.library_dirs = value.split(os.pathsep)
-
     def test_build_ext(self):
         global ALREADY_TESTED
         copy_xxmodule_c(self.tmp_dir)
@@ -65,7 +46,7 @@
         dist = Distribution({'name': 'xx', 'ext_modules': [xx_ext]})
         dist.package_dir = self.tmp_dir
         cmd = build_ext(dist)
-        self._fixup_command(cmd)
+        fixup_build_ext(cmd)
         if os.name == "nt":
             # On Windows, we must build a debug version iff running
             # a debug build of Python
@@ -162,9 +143,9 @@
 
         # see if include_dirs and library_dirs
         # were set
-        self.assertTrue(lib in cmd.library_dirs)
-        self.assertTrue(lib in cmd.rpath)
-        self.assertTrue(incl in cmd.include_dirs)
+        self.assertIn(lib, cmd.library_dirs)
+        self.assertIn(lib, cmd.rpath)
+        self.assertIn(incl, cmd.include_dirs)
 
     def test_optional_extension(self):
 
@@ -320,7 +301,7 @@
         dist = Distribution({'name': 'xx',
                              'ext_modules': [ext]})
         cmd = build_ext(dist)
-        self._fixup_command(cmd)
+        fixup_build_ext(cmd)
         cmd.ensure_finalized()
         self.assertEqual(len(cmd.get_outputs()), 1)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:41 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:41 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Rework_test=5Fr?=
 =?utf8?q?ecord_a_bit_to_make_the_test_more_exact?=
Message-ID: <E1R06Ph-0002aY-P1@dinsdale.python.org>

http://hg.python.org/cpython/rev/09873d803c1e
changeset:   72255:09873d803c1e
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Aug 20 07:00:41 2011 +0200
summary:
  Rework test_record a bit to make the test more exact

files:
  Lib/distutils/tests/test_install.py |  24 +++++++++-------
  1 files changed, 13 insertions(+), 11 deletions(-)


diff --git a/Lib/distutils/tests/test_install.py b/Lib/distutils/tests/test_install.py
--- a/Lib/distutils/tests/test_install.py
+++ b/Lib/distutils/tests/test_install.py
@@ -1,7 +1,6 @@
 """Tests for distutils.command.install."""
 
 import os
-import os.path
 import sys
 import unittest
 import site
@@ -167,33 +166,36 @@
         self.assertRaises(DistutilsOptionError, cmd.finalize_options)
 
     def test_record(self):
+        install_dir = self.mkdtemp()
+        project_dir, dist = self.create_dist(scripts=['hello'])
+        self.addCleanup(os.chdir, os.getcwd())
+        os.chdir(project_dir)
+        self.write_file('hello', "print('o hai')")
 
-        install_dir = self.mkdtemp()
-        pkgdir, dist = self.create_dist()
-
-        dist = Distribution()
         cmd = install(dist)
         dist.command_obj['install'] = cmd
         cmd.root = install_dir
-        cmd.record = os.path.join(pkgdir, 'RECORD')
+        cmd.record = os.path.join(project_dir, 'RECORD')
         cmd.ensure_finalized()
-
         cmd.run()
 
-        # let's check the RECORD file was created with one
-        # line (the egg info file)
         f = open(cmd.record)
         try:
-            self.assertEqual(len(f.readlines()), 1)
+            content = f.read()
         finally:
             f.close()
 
+        found = [os.path.basename(line) for line in content.splitlines()]
+        expected = ['hello',
+                    'UNKNOWN-0.0.0-py%s.%s.egg-info' % sys.version_info[:2]]
+        self.assertEqual(found, expected)
+
     def test_debug_mode(self):
         # this covers the code called when DEBUG is set
         old_logs_len = len(self.logs)
         install_module.DEBUG = True
         try:
-            with captured_stdout() as stdout:
+            with captured_stdout():
                 self.test_record()
         finally:
             install_module.DEBUG = False

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:42 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:42 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Add_a_test_for_?=
 =?utf8?q?extension_modules_in_the_distutils_record_file=2E?=
Message-ID: <E1R06Pi-0002ac-Ab@dinsdale.python.org>

http://hg.python.org/cpython/rev/568fc012543a
changeset:   72256:568fc012543a
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Aug 20 07:08:51 2011 +0200
summary:
  Add a test for extension modules in the distutils record file.

I made a note a month ago that install --record wrote incorrect entries
for extension modules (I think the problem was that the first character
of the file was stripped), so I?m now adding a test to try to reproduce
that in the current versions.

files:
  Lib/distutils/tests/test_install.py |  33 +++++++++++++++++
  1 files changed, 33 insertions(+), 0 deletions(-)


diff --git a/Lib/distutils/tests/test_install.py b/Lib/distutils/tests/test_install.py
--- a/Lib/distutils/tests/test_install.py
+++ b/Lib/distutils/tests/test_install.py
@@ -7,11 +7,14 @@
 
 from test.support import captured_stdout, run_unittest
 
+from distutils import sysconfig
 from distutils.command.install import install
 from distutils.command import install as install_module
+from distutils.command.build_ext import build_ext
 from distutils.command.install import INSTALL_SCHEMES
 from distutils.core import Distribution
 from distutils.errors import DistutilsOptionError
+from distutils.extension import Extension
 
 from distutils.tests import support
 
@@ -190,6 +193,36 @@
                     'UNKNOWN-0.0.0-py%s.%s.egg-info' % sys.version_info[:2]]
         self.assertEqual(found, expected)
 
+    def test_record_extensions(self):
+        install_dir = self.mkdtemp()
+        project_dir, dist = self.create_dist(ext_modules=[
+            Extension('xx', ['xxmodule.c'])])
+        self.addCleanup(os.chdir, os.getcwd())
+        os.chdir(project_dir)
+        support.copy_xxmodule_c(project_dir)
+
+        buildcmd = build_ext(dist)
+        buildcmd.ensure_finalized()
+        buildcmd.run()
+
+        cmd = install(dist)
+        dist.command_obj['install'] = cmd
+        cmd.root = install_dir
+        cmd.record = os.path.join(project_dir, 'RECORD')
+        cmd.ensure_finalized()
+        cmd.run()
+
+        f = open(cmd.record)
+        try:
+            content = f.read()
+        finally:
+            f.close()
+
+        found = [os.path.basename(line) for line in content.splitlines()]
+        expected = ['xx%s' % sysconfig.get_config_var('SO'),
+                    'UNKNOWN-0.0.0-py%s.%s.egg-info' % sys.version_info[:2]]
+        self.assertEqual(found, expected)
+
     def test_debug_mode(self):
         # this covers the code called when DEBUG is set
         old_logs_len = len(self.logs)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:42 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:42 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_distutils_t?=
 =?utf8?q?est=5Finstall_for_shared_CPython_builds?=
Message-ID: <E1R06Pi-0002ag-SM@dinsdale.python.org>

http://hg.python.org/cpython/rev/8c940df0c578
changeset:   72257:8c940df0c578
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Aug 21 17:03:19 2011 +0200
summary:
  Fix distutils test_install for shared CPython builds

files:
  Lib/distutils/tests/test_install.py |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Lib/distutils/tests/test_install.py b/Lib/distutils/tests/test_install.py
--- a/Lib/distutils/tests/test_install.py
+++ b/Lib/distutils/tests/test_install.py
@@ -202,6 +202,7 @@
         support.copy_xxmodule_c(project_dir)
 
         buildcmd = build_ext(dist)
+        support.fixup_build_ext(buildcmd)
         buildcmd.ensure_finalized()
         buildcmd.run()
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:43 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:43 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyNjc4?=
 =?utf8?q?=3A_Fix_distutils_sdist_test_on_Windows=2E?=
Message-ID: <E1R06Pj-0002ak-Do@dinsdale.python.org>

http://hg.python.org/cpython/rev/6fe19f421a16
changeset:   72258:6fe19f421a16
branch:      3.2
user:        Nadeem Vawda <nadeem.vawda at gmail.com>
date:        Sun Aug 21 22:35:41 2011 +0200
summary:
  Issue #12678: Fix distutils sdist test on Windows.

Patch by Jeremy Kloth.

files:
  Lib/distutils/tests/test_sdist.py |  1 +
  Misc/ACKS                         |  1 +
  2 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -365,6 +365,7 @@
     def test_manual_manifest(self):
         # check that a MANIFEST without a marker is left alone
         dist, cmd = self.get_cmd()
+        cmd.formats = ['gztar']
         cmd.ensure_finalized()
         self.write_file((self.tmp_dir, cmd.manifest), 'README.manual')
         self.write_file((self.tmp_dir, 'README.manual'),
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -484,6 +484,7 @@
 Bastian Kleineidam
 Bob Kline
 Matthias Klose
+Jeremy Kloth
 Kim Knapp
 Lenny Kneler
 Pat Knight

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:44 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:44 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_distutils_t?=
 =?utf8?q?ests_on_Windows_=28=2312678=29=2E?=
Message-ID: <E1R06Pk-0002ao-31@dinsdale.python.org>

http://hg.python.org/cpython/rev/b173cfbf49a0
changeset:   72259:b173cfbf49a0
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Wed Aug 24 01:29:10 2011 +0200
summary:
  Fix distutils tests on Windows (#12678).

- First, support.fixup_build_ext (already used to set proper
  library_dirs value under Unix shared builds) gains the ability to
  correctly set the debug attribute under Windows debug builds.

- Second, the filename for the extension module gets a _d suffix under
  debug builds.

- Third, the test code properly puts our customized build_ext object
  into an internal dictionary to make sure that the install command will
  later use our object instead of re-creating one.  That?s the downside
  of using low-level APIs in our test code: we have to manually push
  knobs and turn handles that would otherwise be handled behind the
  scenes.

Thanks to Nadeem for the testing.

files:
  Lib/distutils/tests/support.py        |  28 +++++++++-----
  Lib/distutils/tests/test_build_ext.py |   7 ---
  Lib/distutils/tests/test_install.py   |  18 ++++++--
  3 files changed, 30 insertions(+), 23 deletions(-)


diff --git a/Lib/distutils/tests/support.py b/Lib/distutils/tests/support.py
--- a/Lib/distutils/tests/support.py
+++ b/Lib/distutils/tests/support.py
@@ -169,23 +169,29 @@
 
 
 def fixup_build_ext(cmd):
-    """Function needed to make build_ext tests pass on shared builds.
+    """Function needed to make build_ext tests pass.
 
-    When Python was build with --enable-shared, -L. is not good enough to find
-    the libpython<blah>.so.  This is because regrtest runs it under a tempdir,
-    not in the top level where the .so lives.  By the time we've gotten here,
-    Python's already been chdir'd to the tempdir.  This function work arounds
-    that.  Example use:
+    When Python was build with --enable-shared on Unix, -L. is not good
+    enough to find the libpython<blah>.so.  This is because regrtest runs
+    it under a tempdir, not in the top level where the .so lives.  By the
+    time we've gotten here, Python's already been chdir'd to the tempdir.
+
+    When Python was built with in debug mode on Windows, build_ext commands
+    need their debug attribute set, and it is not done automatically for
+    some reason.
+
+    This function handles both of these things.  Example use:
 
         cmd = build_ext(dist)
         support.fixup_build_ext(cmd)
         cmd.ensure_finalized()
     """
-    # To further add to the fun, we can't just add library_dirs to the
-    # Extension() instance because that doesn't get plumbed through to the
-    # final compiler command.
-    if (sysconfig.get_config_var('Py_ENABLE_SHARED') and
-        not sys.platform.startswith('win')):
+    if os.name == 'nt':
+        cmd.debug = sys.executable.endswith('_d.exe')
+    elif sysconfig.get_config_var('Py_ENABLE_SHARED'):
+        # To further add to the shared builds fun on Unix, we can't just add
+        # library_dirs to the Extension() instance because that doesn't get
+        # plumbed through to the final compiler command.
         runshared = sysconfig.get_config_var('RUNSHARED')
         if runshared is None:
             cmd.library_dirs = ['.']
diff --git a/Lib/distutils/tests/test_build_ext.py b/Lib/distutils/tests/test_build_ext.py
--- a/Lib/distutils/tests/test_build_ext.py
+++ b/Lib/distutils/tests/test_build_ext.py
@@ -47,10 +47,6 @@
         dist.package_dir = self.tmp_dir
         cmd = build_ext(dist)
         fixup_build_ext(cmd)
-        if os.name == "nt":
-            # On Windows, we must build a debug version iff running
-            # a debug build of Python
-            cmd.debug = sys.executable.endswith("_d.exe")
         cmd.build_lib = self.tmp_dir
         cmd.build_temp = self.tmp_dir
 
@@ -305,9 +301,6 @@
         cmd.ensure_finalized()
         self.assertEqual(len(cmd.get_outputs()), 1)
 
-        if os.name == "nt":
-            cmd.debug = sys.executable.endswith("_d.exe")
-
         cmd.build_lib = os.path.join(self.tmp_dir, 'build')
         cmd.build_temp = os.path.join(self.tmp_dir, 'tempt')
 
diff --git a/Lib/distutils/tests/test_install.py b/Lib/distutils/tests/test_install.py
--- a/Lib/distutils/tests/test_install.py
+++ b/Lib/distutils/tests/test_install.py
@@ -18,6 +18,14 @@
 
 from distutils.tests import support
 
+
+def _make_ext_name(modname):
+    if os.name == 'nt':
+        if sys.executable.endswith('_d.exe'):
+            modname += '_d'
+    return modname + sysconfig.get_config_var('SO')
+
+
 class InstallTestCase(support.TempdirManager,
                       support.EnvironGuard,
                       support.LoggingSilencer,
@@ -201,13 +209,13 @@
         os.chdir(project_dir)
         support.copy_xxmodule_c(project_dir)
 
-        buildcmd = build_ext(dist)
-        support.fixup_build_ext(buildcmd)
-        buildcmd.ensure_finalized()
-        buildcmd.run()
+        buildextcmd = build_ext(dist)
+        support.fixup_build_ext(buildextcmd)
+        buildextcmd.ensure_finalized()
 
         cmd = install(dist)
         dist.command_obj['install'] = cmd
+        dist.command_obj['build_ext'] = buildextcmd
         cmd.root = install_dir
         cmd.record = os.path.join(project_dir, 'RECORD')
         cmd.ensure_finalized()
@@ -220,7 +228,7 @@
             f.close()
 
         found = [os.path.basename(line) for line in content.splitlines()]
-        expected = ['xx%s' % sysconfig.get_config_var('SO'),
+        expected = [_make_ext_name('xx'),
                     'UNKNOWN-0.0.0-py%s.%s.egg-info' % sys.version_info[:2]]
         self.assertEqual(found, expected)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:44 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:44 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Make_bdist=5F*_?=
 =?utf8?q?commands_respect_--skip-build_passed_to_bdist_=28=2310946=29?=
Message-ID: <E1R06Pk-0002as-Lu@dinsdale.python.org>

http://hg.python.org/cpython/rev/a78629c62ee8
changeset:   72260:a78629c62ee8
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Aug 29 21:48:39 2011 +0200
summary:
  Make bdist_* commands respect --skip-build passed to bdist (#10946)

files:
  Lib/distutils/command/bdist_dumb.py    |   5 +-
  Lib/distutils/command/bdist_msi.py     |   6 +-
  Lib/distutils/command/bdist_wininst.py |   6 +-
  Lib/distutils/tests/test_bdist.py      |  48 +++++++------
  Misc/NEWS                              |   3 +
  5 files changed, 43 insertions(+), 25 deletions(-)


diff --git a/Lib/distutils/command/bdist_dumb.py b/Lib/distutils/command/bdist_dumb.py
--- a/Lib/distutils/command/bdist_dumb.py
+++ b/Lib/distutils/command/bdist_dumb.py
@@ -47,7 +47,7 @@
         self.format = None
         self.keep_temp = 0
         self.dist_dir = None
-        self.skip_build = 0
+        self.skip_build = None
         self.relative = 0
 
     def finalize_options(self):
@@ -65,7 +65,8 @@
 
         self.set_undefined_options('bdist',
                                    ('dist_dir', 'dist_dir'),
-                                   ('plat_name', 'plat_name'))
+                                   ('plat_name', 'plat_name'),
+                                   ('skip_build', 'skip_build'))
 
     def run(self):
         if not self.skip_build:
diff --git a/Lib/distutils/command/bdist_msi.py b/Lib/distutils/command/bdist_msi.py
--- a/Lib/distutils/command/bdist_msi.py
+++ b/Lib/distutils/command/bdist_msi.py
@@ -130,18 +130,22 @@
         self.no_target_optimize = 0
         self.target_version = None
         self.dist_dir = None
-        self.skip_build = 0
+        self.skip_build = None
         self.install_script = None
         self.pre_install_script = None
         self.versions = None
 
     def finalize_options(self):
+        self.set_undefined_options('bdist', ('skip_build', 'skip_build'))
+
         if self.bdist_dir is None:
             bdist_base = self.get_finalized_command('bdist').bdist_base
             self.bdist_dir = os.path.join(bdist_base, 'msi')
+
         short_version = get_python_version()
         if (not self.target_version) and self.distribution.has_ext_modules():
             self.target_version = short_version
+
         if self.target_version:
             self.versions = [self.target_version]
             if not self.skip_build and self.distribution.has_ext_modules()\
diff --git a/Lib/distutils/command/bdist_wininst.py b/Lib/distutils/command/bdist_wininst.py
--- a/Lib/distutils/command/bdist_wininst.py
+++ b/Lib/distutils/command/bdist_wininst.py
@@ -65,13 +65,15 @@
         self.dist_dir = None
         self.bitmap = None
         self.title = None
-        self.skip_build = 0
+        self.skip_build = None
         self.install_script = None
         self.pre_install_script = None
         self.user_access_control = None
 
 
     def finalize_options(self):
+        self.set_undefined_options('bdist', ('skip_build', 'skip_build'))
+
         if self.bdist_dir is None:
             if self.skip_build and self.plat_name:
                 # If build is skipped and plat_name is overridden, bdist will
@@ -81,8 +83,10 @@
                 # next the command will be initialized using that name
             bdist_base = self.get_finalized_command('bdist').bdist_base
             self.bdist_dir = os.path.join(bdist_base, 'wininst')
+
         if not self.target_version:
             self.target_version = ""
+
         if not self.skip_build and self.distribution.has_ext_modules():
             short_version = get_python_version()
             if self.target_version and self.target_version != short_version:
diff --git a/Lib/distutils/tests/test_bdist.py b/Lib/distutils/tests/test_bdist.py
--- a/Lib/distutils/tests/test_bdist.py
+++ b/Lib/distutils/tests/test_bdist.py
@@ -1,41 +1,47 @@
 """Tests for distutils.command.bdist."""
+import os
 import unittest
-import sys
-import os
-import tempfile
-import shutil
 from test.support import run_unittest
 
-from distutils.core import Distribution
 from distutils.command.bdist import bdist
 from distutils.tests import support
-from distutils.spawn import find_executable
-from distutils import spawn
-from distutils.errors import DistutilsExecError
+
 
 class BuildTestCase(support.TempdirManager,
                     unittest.TestCase):
 
     def test_formats(self):
-
         # let's create a command and make sure
-        # we can fix the format
-        pkg_pth, dist = self.create_dist()
+        # we can set the format
+        dist = self.create_dist()[1]
         cmd = bdist(dist)
         cmd.formats = ['msi']
         cmd.ensure_finalized()
         self.assertEqual(cmd.formats, ['msi'])
 
-        # what format bdist offers ?
-        # XXX an explicit list in bdist is
-        # not the best way to  bdist_* commands
-        # we should add a registry
-        formats = ['rpm', 'zip', 'gztar', 'bztar', 'ztar',
-                   'tar', 'wininst', 'msi']
-        formats.sort()
-        founded = list(cmd.format_command.keys())
-        founded.sort()
-        self.assertEqual(founded, formats)
+        # what formats does bdist offer?
+        formats = ['bztar', 'gztar', 'msi', 'rpm', 'tar',
+                   'wininst', 'zip', 'ztar']
+        found = sorted(cmd.format_command)
+        self.assertEqual(found, formats)
+
+    def test_skip_build(self):
+        # bug #10946: bdist --skip-build should trickle down to subcommands
+        dist = self.create_dist()[1]
+        cmd = bdist(dist)
+        cmd.skip_build = 1
+        cmd.ensure_finalized()
+        dist.command_obj['bdist'] = cmd
+
+        names = ['bdist_dumb', 'bdist_wininst']  # bdist_rpm does not support --skip-build
+        if os.name == 'nt':
+            names.append('bdist_msi')
+
+        for name in names:
+            subcmd = cmd.get_finalized_command(name)
+            self.assertTrue(subcmd.skip_build,
+                            '%s should take --skip-build from bdist' % name)
+
 
 def test_suite():
     return unittest.makeSuite(BuildTestCase)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -19,6 +19,9 @@
 Library
 -------
 
+- Issue #10946: The distutils commands bdist_dumb, bdist_wininst and bdist_msi
+  now respect a --skip-build option given to bdist.
+
 - Issue #12839: Fix crash in zlib module due to version mismatch.
   Fix by Richard M. Tew.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:45 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:45 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Regenerate_pydo?=
	=?utf8?q?c_topics=2E?=
Message-ID: <E1R06Pl-0002aw-9u@dinsdale.python.org>

http://hg.python.org/cpython/rev/e7a5d96efc67
changeset:   72261:e7a5d96efc67
branch:      3.2
user:        Georg Brandl <georg at python.org>
date:        Sat Sep 03 10:37:09 2011 +0200
summary:
  Regenerate pydoc topics.

files:
  Lib/pydoc_data/topics.py |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py
--- a/Lib/pydoc_data/topics.py
+++ b/Lib/pydoc_data/topics.py
@@ -1,4 +1,4 @@
-# Autogenerated by Sphinx on Sat Aug 13 11:28:40 2011
+# Autogenerated by Sphinx on Sat Sep  3 10:35:42 2011
 topics = {'assert': '\nThe ``assert`` statement\n************************\n\nAssert statements are a convenient way to insert debugging assertions\ninto a program:\n\n   assert_stmt ::= "assert" expression ["," expression]\n\nThe simple form, ``assert expression``, is equivalent to\n\n   if __debug__:\n      if not expression: raise AssertionError\n\nThe extended form, ``assert expression1, expression2``, is equivalent\nto\n\n   if __debug__:\n      if not expression1: raise AssertionError(expression2)\n\nThese equivalences assume that ``__debug__`` and ``AssertionError``\nrefer to the built-in variables with those names.  In the current\nimplementation, the built-in variable ``__debug__`` is ``True`` under\nnormal circumstances, ``False`` when optimization is requested\n(command line option -O).  The current code generator emits no code\nfor an assert statement when optimization is requested at compile\ntime.  Note that it is unnecessary to include the source code for the\nexpression that failed in the error message; it will be displayed as\npart of the stack trace.\n\nAssignments to ``__debug__`` are illegal.  The value for the built-in\nvariable is determined when the interpreter starts.\n',
  'assignment': '\nAssignment statements\n*********************\n\nAssignment statements are used to (re)bind names to values and to\nmodify attributes or items of mutable objects:\n\n   assignment_stmt ::= (target_list "=")+ (expression_list | yield_expression)\n   target_list     ::= target ("," target)* [","]\n   target          ::= identifier\n              | "(" target_list ")"\n              | "[" target_list "]"\n              | attributeref\n              | subscription\n              | slicing\n              | "*" target\n\n(See section *Primaries* for the syntax definitions for the last three\nsymbols.)\n\nAn assignment statement evaluates the expression list (remember that\nthis can be a single expression or a comma-separated list, the latter\nyielding a tuple) and assigns the single resulting object to each of\nthe target lists, from left to right.\n\nAssignment is defined recursively depending on the form of the target\n(list). When a target is part of a mutable object (an attribute\nreference, subscription or slicing), the mutable object must\nultimately perform the assignment and decide about its validity, and\nmay raise an exception if the assignment is unacceptable.  The rules\nobserved by various types and the exceptions raised are given with the\ndefinition of the object types (see section *The standard type\nhierarchy*).\n\nAssignment of an object to a target list, optionally enclosed in\nparentheses or square brackets, is recursively defined as follows.\n\n* If the target list is a single target: The object is assigned to\n  that target.\n\n* If the target list is a comma-separated list of targets: The object\n  must be an iterable with the same number of items as there are\n  targets in the target list, and the items are assigned, from left to\n  right, to the corresponding targets.\n\n  * If the target list contains one target prefixed with an asterisk,\n    called a "starred" target: The object must be a sequence with at\n    least as many items as there are targets in the target list, minus\n    one.  The first items of the sequence are assigned, from left to\n    right, to the targets before the starred target.  The final items\n    of the sequence are assigned to the targets after the starred\n    target.  A list of the remaining items in the sequence is then\n    assigned to the starred target (the list can be empty).\n\n  * Else: The object must be a sequence with the same number of items\n    as there are targets in the target list, and the items are\n    assigned, from left to right, to the corresponding targets.\n\nAssignment of an object to a single target is recursively defined as\nfollows.\n\n* If the target is an identifier (name):\n\n  * If the name does not occur in a ``global`` or ``nonlocal``\n    statement in the current code block: the name is bound to the\n    object in the current local namespace.\n\n  * Otherwise: the name is bound to the object in the global namespace\n    or the outer namespace determined by ``nonlocal``, respectively.\n\n  The name is rebound if it was already bound.  This may cause the\n  reference count for the object previously bound to the name to reach\n  zero, causing the object to be deallocated and its destructor (if it\n  has one) to be called.\n\n* If the target is a target list enclosed in parentheses or in square\n  brackets: The object must be an iterable with the same number of\n  items as there are targets in the target list, and its items are\n  assigned, from left to right, to the corresponding targets.\n\n* If the target is an attribute reference: The primary expression in\n  the reference is evaluated.  It should yield an object with\n  assignable attributes; if this is not the case, ``TypeError`` is\n  raised.  That object is then asked to assign the assigned object to\n  the given attribute; if it cannot perform the assignment, it raises\n  an exception (usually but not necessarily ``AttributeError``).\n\n  Note: If the object is a class instance and the attribute reference\n  occurs on both sides of the assignment operator, the RHS expression,\n  ``a.x`` can access either an instance attribute or (if no instance\n  attribute exists) a class attribute.  The LHS target ``a.x`` is\n  always set as an instance attribute, creating it if necessary.\n  Thus, the two occurrences of ``a.x`` do not necessarily refer to the\n  same attribute: if the RHS expression refers to a class attribute,\n  the LHS creates a new instance attribute as the target of the\n  assignment:\n\n     class Cls:\n         x = 3             # class variable\n     inst = Cls()\n     inst.x = inst.x + 1   # writes inst.x as 4 leaving Cls.x as 3\n\n  This description does not necessarily apply to descriptor\n  attributes, such as properties created with ``property()``.\n\n* If the target is a subscription: The primary expression in the\n  reference is evaluated.  It should yield either a mutable sequence\n  object (such as a list) or a mapping object (such as a dictionary).\n  Next, the subscript expression is evaluated.\n\n  If the primary is a mutable sequence object (such as a list), the\n  subscript must yield an integer.  If it is negative, the sequence\'s\n  length is added to it.  The resulting value must be a nonnegative\n  integer less than the sequence\'s length, and the sequence is asked\n  to assign the assigned object to its item with that index.  If the\n  index is out of range, ``IndexError`` is raised (assignment to a\n  subscripted sequence cannot add new items to a list).\n\n  If the primary is a mapping object (such as a dictionary), the\n  subscript must have a type compatible with the mapping\'s key type,\n  and the mapping is then asked to create a key/datum pair which maps\n  the subscript to the assigned object.  This can either replace an\n  existing key/value pair with the same key value, or insert a new\n  key/value pair (if no key with the same value existed).\n\n  For user-defined objects, the ``__setitem__()`` method is called\n  with appropriate arguments.\n\n* If the target is a slicing: The primary expression in the reference\n  is evaluated.  It should yield a mutable sequence object (such as a\n  list).  The assigned object should be a sequence object of the same\n  type.  Next, the lower and upper bound expressions are evaluated,\n  insofar they are present; defaults are zero and the sequence\'s\n  length.  The bounds should evaluate to integers. If either bound is\n  negative, the sequence\'s length is added to it.  The resulting\n  bounds are clipped to lie between zero and the sequence\'s length,\n  inclusive.  Finally, the sequence object is asked to replace the\n  slice with the items of the assigned sequence.  The length of the\n  slice may be different from the length of the assigned sequence,\n  thus changing the length of the target sequence, if the object\n  allows it.\n\n**CPython implementation detail:** In the current implementation, the\nsyntax for targets is taken to be the same as for expressions, and\ninvalid syntax is rejected during the code generation phase, causing\nless detailed error messages.\n\nWARNING: Although the definition of assignment implies that overlaps\nbetween the left-hand side and the right-hand side are \'safe\' (for\nexample ``a, b = b, a`` swaps two variables), overlaps *within* the\ncollection of assigned-to variables are not safe!  For instance, the\nfollowing program prints ``[0, 2]``:\n\n   x = [0, 1]\n   i = 0\n   i, x[i] = 1, 2\n   print(x)\n\nSee also:\n\n   **PEP 3132** - Extended Iterable Unpacking\n      The specification for the ``*target`` feature.\n\n\nAugmented assignment statements\n===============================\n\nAugmented assignment is the combination, in a single statement, of a\nbinary operation and an assignment statement:\n\n   augmented_assignment_stmt ::= augtarget augop (expression_list | yield_expression)\n   augtarget                 ::= identifier | attributeref | subscription | slicing\n   augop                     ::= "+=" | "-=" | "*=" | "/=" | "//=" | "%=" | "**="\n             | ">>=" | "<<=" | "&=" | "^=" | "|="\n\n(See section *Primaries* for the syntax definitions for the last three\nsymbols.)\n\nAn augmented assignment evaluates the target (which, unlike normal\nassignment statements, cannot be an unpacking) and the expression\nlist, performs the binary operation specific to the type of assignment\non the two operands, and assigns the result to the original target.\nThe target is only evaluated once.\n\nAn augmented assignment expression like ``x += 1`` can be rewritten as\n``x = x + 1`` to achieve a similar, but not exactly equal effect. In\nthe augmented version, ``x`` is only evaluated once. Also, when\npossible, the actual operation is performed *in-place*, meaning that\nrather than creating a new object and assigning that to the target,\nthe old object is modified instead.\n\nWith the exception of assigning to tuples and multiple targets in a\nsingle statement, the assignment done by augmented assignment\nstatements is handled the same way as normal assignments. Similarly,\nwith the exception of the possible *in-place* behavior, the binary\noperation performed by augmented assignment is the same as the normal\nbinary operations.\n\nFor targets which are attribute references, the same *caveat about\nclass and instance attributes* applies as for regular assignments.\n',
  'atom-identifiers': '\nIdentifiers (Names)\n*******************\n\nAn identifier occurring as an atom is a name.  See section\n*Identifiers and keywords* for lexical definition and section *Naming\nand binding* for documentation of naming and binding.\n\nWhen the name is bound to an object, evaluation of the atom yields\nthat object. When a name is not bound, an attempt to evaluate it\nraises a ``NameError`` exception.\n\n**Private name mangling:** When an identifier that textually occurs in\na class definition begins with two or more underscore characters and\ndoes not end in two or more underscores, it is considered a *private\nname* of that class. Private names are transformed to a longer form\nbefore code is generated for them.  The transformation inserts the\nclass name in front of the name, with leading underscores removed, and\na single underscore inserted in front of the class name.  For example,\nthe identifier ``__spam`` occurring in a class named ``Ham`` will be\ntransformed to ``_Ham__spam``.  This transformation is independent of\nthe syntactical context in which the identifier is used.  If the\ntransformed name is extremely long (longer than 255 characters),\nimplementation defined truncation may happen.  If the class name\nconsists only of underscores, no transformation is done.\n',
@@ -33,7 +33,7 @@
  'exprlists': '\nExpression lists\n****************\n\n   expression_list ::= expression ( "," expression )* [","]\n\nAn expression list containing at least one comma yields a tuple.  The\nlength of the tuple is the number of expressions in the list.  The\nexpressions are evaluated from left to right.\n\nThe trailing comma is required only to create a single tuple (a.k.a. a\n*singleton*); it is optional in all other cases.  A single expression\nwithout a trailing comma doesn\'t create a tuple, but rather yields the\nvalue of that expression. (To create an empty tuple, use an empty pair\nof parentheses: ``()``.)\n',
  'floating': '\nFloating point literals\n***********************\n\nFloating point literals are described by the following lexical\ndefinitions:\n\n   floatnumber   ::= pointfloat | exponentfloat\n   pointfloat    ::= [intpart] fraction | intpart "."\n   exponentfloat ::= (intpart | pointfloat) exponent\n   intpart       ::= digit+\n   fraction      ::= "." digit+\n   exponent      ::= ("e" | "E") ["+" | "-"] digit+\n\nNote that the integer and exponent parts are always interpreted using\nradix 10. For example, ``077e010`` is legal, and denotes the same\nnumber as ``77e10``. The allowed range of floating point literals is\nimplementation-dependent. Some examples of floating point literals:\n\n   3.14    10.    .001    1e100    3.14e-10    0e0\n\nNote that numeric literals do not include a sign; a phrase like ``-1``\nis actually an expression composed of the unary operator ``-`` and the\nliteral ``1``.\n',
  'for': '\nThe ``for`` statement\n*********************\n\nThe ``for`` statement is used to iterate over the elements of a\nsequence (such as a string, tuple or list) or other iterable object:\n\n   for_stmt ::= "for" target_list "in" expression_list ":" suite\n                ["else" ":" suite]\n\nThe expression list is evaluated once; it should yield an iterable\nobject.  An iterator is created for the result of the\n``expression_list``.  The suite is then executed once for each item\nprovided by the iterator, in the order of ascending indices.  Each\nitem in turn is assigned to the target list using the standard rules\nfor assignments (see *Assignment statements*), and then the suite is\nexecuted.  When the items are exhausted (which is immediately when the\nsequence is empty or an iterator raises a ``StopIteration``\nexception), the suite in the ``else`` clause, if present, is executed,\nand the loop terminates.\n\nA ``break`` statement executed in the first suite terminates the loop\nwithout executing the ``else`` clause\'s suite.  A ``continue``\nstatement executed in the first suite skips the rest of the suite and\ncontinues with the next item, or with the ``else`` clause if there was\nno next item.\n\nThe suite may assign to the variable(s) in the target list; this does\nnot affect the next item assigned to it.\n\nNames in the target list are not deleted when the loop is finished,\nbut if the sequence is empty, it will not have been assigned to at all\nby the loop.  Hint: the built-in function ``range()`` returns an\niterator of integers suitable to emulate the effect of Pascal\'s ``for\ni := a to b do``; e.g., ``list(range(3))`` returns the list ``[0, 1,\n2]``.\n\nNote: There is a subtlety when the sequence is being modified by the loop\n  (this can only occur for mutable sequences, i.e. lists).  An\n  internal counter is used to keep track of which item is used next,\n  and this is incremented on each iteration.  When this counter has\n  reached the length of the sequence the loop terminates.  This means\n  that if the suite deletes the current (or a previous) item from the\n  sequence, the next item will be skipped (since it gets the index of\n  the current item which has already been treated).  Likewise, if the\n  suite inserts an item in the sequence before the current item, the\n  current item will be treated again the next time through the loop.\n  This can lead to nasty bugs that can be avoided by making a\n  temporary copy using a slice of the whole sequence, e.g.,\n\n     for x in a[:]:\n         if x < 0: a.remove(x)\n',
- 'formatstrings': '\nFormat String Syntax\n********************\n\nThe ``str.format()`` method and the ``Formatter`` class share the same\nsyntax for format strings (although in the case of ``Formatter``,\nsubclasses can define their own format string syntax).\n\nFormat strings contain "replacement fields" surrounded by curly braces\n``{}``. Anything that is not contained in braces is considered literal\ntext, which is copied unchanged to the output.  If you need to include\na brace character in the literal text, it can be escaped by doubling:\n``{{`` and ``}}``.\n\nThe grammar for a replacement field is as follows:\n\n      replacement_field ::= "{" [field_name] ["!" conversion] [":" format_spec] "}"\n      field_name        ::= arg_name ("." attribute_name | "[" element_index "]")*\n      arg_name          ::= [identifier | integer]\n      attribute_name    ::= identifier\n      element_index     ::= integer | index_string\n      index_string      ::= <any source character except "]"> +\n      conversion        ::= "r" | "s" | "a"\n      format_spec       ::= <described in the next section>\n\nIn less formal terms, the replacement field can start with a\n*field_name* that specifies the object whose value is to be formatted\nand inserted into the output instead of the replacement field. The\n*field_name* is optionally followed by a  *conversion* field, which is\npreceded by an exclamation point ``\'!\'``, and a *format_spec*, which\nis preceded by a colon ``\':\'``.  These specify a non-default format\nfor the replacement value.\n\nSee also the *Format Specification Mini-Language* section.\n\nThe *field_name* itself begins with an *arg_name* that is either\neither a number or a keyword.  If it\'s a number, it refers to a\npositional argument, and if it\'s a keyword, it refers to a named\nkeyword argument.  If the numerical arg_names in a format string are\n0, 1, 2, ... in sequence, they can all be omitted (not just some) and\nthe numbers 0, 1, 2, ... will be automatically inserted in that order.\nThe *arg_name* can be followed by any number of index or attribute\nexpressions. An expression of the form ``\'.name\'`` selects the named\nattribute using ``getattr()``, while an expression of the form\n``\'[index]\'`` does an index lookup using ``__getitem__()``.\n\nChanged in version 3.1: The positional argument specifiers can be\nomitted, so ``\'{} {}\'`` is equivalent to ``\'{0} {1}\'``.\n\nSome simple format string examples:\n\n   "First, thou shalt count to {0}" # References first positional argument\n   "Bring me a {}"                  # Implicitly references the first positional argument\n   "From {} to {}"                  # Same as "From {0} to {1}"\n   "My quest is {name}"             # References keyword argument \'name\'\n   "Weight in tons {0.weight}"      # \'weight\' attribute of first positional arg\n   "Units destroyed: {players[0]}"  # First element of keyword argument \'players\'.\n\nThe *conversion* field causes a type coercion before formatting.\nNormally, the job of formatting a value is done by the\n``__format__()`` method of the value itself.  However, in some cases\nit is desirable to force a type to be formatted as a string,\noverriding its own definition of formatting.  By converting the value\nto a string before calling ``__format__()``, the normal formatting\nlogic is bypassed.\n\nThree conversion flags are currently supported: ``\'!s\'`` which calls\n``str()`` on the value, ``\'!r\'`` which calls ``repr()`` and ``\'!a\'``\nwhich calls ``ascii()``.\n\nSome examples:\n\n   "Harold\'s a clever {0!s}"        # Calls str() on the argument first\n   "Bring out the holy {name!r}"    # Calls repr() on the argument first\n   "More {!a}"                      # Calls ascii() on the argument first\n\nThe *format_spec* field contains a specification of how the value\nshould be presented, including such details as field width, alignment,\npadding, decimal precision and so on.  Each value type can define its\nown "formatting mini-language" or interpretation of the *format_spec*.\n\nMost built-in types support a common formatting mini-language, which\nis described in the next section.\n\nA *format_spec* field can also include nested replacement fields\nwithin it. These nested replacement fields can contain only a field\nname; conversion flags and format specifications are not allowed.  The\nreplacement fields within the format_spec are substituted before the\n*format_spec* string is interpreted. This allows the formatting of a\nvalue to be dynamically specified.\n\nSee the *Format examples* section for some examples.\n\n\nFormat Specification Mini-Language\n==================================\n\n"Format specifications" are used within replacement fields contained\nwithin a format string to define how individual values are presented\n(see *Format String Syntax*).  They can also be passed directly to the\nbuilt-in ``format()`` function.  Each formattable type may define how\nthe format specification is to be interpreted.\n\nMost built-in types implement the following options for format\nspecifications, although some of the formatting options are only\nsupported by the numeric types.\n\nA general convention is that an empty format string (``""``) produces\nthe same result as if you had called ``str()`` on the value. A non-\nempty format string typically modifies the result.\n\nThe general form of a *standard format specifier* is:\n\n   format_spec ::= [[fill]align][sign][#][0][width][,][.precision][type]\n   fill        ::= <a character other than \'}\'>\n   align       ::= "<" | ">" | "=" | "^"\n   sign        ::= "+" | "-" | " "\n   width       ::= integer\n   precision   ::= integer\n   type        ::= "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"\n\nThe *fill* character can be any character other than \'{\' or \'}\'.  The\npresence of a fill character is signaled by the character following\nit, which must be one of the alignment options.  If the second\ncharacter of *format_spec* is not a valid alignment option, then it is\nassumed that both the fill character and the alignment option are\nabsent.\n\nThe meaning of the various alignment options is as follows:\n\n   +-----------+------------------------------------------------------------+\n   | Option    | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'<\'``   | Forces the field to be left-aligned within the available   |\n   |           | space (this is the default for most objects).              |\n   +-----------+------------------------------------------------------------+\n   | ``\'>\'``   | Forces the field to be right-aligned within the available  |\n   |           | space (this is the default for numbers).                   |\n   +-----------+------------------------------------------------------------+\n   | ``\'=\'``   | Forces the padding to be placed after the sign (if any)    |\n   |           | but before the digits.  This is used for printing fields   |\n   |           | in the form \'+000000120\'. This alignment option is only    |\n   |           | valid for numeric types.                                   |\n   +-----------+------------------------------------------------------------+\n   | ``\'^\'``   | Forces the field to be centered within the available       |\n   |           | space.                                                     |\n   +-----------+------------------------------------------------------------+\n\nNote that unless a minimum field width is defined, the field width\nwill always be the same size as the data to fill it, so that the\nalignment option has no meaning in this case.\n\nThe *sign* option is only valid for number types, and can be one of\nthe following:\n\n   +-----------+------------------------------------------------------------+\n   | Option    | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'+\'``   | indicates that a sign should be used for both positive as  |\n   |           | well as negative numbers.                                  |\n   +-----------+------------------------------------------------------------+\n   | ``\'-\'``   | indicates that a sign should be used only for negative     |\n   |           | numbers (this is the default behavior).                    |\n   +-----------+------------------------------------------------------------+\n   | space     | indicates that a leading space should be used on positive  |\n   |           | numbers, and a minus sign on negative numbers.             |\n   +-----------+------------------------------------------------------------+\n\nThe ``\'#\'`` option causes the "alternate form" to be used for the\nconversion.  The alternate form is defined differently for different\ntypes.  This option is only valid for integer, float, complex and\nDecimal types. For integers, when binary, octal, or hexadecimal output\nis used, this option adds the prefix respective ``\'0b\'``, ``\'0o\'``, or\n``\'0x\'`` to the output value. For floats, complex and Decimal the\nalternate form causes the result of the conversion to always contain a\ndecimal-point character, even if no digits follow it. Normally, a\ndecimal-point character appears in the result of these conversions\nonly if a digit follows it. In addition, for ``\'g\'`` and ``\'G\'``\nconversions, trailing zeros are not removed from the result.\n\nThe ``\',\'`` option signals the use of a comma for a thousands\nseparator. For a locale aware separator, use the ``\'n\'`` integer\npresentation type instead.\n\nChanged in version 3.1: Added the ``\',\'`` option (see also **PEP\n378**).\n\n*width* is a decimal integer defining the minimum field width.  If not\nspecified, then the field width will be determined by the content.\n\nIf the *width* field is preceded by a zero (``\'0\'``) character, this\nenables zero-padding.  This is equivalent to an *alignment* type of\n``\'=\'`` and a *fill* character of ``\'0\'``.\n\nThe *precision* is a decimal number indicating how many digits should\nbe displayed after the decimal point for a floating point value\nformatted with ``\'f\'`` and ``\'F\'``, or before and after the decimal\npoint for a floating point value formatted with ``\'g\'`` or ``\'G\'``.\nFor non-number types the field indicates the maximum field size - in\nother words, how many characters will be used from the field content.\nThe *precision* is not allowed for integer values.\n\nFinally, the *type* determines how the data should be presented.\n\nThe available string presentation types are:\n\n   +-----------+------------------------------------------------------------+\n   | Type      | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'s\'``   | String format. This is the default type for strings and    |\n   |           | may be omitted.                                            |\n   +-----------+------------------------------------------------------------+\n   | None      | The same as ``\'s\'``.                                       |\n   +-----------+------------------------------------------------------------+\n\nThe available integer presentation types are:\n\n   +-----------+------------------------------------------------------------+\n   | Type      | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'b\'``   | Binary format. Outputs the number in base 2.               |\n   +-----------+------------------------------------------------------------+\n   | ``\'c\'``   | Character. Converts the integer to the corresponding       |\n   |           | unicode character before printing.                         |\n   +-----------+------------------------------------------------------------+\n   | ``\'d\'``   | Decimal Integer. Outputs the number in base 10.            |\n   +-----------+------------------------------------------------------------+\n   | ``\'o\'``   | Octal format. Outputs the number in base 8.                |\n   +-----------+------------------------------------------------------------+\n   | ``\'x\'``   | Hex format. Outputs the number in base 16, using lower-    |\n   |           | case letters for the digits above 9.                       |\n   +-----------+------------------------------------------------------------+\n   | ``\'X\'``   | Hex format. Outputs the number in base 16, using upper-    |\n   |           | case letters for the digits above 9.                       |\n   +-----------+------------------------------------------------------------+\n   | ``\'n\'``   | Number. This is the same as ``\'d\'``, except that it uses   |\n   |           | the current locale setting to insert the appropriate       |\n   |           | number separator characters.                               |\n   +-----------+------------------------------------------------------------+\n   | None      | The same as ``\'d\'``.                                       |\n   +-----------+------------------------------------------------------------+\n\nIn addition to the above presentation types, integers can be formatted\nwith the floating point presentation types listed below (except\n``\'n\'`` and None). When doing so, ``float()`` is used to convert the\ninteger to a floating point number before formatting.\n\nThe available presentation types for floating point and decimal values\nare:\n\n   +-----------+------------------------------------------------------------+\n   | Type      | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'e\'``   | Exponent notation. Prints the number in scientific         |\n   |           | notation using the letter \'e\' to indicate the exponent.    |\n   +-----------+------------------------------------------------------------+\n   | ``\'E\'``   | Exponent notation. Same as ``\'e\'`` except it uses an upper |\n   |           | case \'E\' as the separator character.                       |\n   +-----------+------------------------------------------------------------+\n   | ``\'f\'``   | Fixed point. Displays the number as a fixed-point number.  |\n   +-----------+------------------------------------------------------------+\n   | ``\'F\'``   | Fixed point. Same as ``\'f\'``, but converts ``nan`` to      |\n   |           | ``NAN`` and ``inf`` to ``INF``.                            |\n   +-----------+------------------------------------------------------------+\n   | ``\'g\'``   | General format.  For a given precision ``p >= 1``, this    |\n   |           | rounds the number to ``p`` significant digits and then     |\n   |           | formats the result in either fixed-point format or in      |\n   |           | scientific notation, depending on its magnitude.  The      |\n   |           | precise rules are as follows: suppose that the result      |\n   |           | formatted with presentation type ``\'e\'`` and precision     |\n   |           | ``p-1`` would have exponent ``exp``.  Then if ``-4 <= exp  |\n   |           | < p``, the number is formatted with presentation type      |\n   |           | ``\'f\'`` and precision ``p-1-exp``. Otherwise, the number   |\n   |           | is formatted with presentation type ``\'e\'`` and precision  |\n   |           | ``p-1``. In both cases insignificant trailing zeros are    |\n   |           | removed from the significand, and the decimal point is     |\n   |           | also removed if there are no remaining digits following    |\n   |           | it.  Positive and negative infinity, positive and negative |\n   |           | zero, and nans, are formatted as ``inf``, ``-inf``, ``0``, |\n   |           | ``-0`` and ``nan`` respectively, regardless of the         |\n   |           | precision.  A precision of ``0`` is treated as equivalent  |\n   |           | to a precision of ``1``.                                   |\n   +-----------+------------------------------------------------------------+\n   | ``\'G\'``   | General format. Same as ``\'g\'`` except switches to ``\'E\'`` |\n   |           | if the number gets too large. The representations of       |\n   |           | infinity and NaN are uppercased, too.                      |\n   +-----------+------------------------------------------------------------+\n   | ``\'n\'``   | Number. This is the same as ``\'g\'``, except that it uses   |\n   |           | the current locale setting to insert the appropriate       |\n   |           | number separator characters.                               |\n   +-----------+------------------------------------------------------------+\n   | ``\'%\'``   | Percentage. Multiplies the number by 100 and displays in   |\n   |           | fixed (``\'f\'``) format, followed by a percent sign.        |\n   +-----------+------------------------------------------------------------+\n   | None      | Similar to ``\'g\'``, except with at least one digit past    |\n   |           | the decimal point and a default precision of 12. This is   |\n   |           | intended to match ``str()``, except you can add the other  |\n   |           | format modifiers.                                          |\n   +-----------+------------------------------------------------------------+\n\n\nFormat examples\n===============\n\nThis section contains examples of the new format syntax and comparison\nwith the old ``%``-formatting.\n\nIn most of the cases the syntax is similar to the old\n``%``-formatting, with the addition of the ``{}`` and with ``:`` used\ninstead of ``%``. For example, ``\'%03.2f\'`` can be translated to\n``\'{:03.2f}\'``.\n\nThe new format syntax also supports new and different options, shown\nin the follow examples.\n\nAccessing arguments by position:\n\n   >>> \'{0}, {1}, {2}\'.format(\'a\', \'b\', \'c\')\n   \'a, b, c\'\n   >>> \'{}, {}, {}\'.format(\'a\', \'b\', \'c\')  # 3.1+ only\n   \'a, b, c\'\n   >>> \'{2}, {1}, {0}\'.format(\'a\', \'b\', \'c\')\n   \'c, b, a\'\n   >>> \'{2}, {1}, {0}\'.format(*\'abc\')      # unpacking argument sequence\n   \'c, b, a\'\n   >>> \'{0}{1}{0}\'.format(\'abra\', \'cad\')   # arguments\' indices can be repeated\n   \'abracadabra\'\n\nAccessing arguments by name:\n\n   >>> \'Coordinates: {latitude}, {longitude}\'.format(latitude=\'37.24N\', longitude=\'-115.81W\')\n   \'Coordinates: 37.24N, -115.81W\'\n   >>> coord = {\'latitude\': \'37.24N\', \'longitude\': \'-115.81W\'}\n   >>> \'Coordinates: {latitude}, {longitude}\'.format(**coord)\n   \'Coordinates: 37.24N, -115.81W\'\n\nAccessing arguments\' attributes:\n\n   >>> c = 3-5j\n   >>> (\'The complex number {0} is formed from the real part {0.real} \'\n   ...  \'and the imaginary part {0.imag}.\').format(c)\n   \'The complex number (3-5j) is formed from the real part 3.0 and the imaginary part -5.0.\'\n   >>> class Point:\n   ...     def __init__(self, x, y):\n   ...         self.x, self.y = x, y\n   ...     def __str__(self):\n   ...         return \'Point({self.x}, {self.y})\'.format(self=self)\n   ...\n   >>> str(Point(4, 2))\n   \'Point(4, 2)\'\n\nAccessing arguments\' items:\n\n   >>> coord = (3, 5)\n   >>> \'X: {0[0]};  Y: {0[1]}\'.format(coord)\n   \'X: 3;  Y: 5\'\n\nReplacing ``%s`` and ``%r``:\n\n   >>> "repr() shows quotes: {!r}; str() doesn\'t: {!s}".format(\'test1\', \'test2\')\n   "repr() shows quotes: \'test1\'; str() doesn\'t: test2"\n\nAligning the text and specifying a width:\n\n   >>> \'{:<30}\'.format(\'left aligned\')\n   \'left aligned                  \'\n   >>> \'{:>30}\'.format(\'right aligned\')\n   \'                 right aligned\'\n   >>> \'{:^30}\'.format(\'centered\')\n   \'           centered           \'\n   >>> \'{:*^30}\'.format(\'centered\')  # use \'*\' as a fill char\n   \'***********centered***********\'\n\nReplacing ``%+f``, ``%-f``, and ``% f`` and specifying a sign:\n\n   >>> \'{:+f}; {:+f}\'.format(3.14, -3.14)  # show it always\n   \'+3.140000; -3.140000\'\n   >>> \'{: f}; {: f}\'.format(3.14, -3.14)  # show a space for positive numbers\n   \' 3.140000; -3.140000\'\n   >>> \'{:-f}; {:-f}\'.format(3.14, -3.14)  # show only the minus -- same as \'{:f}; {:f}\'\n   \'3.140000; -3.140000\'\n\nReplacing ``%x`` and ``%o`` and converting the value to different\nbases:\n\n   >>> # format also supports binary numbers\n   >>> "int: {0:d};  hex: {0:x};  oct: {0:o};  bin: {0:b}".format(42)\n   \'int: 42;  hex: 2a;  oct: 52;  bin: 101010\'\n   >>> # with 0x, 0o, or 0b as prefix:\n   >>> "int: {0:d};  hex: {0:#x};  oct: {0:#o};  bin: {0:#b}".format(42)\n   \'int: 42;  hex: 0x2a;  oct: 0o52;  bin: 0b101010\'\n\nUsing the comma as a thousands separator:\n\n   >>> \'{:,}\'.format(1234567890)\n   \'1,234,567,890\'\n\nExpressing a percentage:\n\n   >>> points = 19\n   >>> total = 22\n   >>> \'Correct answers: {:.2%}.\'.format(points/total)\n   \'Correct answers: 86.36%\'\n\nUsing type-specific formatting:\n\n   >>> import datetime\n   >>> d = datetime.datetime(2010, 7, 4, 12, 15, 58)\n   >>> \'{:%Y-%m-%d %H:%M:%S}\'.format(d)\n   \'2010-07-04 12:15:58\'\n\nNesting arguments and more complex examples:\n\n   >>> for align, text in zip(\'<^>\', [\'left\', \'center\', \'right\']):\n   ...     \'{0:{fill}{align}16}\'.format(text, fill=align, align=align)\n   ...\n   \'left<<<<<<<<<<<<\'\n   \'^^^^^center^^^^^\'\n   \'>>>>>>>>>>>right\'\n   >>>\n   >>> octets = [192, 168, 0, 1]\n   >>> \'{:02X}{:02X}{:02X}{:02X}\'.format(*octets)\n   \'C0A80001\'\n   >>> int(_, 16)\n   3232235521\n   >>>\n   >>> width = 5\n   >>> for num in range(5,12):\n   ...     for base in \'dXob\':\n   ...         print(\'{0:{width}{base}}\'.format(num, base=base, width=width), end=\' \')\n   ...     print()\n   ...\n       5     5     5   101\n       6     6     6   110\n       7     7     7   111\n       8     8    10  1000\n       9     9    11  1001\n      10     A    12  1010\n      11     B    13  1011\n',
+ 'formatstrings': '\nFormat String Syntax\n********************\n\nThe ``str.format()`` method and the ``Formatter`` class share the same\nsyntax for format strings (although in the case of ``Formatter``,\nsubclasses can define their own format string syntax).\n\nFormat strings contain "replacement fields" surrounded by curly braces\n``{}``. Anything that is not contained in braces is considered literal\ntext, which is copied unchanged to the output.  If you need to include\na brace character in the literal text, it can be escaped by doubling:\n``{{`` and ``}}``.\n\nThe grammar for a replacement field is as follows:\n\n      replacement_field ::= "{" [field_name] ["!" conversion] [":" format_spec] "}"\n      field_name        ::= arg_name ("." attribute_name | "[" element_index "]")*\n      arg_name          ::= [identifier | integer]\n      attribute_name    ::= identifier\n      element_index     ::= integer | index_string\n      index_string      ::= <any source character except "]"> +\n      conversion        ::= "r" | "s" | "a"\n      format_spec       ::= <described in the next section>\n\nIn less formal terms, the replacement field can start with a\n*field_name* that specifies the object whose value is to be formatted\nand inserted into the output instead of the replacement field. The\n*field_name* is optionally followed by a  *conversion* field, which is\npreceded by an exclamation point ``\'!\'``, and a *format_spec*, which\nis preceded by a colon ``\':\'``.  These specify a non-default format\nfor the replacement value.\n\nSee also the *Format Specification Mini-Language* section.\n\nThe *field_name* itself begins with an *arg_name* that is either\neither a number or a keyword.  If it\'s a number, it refers to a\npositional argument, and if it\'s a keyword, it refers to a named\nkeyword argument.  If the numerical arg_names in a format string are\n0, 1, 2, ... in sequence, they can all be omitted (not just some) and\nthe numbers 0, 1, 2, ... will be automatically inserted in that order.\nBecause *arg_name* is not quote-delimited, it is not possible to\nspecify arbitrary dictionary keys (e.g., the strings ``\'10\'`` or\n``\':-]\'``) within a format string. The *arg_name* can be followed by\nany number of index or attribute expressions. An expression of the\nform ``\'.name\'`` selects the named attribute using ``getattr()``,\nwhile an expression of the form ``\'[index]\'`` does an index lookup\nusing ``__getitem__()``.\n\nChanged in version 3.1: The positional argument specifiers can be\nomitted, so ``\'{} {}\'`` is equivalent to ``\'{0} {1}\'``.\n\nSome simple format string examples:\n\n   "First, thou shalt count to {0}" # References first positional argument\n   "Bring me a {}"                  # Implicitly references the first positional argument\n   "From {} to {}"                  # Same as "From {0} to {1}"\n   "My quest is {name}"             # References keyword argument \'name\'\n   "Weight in tons {0.weight}"      # \'weight\' attribute of first positional arg\n   "Units destroyed: {players[0]}"  # First element of keyword argument \'players\'.\n\nThe *conversion* field causes a type coercion before formatting.\nNormally, the job of formatting a value is done by the\n``__format__()`` method of the value itself.  However, in some cases\nit is desirable to force a type to be formatted as a string,\noverriding its own definition of formatting.  By converting the value\nto a string before calling ``__format__()``, the normal formatting\nlogic is bypassed.\n\nThree conversion flags are currently supported: ``\'!s\'`` which calls\n``str()`` on the value, ``\'!r\'`` which calls ``repr()`` and ``\'!a\'``\nwhich calls ``ascii()``.\n\nSome examples:\n\n   "Harold\'s a clever {0!s}"        # Calls str() on the argument first\n   "Bring out the holy {name!r}"    # Calls repr() on the argument first\n   "More {!a}"                      # Calls ascii() on the argument first\n\nThe *format_spec* field contains a specification of how the value\nshould be presented, including such details as field width, alignment,\npadding, decimal precision and so on.  Each value type can define its\nown "formatting mini-language" or interpretation of the *format_spec*.\n\nMost built-in types support a common formatting mini-language, which\nis described in the next section.\n\nA *format_spec* field can also include nested replacement fields\nwithin it. These nested replacement fields can contain only a field\nname; conversion flags and format specifications are not allowed.  The\nreplacement fields within the format_spec are substituted before the\n*format_spec* string is interpreted. This allows the formatting of a\nvalue to be dynamically specified.\n\nSee the *Format examples* section for some examples.\n\n\nFormat Specification Mini-Language\n==================================\n\n"Format specifications" are used within replacement fields contained\nwithin a format string to define how individual values are presented\n(see *Format String Syntax*).  They can also be passed directly to the\nbuilt-in ``format()`` function.  Each formattable type may define how\nthe format specification is to be interpreted.\n\nMost built-in types implement the following options for format\nspecifications, although some of the formatting options are only\nsupported by the numeric types.\n\nA general convention is that an empty format string (``""``) produces\nthe same result as if you had called ``str()`` on the value. A non-\nempty format string typically modifies the result.\n\nThe general form of a *standard format specifier* is:\n\n   format_spec ::= [[fill]align][sign][#][0][width][,][.precision][type]\n   fill        ::= <a character other than \'}\'>\n   align       ::= "<" | ">" | "=" | "^"\n   sign        ::= "+" | "-" | " "\n   width       ::= integer\n   precision   ::= integer\n   type        ::= "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"\n\nThe *fill* character can be any character other than \'{\' or \'}\'.  The\npresence of a fill character is signaled by the character following\nit, which must be one of the alignment options.  If the second\ncharacter of *format_spec* is not a valid alignment option, then it is\nassumed that both the fill character and the alignment option are\nabsent.\n\nThe meaning of the various alignment options is as follows:\n\n   +-----------+------------------------------------------------------------+\n   | Option    | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'<\'``   | Forces the field to be left-aligned within the available   |\n   |           | space (this is the default for most objects).              |\n   +-----------+------------------------------------------------------------+\n   | ``\'>\'``   | Forces the field to be right-aligned within the available  |\n   |           | space (this is the default for numbers).                   |\n   +-----------+------------------------------------------------------------+\n   | ``\'=\'``   | Forces the padding to be placed after the sign (if any)    |\n   |           | but before the digits.  This is used for printing fields   |\n   |           | in the form \'+000000120\'. This alignment option is only    |\n   |           | valid for numeric types.                                   |\n   +-----------+------------------------------------------------------------+\n   | ``\'^\'``   | Forces the field to be centered within the available       |\n   |           | space.                                                     |\n   +-----------+------------------------------------------------------------+\n\nNote that unless a minimum field width is defined, the field width\nwill always be the same size as the data to fill it, so that the\nalignment option has no meaning in this case.\n\nThe *sign* option is only valid for number types, and can be one of\nthe following:\n\n   +-----------+------------------------------------------------------------+\n   | Option    | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'+\'``   | indicates that a sign should be used for both positive as  |\n   |           | well as negative numbers.                                  |\n   +-----------+------------------------------------------------------------+\n   | ``\'-\'``   | indicates that a sign should be used only for negative     |\n   |           | numbers (this is the default behavior).                    |\n   +-----------+------------------------------------------------------------+\n   | space     | indicates that a leading space should be used on positive  |\n   |           | numbers, and a minus sign on negative numbers.             |\n   +-----------+------------------------------------------------------------+\n\nThe ``\'#\'`` option causes the "alternate form" to be used for the\nconversion.  The alternate form is defined differently for different\ntypes.  This option is only valid for integer, float, complex and\nDecimal types. For integers, when binary, octal, or hexadecimal output\nis used, this option adds the prefix respective ``\'0b\'``, ``\'0o\'``, or\n``\'0x\'`` to the output value. For floats, complex and Decimal the\nalternate form causes the result of the conversion to always contain a\ndecimal-point character, even if no digits follow it. Normally, a\ndecimal-point character appears in the result of these conversions\nonly if a digit follows it. In addition, for ``\'g\'`` and ``\'G\'``\nconversions, trailing zeros are not removed from the result.\n\nThe ``\',\'`` option signals the use of a comma for a thousands\nseparator. For a locale aware separator, use the ``\'n\'`` integer\npresentation type instead.\n\nChanged in version 3.1: Added the ``\',\'`` option (see also **PEP\n378**).\n\n*width* is a decimal integer defining the minimum field width.  If not\nspecified, then the field width will be determined by the content.\n\nIf the *width* field is preceded by a zero (``\'0\'``) character, this\nenables zero-padding.  This is equivalent to an *alignment* type of\n``\'=\'`` and a *fill* character of ``\'0\'``.\n\nThe *precision* is a decimal number indicating how many digits should\nbe displayed after the decimal point for a floating point value\nformatted with ``\'f\'`` and ``\'F\'``, or before and after the decimal\npoint for a floating point value formatted with ``\'g\'`` or ``\'G\'``.\nFor non-number types the field indicates the maximum field size - in\nother words, how many characters will be used from the field content.\nThe *precision* is not allowed for integer values.\n\nFinally, the *type* determines how the data should be presented.\n\nThe available string presentation types are:\n\n   +-----------+------------------------------------------------------------+\n   | Type      | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'s\'``   | String format. This is the default type for strings and    |\n   |           | may be omitted.                                            |\n   +-----------+------------------------------------------------------------+\n   | None      | The same as ``\'s\'``.                                       |\n   +-----------+------------------------------------------------------------+\n\nThe available integer presentation types are:\n\n   +-----------+------------------------------------------------------------+\n   | Type      | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'b\'``   | Binary format. Outputs the number in base 2.               |\n   +-----------+------------------------------------------------------------+\n   | ``\'c\'``   | Character. Converts the integer to the corresponding       |\n   |           | unicode character before printing.                         |\n   +-----------+------------------------------------------------------------+\n   | ``\'d\'``   | Decimal Integer. Outputs the number in base 10.            |\n   +-----------+------------------------------------------------------------+\n   | ``\'o\'``   | Octal format. Outputs the number in base 8.                |\n   +-----------+------------------------------------------------------------+\n   | ``\'x\'``   | Hex format. Outputs the number in base 16, using lower-    |\n   |           | case letters for the digits above 9.                       |\n   +-----------+------------------------------------------------------------+\n   | ``\'X\'``   | Hex format. Outputs the number in base 16, using upper-    |\n   |           | case letters for the digits above 9.                       |\n   +-----------+------------------------------------------------------------+\n   | ``\'n\'``   | Number. This is the same as ``\'d\'``, except that it uses   |\n   |           | the current locale setting to insert the appropriate       |\n   |           | number separator characters.                               |\n   +-----------+------------------------------------------------------------+\n   | None      | The same as ``\'d\'``.                                       |\n   +-----------+------------------------------------------------------------+\n\nIn addition to the above presentation types, integers can be formatted\nwith the floating point presentation types listed below (except\n``\'n\'`` and None). When doing so, ``float()`` is used to convert the\ninteger to a floating point number before formatting.\n\nThe available presentation types for floating point and decimal values\nare:\n\n   +-----------+------------------------------------------------------------+\n   | Type      | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'e\'``   | Exponent notation. Prints the number in scientific         |\n   |           | notation using the letter \'e\' to indicate the exponent.    |\n   +-----------+------------------------------------------------------------+\n   | ``\'E\'``   | Exponent notation. Same as ``\'e\'`` except it uses an upper |\n   |           | case \'E\' as the separator character.                       |\n   +-----------+------------------------------------------------------------+\n   | ``\'f\'``   | Fixed point. Displays the number as a fixed-point number.  |\n   +-----------+------------------------------------------------------------+\n   | ``\'F\'``   | Fixed point. Same as ``\'f\'``, but converts ``nan`` to      |\n   |           | ``NAN`` and ``inf`` to ``INF``.                            |\n   +-----------+------------------------------------------------------------+\n   | ``\'g\'``   | General format.  For a given precision ``p >= 1``, this    |\n   |           | rounds the number to ``p`` significant digits and then     |\n   |           | formats the result in either fixed-point format or in      |\n   |           | scientific notation, depending on its magnitude.  The      |\n   |           | precise rules are as follows: suppose that the result      |\n   |           | formatted with presentation type ``\'e\'`` and precision     |\n   |           | ``p-1`` would have exponent ``exp``.  Then if ``-4 <= exp  |\n   |           | < p``, the number is formatted with presentation type      |\n   |           | ``\'f\'`` and precision ``p-1-exp``. Otherwise, the number   |\n   |           | is formatted with presentation type ``\'e\'`` and precision  |\n   |           | ``p-1``. In both cases insignificant trailing zeros are    |\n   |           | removed from the significand, and the decimal point is     |\n   |           | also removed if there are no remaining digits following    |\n   |           | it.  Positive and negative infinity, positive and negative |\n   |           | zero, and nans, are formatted as ``inf``, ``-inf``, ``0``, |\n   |           | ``-0`` and ``nan`` respectively, regardless of the         |\n   |           | precision.  A precision of ``0`` is treated as equivalent  |\n   |           | to a precision of ``1``.                                   |\n   +-----------+------------------------------------------------------------+\n   | ``\'G\'``   | General format. Same as ``\'g\'`` except switches to ``\'E\'`` |\n   |           | if the number gets too large. The representations of       |\n   |           | infinity and NaN are uppercased, too.                      |\n   +-----------+------------------------------------------------------------+\n   | ``\'n\'``   | Number. This is the same as ``\'g\'``, except that it uses   |\n   |           | the current locale setting to insert the appropriate       |\n   |           | number separator characters.                               |\n   +-----------+------------------------------------------------------------+\n   | ``\'%\'``   | Percentage. Multiplies the number by 100 and displays in   |\n   |           | fixed (``\'f\'``) format, followed by a percent sign.        |\n   +-----------+------------------------------------------------------------+\n   | None      | Similar to ``\'g\'``, except with at least one digit past    |\n   |           | the decimal point and a default precision of 12. This is   |\n   |           | intended to match ``str()``, except you can add the other  |\n   |           | format modifiers.                                          |\n   +-----------+------------------------------------------------------------+\n\n\nFormat examples\n===============\n\nThis section contains examples of the new format syntax and comparison\nwith the old ``%``-formatting.\n\nIn most of the cases the syntax is similar to the old\n``%``-formatting, with the addition of the ``{}`` and with ``:`` used\ninstead of ``%``. For example, ``\'%03.2f\'`` can be translated to\n``\'{:03.2f}\'``.\n\nThe new format syntax also supports new and different options, shown\nin the follow examples.\n\nAccessing arguments by position:\n\n   >>> \'{0}, {1}, {2}\'.format(\'a\', \'b\', \'c\')\n   \'a, b, c\'\n   >>> \'{}, {}, {}\'.format(\'a\', \'b\', \'c\')  # 3.1+ only\n   \'a, b, c\'\n   >>> \'{2}, {1}, {0}\'.format(\'a\', \'b\', \'c\')\n   \'c, b, a\'\n   >>> \'{2}, {1}, {0}\'.format(*\'abc\')      # unpacking argument sequence\n   \'c, b, a\'\n   >>> \'{0}{1}{0}\'.format(\'abra\', \'cad\')   # arguments\' indices can be repeated\n   \'abracadabra\'\n\nAccessing arguments by name:\n\n   >>> \'Coordinates: {latitude}, {longitude}\'.format(latitude=\'37.24N\', longitude=\'-115.81W\')\n   \'Coordinates: 37.24N, -115.81W\'\n   >>> coord = {\'latitude\': \'37.24N\', \'longitude\': \'-115.81W\'}\n   >>> \'Coordinates: {latitude}, {longitude}\'.format(**coord)\n   \'Coordinates: 37.24N, -115.81W\'\n\nAccessing arguments\' attributes:\n\n   >>> c = 3-5j\n   >>> (\'The complex number {0} is formed from the real part {0.real} \'\n   ...  \'and the imaginary part {0.imag}.\').format(c)\n   \'The complex number (3-5j) is formed from the real part 3.0 and the imaginary part -5.0.\'\n   >>> class Point:\n   ...     def __init__(self, x, y):\n   ...         self.x, self.y = x, y\n   ...     def __str__(self):\n   ...         return \'Point({self.x}, {self.y})\'.format(self=self)\n   ...\n   >>> str(Point(4, 2))\n   \'Point(4, 2)\'\n\nAccessing arguments\' items:\n\n   >>> coord = (3, 5)\n   >>> \'X: {0[0]};  Y: {0[1]}\'.format(coord)\n   \'X: 3;  Y: 5\'\n\nReplacing ``%s`` and ``%r``:\n\n   >>> "repr() shows quotes: {!r}; str() doesn\'t: {!s}".format(\'test1\', \'test2\')\n   "repr() shows quotes: \'test1\'; str() doesn\'t: test2"\n\nAligning the text and specifying a width:\n\n   >>> \'{:<30}\'.format(\'left aligned\')\n   \'left aligned                  \'\n   >>> \'{:>30}\'.format(\'right aligned\')\n   \'                 right aligned\'\n   >>> \'{:^30}\'.format(\'centered\')\n   \'           centered           \'\n   >>> \'{:*^30}\'.format(\'centered\')  # use \'*\' as a fill char\n   \'***********centered***********\'\n\nReplacing ``%+f``, ``%-f``, and ``% f`` and specifying a sign:\n\n   >>> \'{:+f}; {:+f}\'.format(3.14, -3.14)  # show it always\n   \'+3.140000; -3.140000\'\n   >>> \'{: f}; {: f}\'.format(3.14, -3.14)  # show a space for positive numbers\n   \' 3.140000; -3.140000\'\n   >>> \'{:-f}; {:-f}\'.format(3.14, -3.14)  # show only the minus -- same as \'{:f}; {:f}\'\n   \'3.140000; -3.140000\'\n\nReplacing ``%x`` and ``%o`` and converting the value to different\nbases:\n\n   >>> # format also supports binary numbers\n   >>> "int: {0:d};  hex: {0:x};  oct: {0:o};  bin: {0:b}".format(42)\n   \'int: 42;  hex: 2a;  oct: 52;  bin: 101010\'\n   >>> # with 0x, 0o, or 0b as prefix:\n   >>> "int: {0:d};  hex: {0:#x};  oct: {0:#o};  bin: {0:#b}".format(42)\n   \'int: 42;  hex: 0x2a;  oct: 0o52;  bin: 0b101010\'\n\nUsing the comma as a thousands separator:\n\n   >>> \'{:,}\'.format(1234567890)\n   \'1,234,567,890\'\n\nExpressing a percentage:\n\n   >>> points = 19\n   >>> total = 22\n   >>> \'Correct answers: {:.2%}.\'.format(points/total)\n   \'Correct answers: 86.36%\'\n\nUsing type-specific formatting:\n\n   >>> import datetime\n   >>> d = datetime.datetime(2010, 7, 4, 12, 15, 58)\n   >>> \'{:%Y-%m-%d %H:%M:%S}\'.format(d)\n   \'2010-07-04 12:15:58\'\n\nNesting arguments and more complex examples:\n\n   >>> for align, text in zip(\'<^>\', [\'left\', \'center\', \'right\']):\n   ...     \'{0:{fill}{align}16}\'.format(text, fill=align, align=align)\n   ...\n   \'left<<<<<<<<<<<<\'\n   \'^^^^^center^^^^^\'\n   \'>>>>>>>>>>>right\'\n   >>>\n   >>> octets = [192, 168, 0, 1]\n   >>> \'{:02X}{:02X}{:02X}{:02X}\'.format(*octets)\n   \'C0A80001\'\n   >>> int(_, 16)\n   3232235521\n   >>>\n   >>> width = 5\n   >>> for num in range(5,12):\n   ...     for base in \'dXob\':\n   ...         print(\'{0:{width}{base}}\'.format(num, base=base, width=width), end=\' \')\n   ...     print()\n   ...\n       5     5     5   101\n       6     6     6   110\n       7     7     7   111\n       8     8    10  1000\n       9     9    11  1001\n      10     A    12  1010\n      11     B    13  1011\n',
  'function': '\nFunction definitions\n********************\n\nA function definition defines a user-defined function object (see\nsection *The standard type hierarchy*):\n\n   funcdef        ::= [decorators] "def" funcname "(" [parameter_list] ")" ["->" expression] ":" suite\n   decorators     ::= decorator+\n   decorator      ::= "@" dotted_name ["(" [argument_list [","]] ")"] NEWLINE\n   dotted_name    ::= identifier ("." identifier)*\n   parameter_list ::= (defparameter ",")*\n                      (  "*" [parameter] ("," defparameter)*\n                      [, "**" parameter]\n                      | "**" parameter\n                      | defparameter [","] )\n   parameter      ::= identifier [":" expression]\n   defparameter   ::= parameter ["=" expression]\n   funcname       ::= identifier\n\nA function definition is an executable statement.  Its execution binds\nthe function name in the current local namespace to a function object\n(a wrapper around the executable code for the function).  This\nfunction object contains a reference to the current global namespace\nas the global namespace to be used when the function is called.\n\nThe function definition does not execute the function body; this gets\nexecuted only when the function is called. [3]\n\nA function definition may be wrapped by one or more *decorator*\nexpressions. Decorator expressions are evaluated when the function is\ndefined, in the scope that contains the function definition.  The\nresult must be a callable, which is invoked with the function object\nas the only argument. The returned value is bound to the function name\ninstead of the function object.  Multiple decorators are applied in\nnested fashion. For example, the following code\n\n   @f1(arg)\n   @f2\n   def func(): pass\n\nis equivalent to\n\n   def func(): pass\n   func = f1(arg)(f2(func))\n\nWhen one or more parameters have the form *parameter* ``=``\n*expression*, the function is said to have "default parameter values."\nFor a parameter with a default value, the corresponding argument may\nbe omitted from a call, in which case the parameter\'s default value is\nsubstituted.  If a parameter has a default value, all following\nparameters up until the "``*``" must also have a default value ---\nthis is a syntactic restriction that is not expressed by the grammar.\n\n**Default parameter values are evaluated when the function definition\nis executed.** This means that the expression is evaluated once, when\nthe function is defined, and that that same "pre-computed" value is\nused for each call.  This is especially important to understand when a\ndefault parameter is a mutable object, such as a list or a dictionary:\nif the function modifies the object (e.g. by appending an item to a\nlist), the default value is in effect modified. This is generally not\nwhat was intended.  A way around this is to use ``None`` as the\ndefault, and explicitly test for it in the body of the function, e.g.:\n\n   def whats_on_the_telly(penguin=None):\n       if penguin is None:\n           penguin = []\n       penguin.append("property of the zoo")\n       return penguin\n\nFunction call semantics are described in more detail in section\n*Calls*. A function call always assigns values to all parameters\nmentioned in the parameter list, either from position arguments, from\nkeyword arguments, or from default values.  If the form\n"``*identifier``" is present, it is initialized to a tuple receiving\nany excess positional parameters, defaulting to the empty tuple.  If\nthe form "``**identifier``" is present, it is initialized to a new\ndictionary receiving any excess keyword arguments, defaulting to a new\nempty dictionary. Parameters after "``*``" or "``*identifier``" are\nkeyword-only parameters and may only be passed used keyword arguments.\n\nParameters may have annotations of the form "``: expression``"\nfollowing the parameter name.  Any parameter may have an annotation\neven those of the form ``*identifier`` or ``**identifier``.  Functions\nmay have "return" annotation of the form "``-> expression``" after the\nparameter list.  These annotations can be any valid Python expression\nand are evaluated when the function definition is executed.\nAnnotations may be evaluated in a different order than they appear in\nthe source code.  The presence of annotations does not change the\nsemantics of a function.  The annotation values are available as\nvalues of a dictionary keyed by the parameters\' names in the\n``__annotations__`` attribute of the function object.\n\nIt is also possible to create anonymous functions (functions not bound\nto a name), for immediate use in expressions.  This uses lambda forms,\ndescribed in section *Lambdas*.  Note that the lambda form is merely a\nshorthand for a simplified function definition; a function defined in\na "``def``" statement can be passed around or assigned to another name\njust like a function defined by a lambda form.  The "``def``" form is\nactually more powerful since it allows the execution of multiple\nstatements and annotations.\n\n**Programmer\'s note:** Functions are first-class objects.  A "``def``"\nform executed inside a function definition defines a local function\nthat can be returned or passed around.  Free variables used in the\nnested function can access the local variables of the function\ncontaining the def.  See section *Naming and binding* for details.\n',
  'global': '\nThe ``global`` statement\n************************\n\n   global_stmt ::= "global" identifier ("," identifier)*\n\nThe ``global`` statement is a declaration which holds for the entire\ncurrent code block.  It means that the listed identifiers are to be\ninterpreted as globals.  It would be impossible to assign to a global\nvariable without ``global``, although free variables may refer to\nglobals without being declared global.\n\nNames listed in a ``global`` statement must not be used in the same\ncode block textually preceding that ``global`` statement.\n\nNames listed in a ``global`` statement must not be defined as formal\nparameters or in a ``for`` loop control target, ``class`` definition,\nfunction definition, or ``import`` statement.\n\n**CPython implementation detail:** The current implementation does not\nenforce the latter two restrictions, but programs should not abuse\nthis freedom, as future implementations may enforce them or silently\nchange the meaning of the program.\n\n**Programmer\'s note:** the ``global`` is a directive to the parser.\nIt applies only to code parsed at the same time as the ``global``\nstatement. In particular, a ``global`` statement contained in a string\nor code object supplied to the built-in ``exec()`` function does not\naffect the code block *containing* the function call, and code\ncontained in such a string is unaffected by ``global`` statements in\nthe code containing the function call.  The same applies to the\n``eval()`` and ``compile()`` functions.\n',
  'id-classes': '\nReserved classes of identifiers\n*******************************\n\nCertain classes of identifiers (besides keywords) have special\nmeanings.  These classes are identified by the patterns of leading and\ntrailing underscore characters:\n\n``_*``\n   Not imported by ``from module import *``.  The special identifier\n   ``_`` is used in the interactive interpreter to store the result of\n   the last evaluation; it is stored in the ``builtins`` module.  When\n   not in interactive mode, ``_`` has no special meaning and is not\n   defined. See section *The import statement*.\n\n   Note: The name ``_`` is often used in conjunction with\n     internationalization; refer to the documentation for the\n     ``gettext`` module for more information on this convention.\n\n``__*__``\n   System-defined names. These names are defined by the interpreter\n   and its implementation (including the standard library).  Current\n   system names are discussed in the *Special method names* section\n   and elsewhere.  More will likely be defined in future versions of\n   Python.  *Any* use of ``__*__`` names, in any context, that does\n   not follow explicitly documented use, is subject to breakage\n   without warning.\n\n``__*``\n   Class-private names.  Names in this category, when used within the\n   context of a class definition, are re-written to use a mangled form\n   to help avoid name clashes between "private" attributes of base and\n   derived classes. See section *Identifiers (Names)*.\n',

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:45 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:45 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogQnVtcCB0byAzLjIu?=
	=?utf8?q?2=2E?=
Message-ID: <E1R06Pl-0002b0-SZ@dinsdale.python.org>

http://hg.python.org/cpython/rev/2661b7505f72
changeset:   72262:2661b7505f72
branch:      3.2
user:        Georg Brandl <georg at python.org>
date:        Sat Sep 03 11:17:55 2011 +0200
summary:
  Bump to 3.2.2.

files:
  Include/patchlevel.h      |  6 +++---
  Lib/distutils/__init__.py |  2 +-
  Lib/idlelib/idlever.py    |  2 +-
  Misc/NEWS                 |  2 +-
  Misc/RPM/python-3.2.spec  |  2 +-
  README                    |  4 ++--
  6 files changed, 9 insertions(+), 9 deletions(-)


diff --git a/Include/patchlevel.h b/Include/patchlevel.h
--- a/Include/patchlevel.h
+++ b/Include/patchlevel.h
@@ -19,11 +19,11 @@
 #define PY_MAJOR_VERSION	3
 #define PY_MINOR_VERSION	2
 #define PY_MICRO_VERSION	2
-#define PY_RELEASE_LEVEL	PY_RELEASE_LEVEL_GAMMA
-#define PY_RELEASE_SERIAL	1
+#define PY_RELEASE_LEVEL	PY_RELEASE_LEVEL_FINAL
+#define PY_RELEASE_SERIAL	0
 
 /* Version as a string */
-#define PY_VERSION      	"3.2.2rc1+"
+#define PY_VERSION      	"3.2.2"
 /*--end constants--*/
 
 /* Subversion Revision number of this file (not of the repository). Empty
diff --git a/Lib/distutils/__init__.py b/Lib/distutils/__init__.py
--- a/Lib/distutils/__init__.py
+++ b/Lib/distutils/__init__.py
@@ -13,5 +13,5 @@
 # Updated automatically by the Python release process.
 #
 #--start constants--
-__version__ = "3.2.2rc1"
+__version__ = "3.2.2"
 #--end constants--
diff --git a/Lib/idlelib/idlever.py b/Lib/idlelib/idlever.py
--- a/Lib/idlelib/idlever.py
+++ b/Lib/idlelib/idlever.py
@@ -1,1 +1,1 @@
-IDLE_VERSION = "3.2.2rc1"
+IDLE_VERSION = "3.2.2"
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -5,7 +5,7 @@
 What's New in Python 3.2.2?
 ===========================
 
-*Release date: XXXX-XX-XX*
+*Release date: 03-Sep-2011*
 
 Core and Builtins
 -----------------
diff --git a/Misc/RPM/python-3.2.spec b/Misc/RPM/python-3.2.spec
--- a/Misc/RPM/python-3.2.spec
+++ b/Misc/RPM/python-3.2.spec
@@ -39,7 +39,7 @@
 
 %define name python
 #--start constants--
-%define version 3.2.2rc1
+%define version 3.2.2
 %define libvers 3.2
 #--end constants--
 %define release 1pydotorg
diff --git a/README b/README
--- a/README
+++ b/README
@@ -1,5 +1,5 @@
-This is Python version 3.2.2 release candidate 1
-================================================
+This is Python version 3.2.2
+============================
 
 Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011
 Python Software Foundation.  All rights reserved.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:46 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:46 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyMzMz?=
 =?utf8?q?=3A_fix_test=5Fdistutils_failures_under_Solaris_and_derivatives?=
Message-ID: <E1R06Pm-0002b4-Fy@dinsdale.python.org>

http://hg.python.org/cpython/rev/137e45f15c0b
changeset:   72263:137e45f15c0b
branch:      3.2
tag:         v3.2.2
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu Aug 25 18:32:02 2011 +0200
summary:
  Issue #12333: fix test_distutils failures under Solaris and derivatives

files:
  Lib/distutils/tests/support.py |  4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Lib/distutils/tests/support.py b/Lib/distutils/tests/support.py
--- a/Lib/distutils/tests/support.py
+++ b/Lib/distutils/tests/support.py
@@ -54,9 +54,13 @@
 
     def setUp(self):
         super().setUp()
+        self.old_cwd = os.getcwd()
         self.tempdirs = []
 
     def tearDown(self):
+        # Restore working dir, for Solaris and derivatives, where rmdir()
+        # on the current directory fails.
+        os.chdir(self.old_cwd)
         super().tearDown()
         while self.tempdirs:
             d = self.tempdirs.pop()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:47 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:47 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Added_tag_v3=2E?=
 =?utf8?q?2=2E2_for_changeset_137e45f15c0b?=
Message-ID: <E1R06Pn-0002b8-1d@dinsdale.python.org>

http://hg.python.org/cpython/rev/7fbcde1b5229
changeset:   72264:7fbcde1b5229
branch:      3.2
user:        Georg Brandl <georg at python.org>
date:        Sat Sep 03 18:08:11 2011 +0200
summary:
  Added tag v3.2.2 for changeset 137e45f15c0b

files:
  .hgtags |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/.hgtags b/.hgtags
--- a/.hgtags
+++ b/.hgtags
@@ -92,3 +92,4 @@
 5df549718fb4841ff521fe051f6b54f290fad5d8 v3.2.1rc2
 ac1f7e5c05104d557d5acd922e95625ba5d1fe10 v3.2.1
 c860feaa348d663e598986894ee4680480577e15 v3.2.2rc1
+137e45f15c0bd262c9ad4c032d97425bc0589456 v3.2.2

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:47 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:47 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMik6?=
 =?utf8?q?_Merge_with_release_clone=2E?=
Message-ID: <E1R06Pn-0002bC-Sn@dinsdale.python.org>

http://hg.python.org/cpython/rev/098b0a043d54
changeset:   72265:098b0a043d54
branch:      3.2
parent:      72218:2ecaf6aa3fee
parent:      72264:7fbcde1b5229
user:        Georg Brandl <georg at python.org>
date:        Sun Sep 04 08:35:54 2011 +0200
summary:
  Merge with release clone.

files:
  .hgtags                              |    2 +
  Doc/library/sys.rst                  |   40 +++--
  Doc/license.rst                      |   10 +-
  Doc/tools/sphinxext/susp-ignored.csv |   16 +-
  Include/patchlevel.h                 |    4 +-
  LICENSE                              |    6 +-
  Lib/distutils/__init__.py            |    2 +-
  Lib/idlelib/idlever.py               |    2 +-
  Lib/pydoc_data/topics.py             |   12 +-
  Misc/NEWS                            |  109 ++++++++------
  Misc/RPM/python-3.2.spec             |    2 +-
  README                               |    2 +-
  12 files changed, 124 insertions(+), 83 deletions(-)


diff --git a/.hgtags b/.hgtags
--- a/.hgtags
+++ b/.hgtags
@@ -91,3 +91,5 @@
 cfa9364997c7f2e67b9cbb45c3a5fa3bba4e4999 v3.2.1rc1
 5df549718fb4841ff521fe051f6b54f290fad5d8 v3.2.1rc2
 ac1f7e5c05104d557d5acd922e95625ba5d1fe10 v3.2.1
+c860feaa348d663e598986894ee4680480577e15 v3.2.2rc1
+137e45f15c0bd262c9ad4c032d97425bc0589456 v3.2.2
diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -699,26 +699,36 @@
    This string contains a platform identifier that can be used to append
    platform-specific components to :data:`sys.path`, for instance.
 
-   For Unix systems, this is the lowercased OS name as returned by ``uname -s``
-   with the first part of the version as returned by ``uname -r`` appended,
-   e.g. ``'sunos5'`` or ``'linux2'``, *at the time when Python was built*.
-   Unless you want to test for a specific system version, it is therefore
-   recommended to use the following idiom::
+   For most Unix systems, this is the lowercased OS name as returned by ``uname
+   -s`` with the first part of the version as returned by ``uname -r`` appended,
+   e.g. ``'sunos5'``, *at the time when Python was built*.  Unless you want to
+   test for a specific system version, it is therefore recommended to use the
+   following idiom::
 
-      if sys.platform.startswith('linux'):
+      if sys.platform.startswith('freebsd'):
+          # FreeBSD-specific code here...
+      elif sys.platform.startswith('linux'):
           # Linux-specific code here...
 
+   .. versionchanged:: 3.2.2
+      Since lots of code check for ``sys.platform == 'linux2'``, and there is
+      no essential change between Linux 2.x and 3.x, ``sys.platform`` is always
+      set to ``'linux2'``, even on Linux 3.x.  In Python 3.3 and later, the
+      value will always be set to ``'linux'``, so it is recommended to always
+      use the ``startswith`` idiom presented above.
+
    For other systems, the values are:
 
-   ================ ===========================
-   System           :data:`platform` value
-   ================ ===========================
-   Windows          ``'win32'``
-   Windows/Cygwin   ``'cygwin'``
-   Mac OS X         ``'darwin'``
-   OS/2             ``'os2'``
-   OS/2 EMX         ``'os2emx'``
-   ================ ===========================
+   ====================== ===========================
+   System                 :data:`platform` value
+   ====================== ===========================
+   Linux (2.x *and* 3.x)  ``'linux2'``
+   Windows                ``'win32'``
+   Windows/Cygwin         ``'cygwin'``
+   Mac OS X               ``'darwin'``
+   OS/2                   ``'os2'``
+   OS/2 EMX               ``'os2emx'``
+   ====================== ===========================
 
    .. seealso::
       :attr:`os.name` has a coarser granularity.  :func:`os.uname` gives
diff --git a/Doc/license.rst b/Doc/license.rst
--- a/Doc/license.rst
+++ b/Doc/license.rst
@@ -106,10 +106,18 @@
 +----------------+--------------+------------+------------+-----------------+
 | 3.1.1          | 3.1          | 2009       | PSF        | yes             |
 +----------------+--------------+------------+------------+-----------------+
-| 3.1.2          | 3.1          | 2010       | PSF        | yes             |
+| 3.1.2          | 3.1.1        | 2010       | PSF        | yes             |
++----------------+--------------+------------+------------+-----------------+
+| 3.1.3          | 3.1.2        | 2010       | PSF        | yes             |
++----------------+--------------+------------+------------+-----------------+
+| 3.1.4          | 3.1.3        | 2011       | PSF        | yes             |
 +----------------+--------------+------------+------------+-----------------+
 | 3.2            | 3.1          | 2011       | PSF        | yes             |
 +----------------+--------------+------------+------------+-----------------+
+| 3.2.1          | 3.2          | 2011       | PSF        | yes             |
++----------------+--------------+------------+------------+-----------------+
+| 3.2.2          | 3.2.1        | 2011       | PSF        | yes             |
++----------------+--------------+------------+------------+-----------------+
 
 .. note::
 
diff --git a/Doc/tools/sphinxext/susp-ignored.csv b/Doc/tools/sphinxext/susp-ignored.csv
--- a/Doc/tools/sphinxext/susp-ignored.csv
+++ b/Doc/tools/sphinxext/susp-ignored.csv
@@ -193,10 +193,10 @@
 documenting/rest,187,.. function:,.. function:: foo(x)
 documenting/rest,187,:bar,:bar: no
 documenting/rest,208,.. rubric:,.. rubric:: Footnotes
-faq/programming,762,:reduce,"print((lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+y,map(lambda y,"
-faq/programming,762,:reduce,"Sx=Sx,Sy=Sy:reduce(lambda x,y:x+y,map(lambda x,xc=Ru,yc=yc,Ru=Ru,Ro=Ro,"
-faq/programming,762,:chr,">=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr("
-faq/programming,1047,::,for x in sequence[::-1]:
+faq/programming,,:reduce,"print((lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+y,map(lambda y,"
+faq/programming,,:reduce,"Sx=Sx,Sy=Sy:reduce(lambda x,y:x+y,map(lambda x,xc=Ru,yc=yc,Ru=Ru,Ro=Ro,"
+faq/programming,,:chr,">=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr("
+faq/programming,,::,for x in sequence[::-1]:
 faq/windows,229,:EOF, at setlocal enableextensions & python -x %~f0 %* & goto :EOF
 faq/windows,393,:REG,.py :REG_SZ: c:\<path to python>\python.exe -u %s %s
 library/bisect,32,:hi,all(val >= x for val in a[i:hi])
@@ -217,10 +217,10 @@
 library/xmlrpc.client,103,:pass,http://user:pass at host:port/path
 library/xmlrpc.client,103,:port,http://user:pass at host:port/path
 library/xmlrpc.client,103,:pass,user:pass
-license,717,`,* THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
-license,717,`,* THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND
-license,879,`,"``Software''), to deal in the Software without restriction, including"
-license,879,`,"THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND,"
+license,,`,* THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
+license,,`,* THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND
+license,,`,"``Software''), to deal in the Software without restriction, including"
+license,,`,"THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND,"
 reference/lexical_analysis,704,`,$       ?       `
 whatsnew/2.7,735,:Sunday,'2009:4:Sunday'
 whatsnew/2.7,862,::,"export PYTHONWARNINGS=all,error:::Cookie:0"
diff --git a/Include/patchlevel.h b/Include/patchlevel.h
--- a/Include/patchlevel.h
+++ b/Include/patchlevel.h
@@ -18,12 +18,12 @@
 /*--start constants--*/
 #define PY_MAJOR_VERSION	3
 #define PY_MINOR_VERSION	2
-#define PY_MICRO_VERSION	1
+#define PY_MICRO_VERSION	2
 #define PY_RELEASE_LEVEL	PY_RELEASE_LEVEL_FINAL
 #define PY_RELEASE_SERIAL	0
 
 /* Version as a string */
-#define PY_VERSION      	"3.2.1+"
+#define PY_VERSION      	"3.2.2"
 /*--end constants--*/
 
 /* Subversion Revision number of this file (not of the repository). Empty
diff --git a/LICENSE b/LICENSE
--- a/LICENSE
+++ b/LICENSE
@@ -67,8 +67,12 @@
     3.0.1           3.0         2009        PSF         yes
     3.1             3.0.1       2009        PSF         yes
     3.1.1           3.1         2009        PSF         yes
-    3.1.2           3.1         2010        PSF         yes
+    3.1.2           3.1.1       2010        PSF         yes
+    3.1.3           3.1.2       2010        PSF         yes
+    3.1.4           3.1.3       2011        PSF         yes
     3.2             3.1         2011        PSF         yes
+    3.2.1           3.2         2011        PSF         yes
+    3.2.2           3.2.1       2011        PSF         yes
 
 Footnotes:
 
diff --git a/Lib/distutils/__init__.py b/Lib/distutils/__init__.py
--- a/Lib/distutils/__init__.py
+++ b/Lib/distutils/__init__.py
@@ -13,5 +13,5 @@
 # Updated automatically by the Python release process.
 #
 #--start constants--
-__version__ = "3.2.1"
+__version__ = "3.2.2"
 #--end constants--
diff --git a/Lib/idlelib/idlever.py b/Lib/idlelib/idlever.py
--- a/Lib/idlelib/idlever.py
+++ b/Lib/idlelib/idlever.py
@@ -1,1 +1,1 @@
-IDLE_VERSION = "3.2.1"
+IDLE_VERSION = "3.2.2"
diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py
--- a/Lib/pydoc_data/topics.py
+++ b/Lib/pydoc_data/topics.py
@@ -1,4 +1,4 @@
-# Autogenerated by Sphinx on Sun Jul  3 09:27:35 2011
+# Autogenerated by Sphinx on Sat Sep  3 10:35:42 2011
 topics = {'assert': '\nThe ``assert`` statement\n************************\n\nAssert statements are a convenient way to insert debugging assertions\ninto a program:\n\n   assert_stmt ::= "assert" expression ["," expression]\n\nThe simple form, ``assert expression``, is equivalent to\n\n   if __debug__:\n      if not expression: raise AssertionError\n\nThe extended form, ``assert expression1, expression2``, is equivalent\nto\n\n   if __debug__:\n      if not expression1: raise AssertionError(expression2)\n\nThese equivalences assume that ``__debug__`` and ``AssertionError``\nrefer to the built-in variables with those names.  In the current\nimplementation, the built-in variable ``__debug__`` is ``True`` under\nnormal circumstances, ``False`` when optimization is requested\n(command line option -O).  The current code generator emits no code\nfor an assert statement when optimization is requested at compile\ntime.  Note that it is unnecessary to include the source code for the\nexpression that failed in the error message; it will be displayed as\npart of the stack trace.\n\nAssignments to ``__debug__`` are illegal.  The value for the built-in\nvariable is determined when the interpreter starts.\n',
  'assignment': '\nAssignment statements\n*********************\n\nAssignment statements are used to (re)bind names to values and to\nmodify attributes or items of mutable objects:\n\n   assignment_stmt ::= (target_list "=")+ (expression_list | yield_expression)\n   target_list     ::= target ("," target)* [","]\n   target          ::= identifier\n              | "(" target_list ")"\n              | "[" target_list "]"\n              | attributeref\n              | subscription\n              | slicing\n              | "*" target\n\n(See section *Primaries* for the syntax definitions for the last three\nsymbols.)\n\nAn assignment statement evaluates the expression list (remember that\nthis can be a single expression or a comma-separated list, the latter\nyielding a tuple) and assigns the single resulting object to each of\nthe target lists, from left to right.\n\nAssignment is defined recursively depending on the form of the target\n(list). When a target is part of a mutable object (an attribute\nreference, subscription or slicing), the mutable object must\nultimately perform the assignment and decide about its validity, and\nmay raise an exception if the assignment is unacceptable.  The rules\nobserved by various types and the exceptions raised are given with the\ndefinition of the object types (see section *The standard type\nhierarchy*).\n\nAssignment of an object to a target list, optionally enclosed in\nparentheses or square brackets, is recursively defined as follows.\n\n* If the target list is a single target: The object is assigned to\n  that target.\n\n* If the target list is a comma-separated list of targets: The object\n  must be an iterable with the same number of items as there are\n  targets in the target list, and the items are assigned, from left to\n  right, to the corresponding targets.\n\n  * If the target list contains one target prefixed with an asterisk,\n    called a "starred" target: The object must be a sequence with at\n    least as many items as there are targets in the target list, minus\n    one.  The first items of the sequence are assigned, from left to\n    right, to the targets before the starred target.  The final items\n    of the sequence are assigned to the targets after the starred\n    target.  A list of the remaining items in the sequence is then\n    assigned to the starred target (the list can be empty).\n\n  * Else: The object must be a sequence with the same number of items\n    as there are targets in the target list, and the items are\n    assigned, from left to right, to the corresponding targets.\n\nAssignment of an object to a single target is recursively defined as\nfollows.\n\n* If the target is an identifier (name):\n\n  * If the name does not occur in a ``global`` or ``nonlocal``\n    statement in the current code block: the name is bound to the\n    object in the current local namespace.\n\n  * Otherwise: the name is bound to the object in the global namespace\n    or the outer namespace determined by ``nonlocal``, respectively.\n\n  The name is rebound if it was already bound.  This may cause the\n  reference count for the object previously bound to the name to reach\n  zero, causing the object to be deallocated and its destructor (if it\n  has one) to be called.\n\n* If the target is a target list enclosed in parentheses or in square\n  brackets: The object must be an iterable with the same number of\n  items as there are targets in the target list, and its items are\n  assigned, from left to right, to the corresponding targets.\n\n* If the target is an attribute reference: The primary expression in\n  the reference is evaluated.  It should yield an object with\n  assignable attributes; if this is not the case, ``TypeError`` is\n  raised.  That object is then asked to assign the assigned object to\n  the given attribute; if it cannot perform the assignment, it raises\n  an exception (usually but not necessarily ``AttributeError``).\n\n  Note: If the object is a class instance and the attribute reference\n  occurs on both sides of the assignment operator, the RHS expression,\n  ``a.x`` can access either an instance attribute or (if no instance\n  attribute exists) a class attribute.  The LHS target ``a.x`` is\n  always set as an instance attribute, creating it if necessary.\n  Thus, the two occurrences of ``a.x`` do not necessarily refer to the\n  same attribute: if the RHS expression refers to a class attribute,\n  the LHS creates a new instance attribute as the target of the\n  assignment:\n\n     class Cls:\n         x = 3             # class variable\n     inst = Cls()\n     inst.x = inst.x + 1   # writes inst.x as 4 leaving Cls.x as 3\n\n  This description does not necessarily apply to descriptor\n  attributes, such as properties created with ``property()``.\n\n* If the target is a subscription: The primary expression in the\n  reference is evaluated.  It should yield either a mutable sequence\n  object (such as a list) or a mapping object (such as a dictionary).\n  Next, the subscript expression is evaluated.\n\n  If the primary is a mutable sequence object (such as a list), the\n  subscript must yield an integer.  If it is negative, the sequence\'s\n  length is added to it.  The resulting value must be a nonnegative\n  integer less than the sequence\'s length, and the sequence is asked\n  to assign the assigned object to its item with that index.  If the\n  index is out of range, ``IndexError`` is raised (assignment to a\n  subscripted sequence cannot add new items to a list).\n\n  If the primary is a mapping object (such as a dictionary), the\n  subscript must have a type compatible with the mapping\'s key type,\n  and the mapping is then asked to create a key/datum pair which maps\n  the subscript to the assigned object.  This can either replace an\n  existing key/value pair with the same key value, or insert a new\n  key/value pair (if no key with the same value existed).\n\n  For user-defined objects, the ``__setitem__()`` method is called\n  with appropriate arguments.\n\n* If the target is a slicing: The primary expression in the reference\n  is evaluated.  It should yield a mutable sequence object (such as a\n  list).  The assigned object should be a sequence object of the same\n  type.  Next, the lower and upper bound expressions are evaluated,\n  insofar they are present; defaults are zero and the sequence\'s\n  length.  The bounds should evaluate to integers. If either bound is\n  negative, the sequence\'s length is added to it.  The resulting\n  bounds are clipped to lie between zero and the sequence\'s length,\n  inclusive.  Finally, the sequence object is asked to replace the\n  slice with the items of the assigned sequence.  The length of the\n  slice may be different from the length of the assigned sequence,\n  thus changing the length of the target sequence, if the object\n  allows it.\n\n**CPython implementation detail:** In the current implementation, the\nsyntax for targets is taken to be the same as for expressions, and\ninvalid syntax is rejected during the code generation phase, causing\nless detailed error messages.\n\nWARNING: Although the definition of assignment implies that overlaps\nbetween the left-hand side and the right-hand side are \'safe\' (for\nexample ``a, b = b, a`` swaps two variables), overlaps *within* the\ncollection of assigned-to variables are not safe!  For instance, the\nfollowing program prints ``[0, 2]``:\n\n   x = [0, 1]\n   i = 0\n   i, x[i] = 1, 2\n   print(x)\n\nSee also:\n\n   **PEP 3132** - Extended Iterable Unpacking\n      The specification for the ``*target`` feature.\n\n\nAugmented assignment statements\n===============================\n\nAugmented assignment is the combination, in a single statement, of a\nbinary operation and an assignment statement:\n\n   augmented_assignment_stmt ::= augtarget augop (expression_list | yield_expression)\n   augtarget                 ::= identifier | attributeref | subscription | slicing\n   augop                     ::= "+=" | "-=" | "*=" | "/=" | "//=" | "%=" | "**="\n             | ">>=" | "<<=" | "&=" | "^=" | "|="\n\n(See section *Primaries* for the syntax definitions for the last three\nsymbols.)\n\nAn augmented assignment evaluates the target (which, unlike normal\nassignment statements, cannot be an unpacking) and the expression\nlist, performs the binary operation specific to the type of assignment\non the two operands, and assigns the result to the original target.\nThe target is only evaluated once.\n\nAn augmented assignment expression like ``x += 1`` can be rewritten as\n``x = x + 1`` to achieve a similar, but not exactly equal effect. In\nthe augmented version, ``x`` is only evaluated once. Also, when\npossible, the actual operation is performed *in-place*, meaning that\nrather than creating a new object and assigning that to the target,\nthe old object is modified instead.\n\nWith the exception of assigning to tuples and multiple targets in a\nsingle statement, the assignment done by augmented assignment\nstatements is handled the same way as normal assignments. Similarly,\nwith the exception of the possible *in-place* behavior, the binary\noperation performed by augmented assignment is the same as the normal\nbinary operations.\n\nFor targets which are attribute references, the same *caveat about\nclass and instance attributes* applies as for regular assignments.\n',
  'atom-identifiers': '\nIdentifiers (Names)\n*******************\n\nAn identifier occurring as an atom is a name.  See section\n*Identifiers and keywords* for lexical definition and section *Naming\nand binding* for documentation of naming and binding.\n\nWhen the name is bound to an object, evaluation of the atom yields\nthat object. When a name is not bound, an attempt to evaluate it\nraises a ``NameError`` exception.\n\n**Private name mangling:** When an identifier that textually occurs in\na class definition begins with two or more underscore characters and\ndoes not end in two or more underscores, it is considered a *private\nname* of that class. Private names are transformed to a longer form\nbefore code is generated for them.  The transformation inserts the\nclass name in front of the name, with leading underscores removed, and\na single underscore inserted in front of the class name.  For example,\nthe identifier ``__spam`` occurring in a class named ``Ham`` will be\ntransformed to ``_Ham__spam``.  This transformation is independent of\nthe syntactical context in which the identifier is used.  If the\ntransformed name is extremely long (longer than 255 characters),\nimplementation defined truncation may happen.  If the class name\nconsists only of underscores, no transformation is done.\n',
@@ -15,7 +15,7 @@
  'booleans': '\nBoolean operations\n******************\n\n   or_test  ::= and_test | or_test "or" and_test\n   and_test ::= not_test | and_test "and" not_test\n   not_test ::= comparison | "not" not_test\n\nIn the context of Boolean operations, and also when expressions are\nused by control flow statements, the following values are interpreted\nas false: ``False``, ``None``, numeric zero of all types, and empty\nstrings and containers (including strings, tuples, lists,\ndictionaries, sets and frozensets).  All other values are interpreted\nas true.  User-defined objects can customize their truth value by\nproviding a ``__bool__()`` method.\n\nThe operator ``not`` yields ``True`` if its argument is false,\n``False`` otherwise.\n\nThe expression ``x and y`` first evaluates *x*; if *x* is false, its\nvalue is returned; otherwise, *y* is evaluated and the resulting value\nis returned.\n\nThe expression ``x or y`` first evaluates *x*; if *x* is true, its\nvalue is returned; otherwise, *y* is evaluated and the resulting value\nis returned.\n\n(Note that neither ``and`` nor ``or`` restrict the value and type they\nreturn to ``False`` and ``True``, but rather return the last evaluated\nargument.  This is sometimes useful, e.g., if ``s`` is a string that\nshould be replaced by a default value if it is empty, the expression\n``s or \'foo\'`` yields the desired value.  Because ``not`` has to\ninvent a value anyway, it does not bother to return a value of the\nsame type as its argument, so e.g., ``not \'foo\'`` yields ``False``,\nnot ``\'\'``.)\n',
  'break': '\nThe ``break`` statement\n***********************\n\n   break_stmt ::= "break"\n\n``break`` may only occur syntactically nested in a ``for`` or\n``while`` loop, but not nested in a function or class definition\nwithin that loop.\n\nIt terminates the nearest enclosing loop, skipping the optional\n``else`` clause if the loop has one.\n\nIf a ``for`` loop is terminated by ``break``, the loop control target\nkeeps its current value.\n\nWhen ``break`` passes control out of a ``try`` statement with a\n``finally`` clause, that ``finally`` clause is executed before really\nleaving the loop.\n',
  'callable-types': '\nEmulating callable objects\n**************************\n\nobject.__call__(self[, args...])\n\n   Called when the instance is "called" as a function; if this method\n   is defined, ``x(arg1, arg2, ...)`` is a shorthand for\n   ``x.__call__(arg1, arg2, ...)``.\n',
- 'calls': '\nCalls\n*****\n\nA call calls a callable object (e.g., a function) with a possibly\nempty series of arguments:\n\n   call                 ::= primary "(" [argument_list [","] | comprehension] ")"\n   argument_list        ::= positional_arguments ["," keyword_arguments]\n                       ["," "*" expression] ["," keyword_arguments]\n                       ["," "**" expression]\n                     | keyword_arguments ["," "*" expression]\n                       ["," keyword_arguments] ["," "**" expression]\n                     | "*" expression ["," keyword_arguments] ["," "**" expression]\n                     | "**" expression\n   positional_arguments ::= expression ("," expression)*\n   keyword_arguments    ::= keyword_item ("," keyword_item)*\n   keyword_item         ::= identifier "=" expression\n\nA trailing comma may be present after the positional and keyword\narguments but does not affect the semantics.\n\nThe primary must evaluate to a callable object (user-defined\nfunctions, built-in functions, methods of built-in objects, class\nobjects, methods of class instances, and all objects having a\n``__call__()`` method are callable).  All argument expressions are\nevaluated before the call is attempted.  Please refer to section\n*Function definitions* for the syntax of formal parameter lists.\n\nIf keyword arguments are present, they are first converted to\npositional arguments, as follows.  First, a list of unfilled slots is\ncreated for the formal parameters.  If there are N positional\narguments, they are placed in the first N slots.  Next, for each\nkeyword argument, the identifier is used to determine the\ncorresponding slot (if the identifier is the same as the first formal\nparameter name, the first slot is used, and so on).  If the slot is\nalready filled, a ``TypeError`` exception is raised. Otherwise, the\nvalue of the argument is placed in the slot, filling it (even if the\nexpression is ``None``, it fills the slot).  When all arguments have\nbeen processed, the slots that are still unfilled are filled with the\ncorresponding default value from the function definition.  (Default\nvalues are calculated, once, when the function is defined; thus, a\nmutable object such as a list or dictionary used as default value will\nbe shared by all calls that don\'t specify an argument value for the\ncorresponding slot; this should usually be avoided.)  If there are any\nunfilled slots for which no default value is specified, a\n``TypeError`` exception is raised.  Otherwise, the list of filled\nslots is used as the argument list for the call.\n\n**CPython implementation detail:** An implementation may provide\nbuilt-in functions whose positional parameters do not have names, even\nif they are \'named\' for the purpose of documentation, and which\ntherefore cannot be supplied by keyword.  In CPython, this is the case\nfor functions implemented in C that use ``PyArg_ParseTuple()`` to\nparse their arguments.\n\nIf there are more positional arguments than there are formal parameter\nslots, a ``TypeError`` exception is raised, unless a formal parameter\nusing the syntax ``*identifier`` is present; in this case, that formal\nparameter receives a tuple containing the excess positional arguments\n(or an empty tuple if there were no excess positional arguments).\n\nIf any keyword argument does not correspond to a formal parameter\nname, a ``TypeError`` exception is raised, unless a formal parameter\nusing the syntax ``**identifier`` is present; in this case, that\nformal parameter receives a dictionary containing the excess keyword\narguments (using the keywords as keys and the argument values as\ncorresponding values), or a (new) empty dictionary if there were no\nexcess keyword arguments.\n\nIf the syntax ``*expression`` appears in the function call,\n``expression`` must evaluate to a sequence.  Elements from this\nsequence are treated as if they were additional positional arguments;\nif there are positional arguments *x1*,..., *xN*, and ``expression``\nevaluates to a sequence *y1*, ..., *yM*, this is equivalent to a call\nwith M+N positional arguments *x1*, ..., *xN*, *y1*, ..., *yM*.\n\nA consequence of this is that although the ``*expression`` syntax may\nappear *after* some keyword arguments, it is processed *before* the\nkeyword arguments (and the ``**expression`` argument, if any -- see\nbelow).  So:\n\n   >>> def f(a, b):\n   ...  print(a, b)\n   ...\n   >>> f(b=1, *(2,))\n   2 1\n   >>> f(a=1, *(2,))\n   Traceback (most recent call last):\n     File "<stdin>", line 1, in ?\n   TypeError: f() got multiple values for keyword argument \'a\'\n   >>> f(1, *(2,))\n   1 2\n\nIt is unusual for both keyword arguments and the ``*expression``\nsyntax to be used in the same call, so in practice this confusion does\nnot arise.\n\nIf the syntax ``**expression`` appears in the function call,\n``expression`` must evaluate to a mapping, the contents of which are\ntreated as additional keyword arguments.  In the case of a keyword\nappearing in both ``expression`` and as an explicit keyword argument,\na ``TypeError`` exception is raised.\n\nFormal parameters using the syntax ``*identifier`` or ``**identifier``\ncannot be used as positional argument slots or as keyword argument\nnames.\n\nA call always returns some value, possibly ``None``, unless it raises\nan exception.  How this value is computed depends on the type of the\ncallable object.\n\nIf it is---\n\na user-defined function:\n   The code block for the function is executed, passing it the\n   argument list.  The first thing the code block will do is bind the\n   formal parameters to the arguments; this is described in section\n   *Function definitions*.  When the code block executes a ``return``\n   statement, this specifies the return value of the function call.\n\na built-in function or method:\n   The result is up to the interpreter; see *Built-in Functions* for\n   the descriptions of built-in functions and methods.\n\na class object:\n   A new instance of that class is returned.\n\na class instance method:\n   The corresponding user-defined function is called, with an argument\n   list that is one longer than the argument list of the call: the\n   instance becomes the first argument.\n\na class instance:\n   The class must define a ``__call__()`` method; the effect is then\n   the same as if that method was called.\n',
+ 'calls': '\nCalls\n*****\n\nA call calls a callable object (e.g., a function) with a possibly\nempty series of arguments:\n\n   call                 ::= primary "(" [argument_list [","] | comprehension] ")"\n   argument_list        ::= positional_arguments ["," keyword_arguments]\n                       ["," "*" expression] ["," keyword_arguments]\n                       ["," "**" expression]\n                     | keyword_arguments ["," "*" expression]\n                       ["," keyword_arguments] ["," "**" expression]\n                     | "*" expression ["," keyword_arguments] ["," "**" expression]\n                     | "**" expression\n   positional_arguments ::= expression ("," expression)*\n   keyword_arguments    ::= keyword_item ("," keyword_item)*\n   keyword_item         ::= identifier "=" expression\n\nA trailing comma may be present after the positional and keyword\narguments but does not affect the semantics.\n\nThe primary must evaluate to a callable object (user-defined\nfunctions, built-in functions, methods of built-in objects, class\nobjects, methods of class instances, and all objects having a\n``__call__()`` method are callable).  All argument expressions are\nevaluated before the call is attempted.  Please refer to section\n*Function definitions* for the syntax of formal parameter lists.\n\nIf keyword arguments are present, they are first converted to\npositional arguments, as follows.  First, a list of unfilled slots is\ncreated for the formal parameters.  If there are N positional\narguments, they are placed in the first N slots.  Next, for each\nkeyword argument, the identifier is used to determine the\ncorresponding slot (if the identifier is the same as the first formal\nparameter name, the first slot is used, and so on).  If the slot is\nalready filled, a ``TypeError`` exception is raised. Otherwise, the\nvalue of the argument is placed in the slot, filling it (even if the\nexpression is ``None``, it fills the slot).  When all arguments have\nbeen processed, the slots that are still unfilled are filled with the\ncorresponding default value from the function definition.  (Default\nvalues are calculated, once, when the function is defined; thus, a\nmutable object such as a list or dictionary used as default value will\nbe shared by all calls that don\'t specify an argument value for the\ncorresponding slot; this should usually be avoided.)  If there are any\nunfilled slots for which no default value is specified, a\n``TypeError`` exception is raised.  Otherwise, the list of filled\nslots is used as the argument list for the call.\n\n**CPython implementation detail:** An implementation may provide\nbuilt-in functions whose positional parameters do not have names, even\nif they are \'named\' for the purpose of documentation, and which\ntherefore cannot be supplied by keyword.  In CPython, this is the case\nfor functions implemented in C that use ``PyArg_ParseTuple()`` to\nparse their arguments.\n\nIf there are more positional arguments than there are formal parameter\nslots, a ``TypeError`` exception is raised, unless a formal parameter\nusing the syntax ``*identifier`` is present; in this case, that formal\nparameter receives a tuple containing the excess positional arguments\n(or an empty tuple if there were no excess positional arguments).\n\nIf any keyword argument does not correspond to a formal parameter\nname, a ``TypeError`` exception is raised, unless a formal parameter\nusing the syntax ``**identifier`` is present; in this case, that\nformal parameter receives a dictionary containing the excess keyword\narguments (using the keywords as keys and the argument values as\ncorresponding values), or a (new) empty dictionary if there were no\nexcess keyword arguments.\n\nIf the syntax ``*expression`` appears in the function call,\n``expression`` must evaluate to an iterable.  Elements from this\niterable are treated as if they were additional positional arguments;\nif there are positional arguments *x1*, ..., *xN*, and ``expression``\nevaluates to a sequence *y1*, ..., *yM*, this is equivalent to a call\nwith M+N positional arguments *x1*, ..., *xN*, *y1*, ..., *yM*.\n\nA consequence of this is that although the ``*expression`` syntax may\nappear *after* some keyword arguments, it is processed *before* the\nkeyword arguments (and the ``**expression`` argument, if any -- see\nbelow).  So:\n\n   >>> def f(a, b):\n   ...  print(a, b)\n   ...\n   >>> f(b=1, *(2,))\n   2 1\n   >>> f(a=1, *(2,))\n   Traceback (most recent call last):\n     File "<stdin>", line 1, in ?\n   TypeError: f() got multiple values for keyword argument \'a\'\n   >>> f(1, *(2,))\n   1 2\n\nIt is unusual for both keyword arguments and the ``*expression``\nsyntax to be used in the same call, so in practice this confusion does\nnot arise.\n\nIf the syntax ``**expression`` appears in the function call,\n``expression`` must evaluate to a mapping, the contents of which are\ntreated as additional keyword arguments.  In the case of a keyword\nappearing in both ``expression`` and as an explicit keyword argument,\na ``TypeError`` exception is raised.\n\nFormal parameters using the syntax ``*identifier`` or ``**identifier``\ncannot be used as positional argument slots or as keyword argument\nnames.\n\nA call always returns some value, possibly ``None``, unless it raises\nan exception.  How this value is computed depends on the type of the\ncallable object.\n\nIf it is---\n\na user-defined function:\n   The code block for the function is executed, passing it the\n   argument list.  The first thing the code block will do is bind the\n   formal parameters to the arguments; this is described in section\n   *Function definitions*.  When the code block executes a ``return``\n   statement, this specifies the return value of the function call.\n\na built-in function or method:\n   The result is up to the interpreter; see *Built-in Functions* for\n   the descriptions of built-in functions and methods.\n\na class object:\n   A new instance of that class is returned.\n\na class instance method:\n   The corresponding user-defined function is called, with an argument\n   list that is one longer than the argument list of the call: the\n   instance becomes the first argument.\n\na class instance:\n   The class must define a ``__call__()`` method; the effect is then\n   the same as if that method was called.\n',
  'class': '\nClass definitions\n*****************\n\nA class definition defines a class object (see section *The standard\ntype hierarchy*):\n\n   classdef    ::= [decorators] "class" classname [inheritance] ":" suite\n   inheritance ::= "(" [argument_list [","] | comprehension] ")"\n   classname   ::= identifier\n\nA class definition is an executable statement.  The inheritance list\nusually gives a list of base classes (see *Customizing class creation*\nfor more advanced uses), so each item in the list should evaluate to a\nclass object which allows subclassing.  Classes without an inheritance\nlist inherit, by default, from the base class ``object``; hence,\n\n   class Foo:\n       pass\n\nis equivalent to\n\n   class Foo(object):\n       pass\n\nThe class\'s suite is then executed in a new execution frame (see\n*Naming and binding*), using a newly created local namespace and the\noriginal global namespace. (Usually, the suite contains mostly\nfunction definitions.)  When the class\'s suite finishes execution, its\nexecution frame is discarded but its local namespace is saved. [4] A\nclass object is then created using the inheritance list for the base\nclasses and the saved local namespace for the attribute dictionary.\nThe class name is bound to this class object in the original local\nnamespace.\n\nClass creation can be customized heavily using *metaclasses*.\n\nClasses can also be decorated: just like when decorating functions,\n\n   @f1(arg)\n   @f2\n   class Foo: pass\n\nis equivalent to\n\n   class Foo: pass\n   Foo = f1(arg)(f2(Foo))\n\nThe evaluation rules for the decorator expressions are the same as for\nfunction decorators.  The result must be a class object, which is then\nbound to the class name.\n\n**Programmer\'s note:** Variables defined in the class definition are\nclass attributes; they are shared by instances.  Instance attributes\ncan be set in a method with ``self.name = value``.  Both class and\ninstance attributes are accessible through the notation\n"``self.name``", and an instance attribute hides a class attribute\nwith the same name when accessed in this way.  Class attributes can be\nused as defaults for instance attributes, but using mutable values\nthere can lead to unexpected results.  *Descriptors* can be used to\ncreate instance variables with different implementation details.\n\nSee also:\n\n   **PEP 3115** - Metaclasses in Python 3 **PEP 3129** - Class\n   Decorators\n\n-[ Footnotes ]-\n\n[1] The exception is propagated to the invocation stack unless there\n    is a ``finally`` clause which happens to raise another exception.\n    That new exception causes the old one to be lost.\n\n[2] Currently, control "flows off the end" except in the case of an\n    exception or the execution of a ``return``, ``continue``, or\n    ``break`` statement.\n\n[3] A string literal appearing as the first statement in the function\n    body is transformed into the function\'s ``__doc__`` attribute and\n    therefore the function\'s *docstring*.\n\n[4] A string literal appearing as the first statement in the class\n    body is transformed into the namespace\'s ``__doc__`` item and\n    therefore the class\'s *docstring*.\n',
  'comparisons': '\nComparisons\n***********\n\nUnlike C, all comparison operations in Python have the same priority,\nwhich is lower than that of any arithmetic, shifting or bitwise\noperation.  Also unlike C, expressions like ``a < b < c`` have the\ninterpretation that is conventional in mathematics:\n\n   comparison    ::= or_expr ( comp_operator or_expr )*\n   comp_operator ::= "<" | ">" | "==" | ">=" | "<=" | "!="\n                     | "is" ["not"] | ["not"] "in"\n\nComparisons yield boolean values: ``True`` or ``False``.\n\nComparisons can be chained arbitrarily, e.g., ``x < y <= z`` is\nequivalent to ``x < y and y <= z``, except that ``y`` is evaluated\nonly once (but in both cases ``z`` is not evaluated at all when ``x <\ny`` is found to be false).\n\nFormally, if *a*, *b*, *c*, ..., *y*, *z* are expressions and *op1*,\n*op2*, ..., *opN* are comparison operators, then ``a op1 b op2 c ... y\nopN z`` is equivalent to ``a op1 b and b op2 c and ... y opN z``,\nexcept that each expression is evaluated at most once.\n\nNote that ``a op1 b op2 c`` doesn\'t imply any kind of comparison\nbetween *a* and *c*, so that, e.g., ``x < y > z`` is perfectly legal\n(though perhaps not pretty).\n\nThe operators ``<``, ``>``, ``==``, ``>=``, ``<=``, and ``!=`` compare\nthe values of two objects.  The objects need not have the same type.\nIf both are numbers, they are converted to a common type.  Otherwise,\nthe ``==`` and ``!=`` operators *always* consider objects of different\ntypes to be unequal, while the ``<``, ``>``, ``>=`` and ``<=``\noperators raise a ``TypeError`` when comparing objects of different\ntypes that do not implement these operators for the given pair of\ntypes.  You can control comparison behavior of objects of non-built-in\ntypes by defining rich comparison methods like ``__gt__()``, described\nin section *Basic customization*.\n\nComparison of objects of the same type depends on the type:\n\n* Numbers are compared arithmetically.\n\n* The values ``float(\'NaN\')`` and ``Decimal(\'NaN\')`` are special. The\n  are identical to themselves, ``x is x`` but are not equal to\n  themselves, ``x != x``.  Additionally, comparing any value to a\n  not-a-number value will return ``False``.  For example, both ``3 <\n  float(\'NaN\')`` and ``float(\'NaN\') < 3`` will return ``False``.\n\n* Bytes objects are compared lexicographically using the numeric\n  values of their elements.\n\n* Strings are compared lexicographically using the numeric equivalents\n  (the result of the built-in function ``ord()``) of their characters.\n  [3] String and bytes object can\'t be compared!\n\n* Tuples and lists are compared lexicographically using comparison of\n  corresponding elements.  This means that to compare equal, each\n  element must compare equal and the two sequences must be of the same\n  type and have the same length.\n\n  If not equal, the sequences are ordered the same as their first\n  differing elements.  For example, ``[1,2,x] <= [1,2,y]`` has the\n  same value as ``x <= y``.  If the corresponding element does not\n  exist, the shorter sequence is ordered first (for example, ``[1,2] <\n  [1,2,3]``).\n\n* Mappings (dictionaries) compare equal if and only if they have the\n  same ``(key, value)`` pairs. Order comparisons ``(\'<\', \'<=\', \'>=\',\n  \'>\')`` raise ``TypeError``.\n\n* Sets and frozensets define comparison operators to mean subset and\n  superset tests.  Those relations do not define total orderings (the\n  two sets ``{1,2}`` and {2,3} are not equal, nor subsets of one\n  another, nor supersets of one another).  Accordingly, sets are not\n  appropriate arguments for functions which depend on total ordering.\n  For example, ``min()``, ``max()``, and ``sorted()`` produce\n  undefined results given a list of sets as inputs.\n\n* Most other objects of built-in types compare unequal unless they are\n  the same object; the choice whether one object is considered smaller\n  or larger than another one is made arbitrarily but consistently\n  within one execution of a program.\n\nComparison of objects of the differing types depends on whether either\nof the types provide explicit support for the comparison.  Most\nnumeric types can be compared with one another, but comparisons of\n``float`` and ``Decimal`` are not supported to avoid the inevitable\nconfusion arising from representation issues such as ``float(\'1.1\')``\nbeing inexactly represented and therefore not exactly equal to\n``Decimal(\'1.1\')`` which is.  When cross-type comparison is not\nsupported, the comparison method returns ``NotImplemented``.  This can\ncreate the illusion of non-transitivity between supported cross-type\ncomparisons and unsupported comparisons.  For example, ``Decimal(2) ==\n2`` and ``2 == float(2)`` but ``Decimal(2) != float(2)``.\n\nThe operators ``in`` and ``not in`` test for membership.  ``x in s``\nevaluates to true if *x* is a member of *s*, and false otherwise.  ``x\nnot in s`` returns the negation of ``x in s``.  All built-in sequences\nand set types support this as well as dictionary, for which ``in``\ntests whether a the dictionary has a given key. For container types\nsuch as list, tuple, set, frozenset, dict, or collections.deque, the\nexpression ``x in y`` is equivalent to ``any(x is e or x == e for e in\ny)``.\n\nFor the string and bytes types, ``x in y`` is true if and only if *x*\nis a substring of *y*.  An equivalent test is ``y.find(x) != -1``.\nEmpty strings are always considered to be a substring of any other\nstring, so ``"" in "abc"`` will return ``True``.\n\nFor user-defined classes which define the ``__contains__()`` method,\n``x in y`` is true if and only if ``y.__contains__(x)`` is true.\n\nFor user-defined classes which do not define ``__contains__()`` but do\ndefine ``__iter__()``, ``x in y`` is true if some value ``z`` with ``x\n== z`` is produced while iterating over ``y``.  If an exception is\nraised during the iteration, it is as if ``in`` raised that exception.\n\nLastly, the old-style iteration protocol is tried: if a class defines\n``__getitem__()``, ``x in y`` is true if and only if there is a non-\nnegative integer index *i* such that ``x == y[i]``, and all lower\ninteger indices do not raise ``IndexError`` exception.  (If any other\nexception is raised, it is as if ``in`` raised that exception).\n\nThe operator ``not in`` is defined to have the inverse true value of\n``in``.\n\nThe operators ``is`` and ``is not`` test for object identity: ``x is\ny`` is true if and only if *x* and *y* are the same object.  ``x is\nnot y`` yields the inverse truth value. [4]\n',
  'compound': '\nCompound statements\n*******************\n\nCompound statements contain (groups of) other statements; they affect\nor control the execution of those other statements in some way.  In\ngeneral, compound statements span multiple lines, although in simple\nincarnations a whole compound statement may be contained in one line.\n\nThe ``if``, ``while`` and ``for`` statements implement traditional\ncontrol flow constructs.  ``try`` specifies exception handlers and/or\ncleanup code for a group of statements, while the ``with`` statement\nallows the execution of initialization and finalization code around a\nblock of code.  Function and class definitions are also syntactically\ncompound statements.\n\nCompound statements consist of one or more \'clauses.\'  A clause\nconsists of a header and a \'suite.\'  The clause headers of a\nparticular compound statement are all at the same indentation level.\nEach clause header begins with a uniquely identifying keyword and ends\nwith a colon.  A suite is a group of statements controlled by a\nclause.  A suite can be one or more semicolon-separated simple\nstatements on the same line as the header, following the header\'s\ncolon, or it can be one or more indented statements on subsequent\nlines.  Only the latter form of suite can contain nested compound\nstatements; the following is illegal, mostly because it wouldn\'t be\nclear to which ``if`` clause a following ``else`` clause would belong:\n\n   if test1: if test2: print(x)\n\nAlso note that the semicolon binds tighter than the colon in this\ncontext, so that in the following example, either all or none of the\n``print()`` calls are executed:\n\n   if x < y < z: print(x); print(y); print(z)\n\nSummarizing:\n\n   compound_stmt ::= if_stmt\n                     | while_stmt\n                     | for_stmt\n                     | try_stmt\n                     | with_stmt\n                     | funcdef\n                     | classdef\n   suite         ::= stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENT\n   statement     ::= stmt_list NEWLINE | compound_stmt\n   stmt_list     ::= simple_stmt (";" simple_stmt)* [";"]\n\nNote that statements always end in a ``NEWLINE`` possibly followed by\na ``DEDENT``.  Also note that optional continuation clauses always\nbegin with a keyword that cannot start a statement, thus there are no\nambiguities (the \'dangling ``else``\' problem is solved in Python by\nrequiring nested ``if`` statements to be indented).\n\nThe formatting of the grammar rules in the following sections places\neach clause on a separate line for clarity.\n\n\nThe ``if`` statement\n====================\n\nThe ``if`` statement is used for conditional execution:\n\n   if_stmt ::= "if" expression ":" suite\n               ( "elif" expression ":" suite )*\n               ["else" ":" suite]\n\nIt selects exactly one of the suites by evaluating the expressions one\nby one until one is found to be true (see section *Boolean operations*\nfor the definition of true and false); then that suite is executed\n(and no other part of the ``if`` statement is executed or evaluated).\nIf all expressions are false, the suite of the ``else`` clause, if\npresent, is executed.\n\n\nThe ``while`` statement\n=======================\n\nThe ``while`` statement is used for repeated execution as long as an\nexpression is true:\n\n   while_stmt ::= "while" expression ":" suite\n                  ["else" ":" suite]\n\nThis repeatedly tests the expression and, if it is true, executes the\nfirst suite; if the expression is false (which may be the first time\nit is tested) the suite of the ``else`` clause, if present, is\nexecuted and the loop terminates.\n\nA ``break`` statement executed in the first suite terminates the loop\nwithout executing the ``else`` clause\'s suite.  A ``continue``\nstatement executed in the first suite skips the rest of the suite and\ngoes back to testing the expression.\n\n\nThe ``for`` statement\n=====================\n\nThe ``for`` statement is used to iterate over the elements of a\nsequence (such as a string, tuple or list) or other iterable object:\n\n   for_stmt ::= "for" target_list "in" expression_list ":" suite\n                ["else" ":" suite]\n\nThe expression list is evaluated once; it should yield an iterable\nobject.  An iterator is created for the result of the\n``expression_list``.  The suite is then executed once for each item\nprovided by the iterator, in the order of ascending indices.  Each\nitem in turn is assigned to the target list using the standard rules\nfor assignments (see *Assignment statements*), and then the suite is\nexecuted.  When the items are exhausted (which is immediately when the\nsequence is empty or an iterator raises a ``StopIteration``\nexception), the suite in the ``else`` clause, if present, is executed,\nand the loop terminates.\n\nA ``break`` statement executed in the first suite terminates the loop\nwithout executing the ``else`` clause\'s suite.  A ``continue``\nstatement executed in the first suite skips the rest of the suite and\ncontinues with the next item, or with the ``else`` clause if there was\nno next item.\n\nThe suite may assign to the variable(s) in the target list; this does\nnot affect the next item assigned to it.\n\nNames in the target list are not deleted when the loop is finished,\nbut if the sequence is empty, it will not have been assigned to at all\nby the loop.  Hint: the built-in function ``range()`` returns an\niterator of integers suitable to emulate the effect of Pascal\'s ``for\ni := a to b do``; e.g., ``list(range(3))`` returns the list ``[0, 1,\n2]``.\n\nNote: There is a subtlety when the sequence is being modified by the loop\n  (this can only occur for mutable sequences, i.e. lists).  An\n  internal counter is used to keep track of which item is used next,\n  and this is incremented on each iteration.  When this counter has\n  reached the length of the sequence the loop terminates.  This means\n  that if the suite deletes the current (or a previous) item from the\n  sequence, the next item will be skipped (since it gets the index of\n  the current item which has already been treated).  Likewise, if the\n  suite inserts an item in the sequence before the current item, the\n  current item will be treated again the next time through the loop.\n  This can lead to nasty bugs that can be avoided by making a\n  temporary copy using a slice of the whole sequence, e.g.,\n\n     for x in a[:]:\n         if x < 0: a.remove(x)\n\n\nThe ``try`` statement\n=====================\n\nThe ``try`` statement specifies exception handlers and/or cleanup code\nfor a group of statements:\n\n   try_stmt  ::= try1_stmt | try2_stmt\n   try1_stmt ::= "try" ":" suite\n                 ("except" [expression ["as" target]] ":" suite)+\n                 ["else" ":" suite]\n                 ["finally" ":" suite]\n   try2_stmt ::= "try" ":" suite\n                 "finally" ":" suite\n\nThe ``except`` clause(s) specify one or more exception handlers. When\nno exception occurs in the ``try`` clause, no exception handler is\nexecuted. When an exception occurs in the ``try`` suite, a search for\nan exception handler is started.  This search inspects the except\nclauses in turn until one is found that matches the exception.  An\nexpression-less except clause, if present, must be last; it matches\nany exception.  For an except clause with an expression, that\nexpression is evaluated, and the clause matches the exception if the\nresulting object is "compatible" with the exception.  An object is\ncompatible with an exception if it is the class or a base class of the\nexception object or a tuple containing an item compatible with the\nexception.\n\nIf no except clause matches the exception, the search for an exception\nhandler continues in the surrounding code and on the invocation stack.\n[1]\n\nIf the evaluation of an expression in the header of an except clause\nraises an exception, the original search for a handler is canceled and\na search starts for the new exception in the surrounding code and on\nthe call stack (it is treated as if the entire ``try`` statement\nraised the exception).\n\nWhen a matching except clause is found, the exception is assigned to\nthe target specified after the ``as`` keyword in that except clause,\nif present, and the except clause\'s suite is executed.  All except\nclauses must have an executable block.  When the end of this block is\nreached, execution continues normally after the entire try statement.\n(This means that if two nested handlers exist for the same exception,\nand the exception occurs in the try clause of the inner handler, the\nouter handler will not handle the exception.)\n\nWhen an exception has been assigned using ``as target``, it is cleared\nat the end of the except clause.  This is as if\n\n   except E as N:\n       foo\n\nwas translated to\n\n   except E as N:\n       try:\n           foo\n       finally:\n           del N\n\nThis means the exception must be assigned to a different name to be\nable to refer to it after the except clause.  Exceptions are cleared\nbecause with the traceback attached to them, they form a reference\ncycle with the stack frame, keeping all locals in that frame alive\nuntil the next garbage collection occurs.\n\nBefore an except clause\'s suite is executed, details about the\nexception are stored in the ``sys`` module and can be access via\n``sys.exc_info()``. ``sys.exc_info()`` returns a 3-tuple consisting of\nthe exception class, the exception instance and a traceback object\n(see section *The standard type hierarchy*) identifying the point in\nthe program where the exception occurred.  ``sys.exc_info()`` values\nare restored to their previous values (before the call) when returning\nfrom a function that handled an exception.\n\nThe optional ``else`` clause is executed if and when control flows off\nthe end of the ``try`` clause. [2] Exceptions in the ``else`` clause\nare not handled by the preceding ``except`` clauses.\n\nIf ``finally`` is present, it specifies a \'cleanup\' handler.  The\n``try`` clause is executed, including any ``except`` and ``else``\nclauses.  If an exception occurs in any of the clauses and is not\nhandled, the exception is temporarily saved. The ``finally`` clause is\nexecuted.  If there is a saved exception, it is re-raised at the end\nof the ``finally`` clause. If the ``finally`` clause raises another\nexception or executes a ``return`` or ``break`` statement, the saved\nexception is lost.  The exception information is not available to the\nprogram during execution of the ``finally`` clause.\n\nWhen a ``return``, ``break`` or ``continue`` statement is executed in\nthe ``try`` suite of a ``try``...``finally`` statement, the\n``finally`` clause is also executed \'on the way out.\' A ``continue``\nstatement is illegal in the ``finally`` clause. (The reason is a\nproblem with the current implementation --- this restriction may be\nlifted in the future).\n\nAdditional information on exceptions can be found in section\n*Exceptions*, and information on using the ``raise`` statement to\ngenerate exceptions may be found in section *The raise statement*.\n\n\nThe ``with`` statement\n======================\n\nThe ``with`` statement is used to wrap the execution of a block with\nmethods defined by a context manager (see section *With Statement\nContext Managers*). This allows common\n``try``...``except``...``finally`` usage patterns to be encapsulated\nfor convenient reuse.\n\n   with_stmt ::= "with" with_item ("," with_item)* ":" suite\n   with_item ::= expression ["as" target]\n\nThe execution of the ``with`` statement with one "item" proceeds as\nfollows:\n\n1. The context expression (the expression given in the ``with_item``)\n   is evaluated to obtain a context manager.\n\n2. The context manager\'s ``__exit__()`` is loaded for later use.\n\n3. The context manager\'s ``__enter__()`` method is invoked.\n\n4. If a target was included in the ``with`` statement, the return\n   value from ``__enter__()`` is assigned to it.\n\n   Note: The ``with`` statement guarantees that if the ``__enter__()``\n     method returns without an error, then ``__exit__()`` will always\n     be called. Thus, if an error occurs during the assignment to the\n     target list, it will be treated the same as an error occurring\n     within the suite would be. See step 6 below.\n\n5. The suite is executed.\n\n6. The context manager\'s ``__exit__()`` method is invoked.  If an\n   exception caused the suite to be exited, its type, value, and\n   traceback are passed as arguments to ``__exit__()``. Otherwise,\n   three ``None`` arguments are supplied.\n\n   If the suite was exited due to an exception, and the return value\n   from the ``__exit__()`` method was false, the exception is\n   reraised.  If the return value was true, the exception is\n   suppressed, and execution continues with the statement following\n   the ``with`` statement.\n\n   If the suite was exited for any reason other than an exception, the\n   return value from ``__exit__()`` is ignored, and execution proceeds\n   at the normal location for the kind of exit that was taken.\n\nWith more than one item, the context managers are processed as if\nmultiple ``with`` statements were nested:\n\n   with A() as a, B() as b:\n       suite\n\nis equivalent to\n\n   with A() as a:\n       with B() as b:\n           suite\n\nChanged in version 3.1: Support for multiple context expressions.\n\nSee also:\n\n   **PEP 0343** - The "with" statement\n      The specification, background, and examples for the Python\n      ``with`` statement.\n\n\nFunction definitions\n====================\n\nA function definition defines a user-defined function object (see\nsection *The standard type hierarchy*):\n\n   funcdef        ::= [decorators] "def" funcname "(" [parameter_list] ")" ["->" expression] ":" suite\n   decorators     ::= decorator+\n   decorator      ::= "@" dotted_name ["(" [argument_list [","]] ")"] NEWLINE\n   dotted_name    ::= identifier ("." identifier)*\n   parameter_list ::= (defparameter ",")*\n                      (  "*" [parameter] ("," defparameter)*\n                      [, "**" parameter]\n                      | "**" parameter\n                      | defparameter [","] )\n   parameter      ::= identifier [":" expression]\n   defparameter   ::= parameter ["=" expression]\n   funcname       ::= identifier\n\nA function definition is an executable statement.  Its execution binds\nthe function name in the current local namespace to a function object\n(a wrapper around the executable code for the function).  This\nfunction object contains a reference to the current global namespace\nas the global namespace to be used when the function is called.\n\nThe function definition does not execute the function body; this gets\nexecuted only when the function is called. [3]\n\nA function definition may be wrapped by one or more *decorator*\nexpressions. Decorator expressions are evaluated when the function is\ndefined, in the scope that contains the function definition.  The\nresult must be a callable, which is invoked with the function object\nas the only argument. The returned value is bound to the function name\ninstead of the function object.  Multiple decorators are applied in\nnested fashion. For example, the following code\n\n   @f1(arg)\n   @f2\n   def func(): pass\n\nis equivalent to\n\n   def func(): pass\n   func = f1(arg)(f2(func))\n\nWhen one or more parameters have the form *parameter* ``=``\n*expression*, the function is said to have "default parameter values."\nFor a parameter with a default value, the corresponding argument may\nbe omitted from a call, in which case the parameter\'s default value is\nsubstituted.  If a parameter has a default value, all following\nparameters up until the "``*``" must also have a default value ---\nthis is a syntactic restriction that is not expressed by the grammar.\n\n**Default parameter values are evaluated when the function definition\nis executed.** This means that the expression is evaluated once, when\nthe function is defined, and that that same "pre-computed" value is\nused for each call.  This is especially important to understand when a\ndefault parameter is a mutable object, such as a list or a dictionary:\nif the function modifies the object (e.g. by appending an item to a\nlist), the default value is in effect modified. This is generally not\nwhat was intended.  A way around this is to use ``None`` as the\ndefault, and explicitly test for it in the body of the function, e.g.:\n\n   def whats_on_the_telly(penguin=None):\n       if penguin is None:\n           penguin = []\n       penguin.append("property of the zoo")\n       return penguin\n\nFunction call semantics are described in more detail in section\n*Calls*. A function call always assigns values to all parameters\nmentioned in the parameter list, either from position arguments, from\nkeyword arguments, or from default values.  If the form\n"``*identifier``" is present, it is initialized to a tuple receiving\nany excess positional parameters, defaulting to the empty tuple.  If\nthe form "``**identifier``" is present, it is initialized to a new\ndictionary receiving any excess keyword arguments, defaulting to a new\nempty dictionary. Parameters after "``*``" or "``*identifier``" are\nkeyword-only parameters and may only be passed used keyword arguments.\n\nParameters may have annotations of the form "``: expression``"\nfollowing the parameter name.  Any parameter may have an annotation\neven those of the form ``*identifier`` or ``**identifier``.  Functions\nmay have "return" annotation of the form "``-> expression``" after the\nparameter list.  These annotations can be any valid Python expression\nand are evaluated when the function definition is executed.\nAnnotations may be evaluated in a different order than they appear in\nthe source code.  The presence of annotations does not change the\nsemantics of a function.  The annotation values are available as\nvalues of a dictionary keyed by the parameters\' names in the\n``__annotations__`` attribute of the function object.\n\nIt is also possible to create anonymous functions (functions not bound\nto a name), for immediate use in expressions.  This uses lambda forms,\ndescribed in section *Lambdas*.  Note that the lambda form is merely a\nshorthand for a simplified function definition; a function defined in\na "``def``" statement can be passed around or assigned to another name\njust like a function defined by a lambda form.  The "``def``" form is\nactually more powerful since it allows the execution of multiple\nstatements and annotations.\n\n**Programmer\'s note:** Functions are first-class objects.  A "``def``"\nform executed inside a function definition defines a local function\nthat can be returned or passed around.  Free variables used in the\nnested function can access the local variables of the function\ncontaining the def.  See section *Naming and binding* for details.\n\n\nClass definitions\n=================\n\nA class definition defines a class object (see section *The standard\ntype hierarchy*):\n\n   classdef    ::= [decorators] "class" classname [inheritance] ":" suite\n   inheritance ::= "(" [argument_list [","] | comprehension] ")"\n   classname   ::= identifier\n\nA class definition is an executable statement.  The inheritance list\nusually gives a list of base classes (see *Customizing class creation*\nfor more advanced uses), so each item in the list should evaluate to a\nclass object which allows subclassing.  Classes without an inheritance\nlist inherit, by default, from the base class ``object``; hence,\n\n   class Foo:\n       pass\n\nis equivalent to\n\n   class Foo(object):\n       pass\n\nThe class\'s suite is then executed in a new execution frame (see\n*Naming and binding*), using a newly created local namespace and the\noriginal global namespace. (Usually, the suite contains mostly\nfunction definitions.)  When the class\'s suite finishes execution, its\nexecution frame is discarded but its local namespace is saved. [4] A\nclass object is then created using the inheritance list for the base\nclasses and the saved local namespace for the attribute dictionary.\nThe class name is bound to this class object in the original local\nnamespace.\n\nClass creation can be customized heavily using *metaclasses*.\n\nClasses can also be decorated: just like when decorating functions,\n\n   @f1(arg)\n   @f2\n   class Foo: pass\n\nis equivalent to\n\n   class Foo: pass\n   Foo = f1(arg)(f2(Foo))\n\nThe evaluation rules for the decorator expressions are the same as for\nfunction decorators.  The result must be a class object, which is then\nbound to the class name.\n\n**Programmer\'s note:** Variables defined in the class definition are\nclass attributes; they are shared by instances.  Instance attributes\ncan be set in a method with ``self.name = value``.  Both class and\ninstance attributes are accessible through the notation\n"``self.name``", and an instance attribute hides a class attribute\nwith the same name when accessed in this way.  Class attributes can be\nused as defaults for instance attributes, but using mutable values\nthere can lead to unexpected results.  *Descriptors* can be used to\ncreate instance variables with different implementation details.\n\nSee also:\n\n   **PEP 3115** - Metaclasses in Python 3 **PEP 3129** - Class\n   Decorators\n\n-[ Footnotes ]-\n\n[1] The exception is propagated to the invocation stack unless there\n    is a ``finally`` clause which happens to raise another exception.\n    That new exception causes the old one to be lost.\n\n[2] Currently, control "flows off the end" except in the case of an\n    exception or the execution of a ``return``, ``continue``, or\n    ``break`` statement.\n\n[3] A string literal appearing as the first statement in the function\n    body is transformed into the function\'s ``__doc__`` attribute and\n    therefore the function\'s *docstring*.\n\n[4] A string literal appearing as the first statement in the class\n    body is transformed into the namespace\'s ``__doc__`` item and\n    therefore the class\'s *docstring*.\n',
@@ -33,7 +33,7 @@
  'exprlists': '\nExpression lists\n****************\n\n   expression_list ::= expression ( "," expression )* [","]\n\nAn expression list containing at least one comma yields a tuple.  The\nlength of the tuple is the number of expressions in the list.  The\nexpressions are evaluated from left to right.\n\nThe trailing comma is required only to create a single tuple (a.k.a. a\n*singleton*); it is optional in all other cases.  A single expression\nwithout a trailing comma doesn\'t create a tuple, but rather yields the\nvalue of that expression. (To create an empty tuple, use an empty pair\nof parentheses: ``()``.)\n',
  'floating': '\nFloating point literals\n***********************\n\nFloating point literals are described by the following lexical\ndefinitions:\n\n   floatnumber   ::= pointfloat | exponentfloat\n   pointfloat    ::= [intpart] fraction | intpart "."\n   exponentfloat ::= (intpart | pointfloat) exponent\n   intpart       ::= digit+\n   fraction      ::= "." digit+\n   exponent      ::= ("e" | "E") ["+" | "-"] digit+\n\nNote that the integer and exponent parts are always interpreted using\nradix 10. For example, ``077e010`` is legal, and denotes the same\nnumber as ``77e10``. The allowed range of floating point literals is\nimplementation-dependent. Some examples of floating point literals:\n\n   3.14    10.    .001    1e100    3.14e-10    0e0\n\nNote that numeric literals do not include a sign; a phrase like ``-1``\nis actually an expression composed of the unary operator ``-`` and the\nliteral ``1``.\n',
  'for': '\nThe ``for`` statement\n*********************\n\nThe ``for`` statement is used to iterate over the elements of a\nsequence (such as a string, tuple or list) or other iterable object:\n\n   for_stmt ::= "for" target_list "in" expression_list ":" suite\n                ["else" ":" suite]\n\nThe expression list is evaluated once; it should yield an iterable\nobject.  An iterator is created for the result of the\n``expression_list``.  The suite is then executed once for each item\nprovided by the iterator, in the order of ascending indices.  Each\nitem in turn is assigned to the target list using the standard rules\nfor assignments (see *Assignment statements*), and then the suite is\nexecuted.  When the items are exhausted (which is immediately when the\nsequence is empty or an iterator raises a ``StopIteration``\nexception), the suite in the ``else`` clause, if present, is executed,\nand the loop terminates.\n\nA ``break`` statement executed in the first suite terminates the loop\nwithout executing the ``else`` clause\'s suite.  A ``continue``\nstatement executed in the first suite skips the rest of the suite and\ncontinues with the next item, or with the ``else`` clause if there was\nno next item.\n\nThe suite may assign to the variable(s) in the target list; this does\nnot affect the next item assigned to it.\n\nNames in the target list are not deleted when the loop is finished,\nbut if the sequence is empty, it will not have been assigned to at all\nby the loop.  Hint: the built-in function ``range()`` returns an\niterator of integers suitable to emulate the effect of Pascal\'s ``for\ni := a to b do``; e.g., ``list(range(3))`` returns the list ``[0, 1,\n2]``.\n\nNote: There is a subtlety when the sequence is being modified by the loop\n  (this can only occur for mutable sequences, i.e. lists).  An\n  internal counter is used to keep track of which item is used next,\n  and this is incremented on each iteration.  When this counter has\n  reached the length of the sequence the loop terminates.  This means\n  that if the suite deletes the current (or a previous) item from the\n  sequence, the next item will be skipped (since it gets the index of\n  the current item which has already been treated).  Likewise, if the\n  suite inserts an item in the sequence before the current item, the\n  current item will be treated again the next time through the loop.\n  This can lead to nasty bugs that can be avoided by making a\n  temporary copy using a slice of the whole sequence, e.g.,\n\n     for x in a[:]:\n         if x < 0: a.remove(x)\n',
- 'formatstrings': '\nFormat String Syntax\n********************\n\nThe ``str.format()`` method and the ``Formatter`` class share the same\nsyntax for format strings (although in the case of ``Formatter``,\nsubclasses can define their own format string syntax).\n\nFormat strings contain "replacement fields" surrounded by curly braces\n``{}``. Anything that is not contained in braces is considered literal\ntext, which is copied unchanged to the output.  If you need to include\na brace character in the literal text, it can be escaped by doubling:\n``{{`` and ``}}``.\n\nThe grammar for a replacement field is as follows:\n\n      replacement_field ::= "{" [field_name] ["!" conversion] [":" format_spec] "}"\n      field_name        ::= arg_name ("." attribute_name | "[" element_index "]")*\n      arg_name          ::= [identifier | integer]\n      attribute_name    ::= identifier\n      element_index     ::= integer | index_string\n      index_string      ::= <any source character except "]"> +\n      conversion        ::= "r" | "s" | "a"\n      format_spec       ::= <described in the next section>\n\nIn less formal terms, the replacement field can start with a\n*field_name* that specifies the object whose value is to be formatted\nand inserted into the output instead of the replacement field. The\n*field_name* is optionally followed by a  *conversion* field, which is\npreceded by an exclamation point ``\'!\'``, and a *format_spec*, which\nis preceded by a colon ``\':\'``.  These specify a non-default format\nfor the replacement value.\n\nSee also the *Format Specification Mini-Language* section.\n\nThe *field_name* itself begins with an *arg_name* that is either\neither a number or a keyword.  If it\'s a number, it refers to a\npositional argument, and if it\'s a keyword, it refers to a named\nkeyword argument.  If the numerical arg_names in a format string are\n0, 1, 2, ... in sequence, they can all be omitted (not just some) and\nthe numbers 0, 1, 2, ... will be automatically inserted in that order.\nThe *arg_name* can be followed by any number of index or attribute\nexpressions. An expression of the form ``\'.name\'`` selects the named\nattribute using ``getattr()``, while an expression of the form\n``\'[index]\'`` does an index lookup using ``__getitem__()``.\n\nChanged in version 3.1: The positional argument specifiers can be\nomitted, so ``\'{} {}\'`` is equivalent to ``\'{0} {1}\'``.\n\nSome simple format string examples:\n\n   "First, thou shalt count to {0}" # References first positional argument\n   "Bring me a {}"                  # Implicitly references the first positional argument\n   "From {} to {}"                  # Same as "From {0} to {1}"\n   "My quest is {name}"             # References keyword argument \'name\'\n   "Weight in tons {0.weight}"      # \'weight\' attribute of first positional arg\n   "Units destroyed: {players[0]}"  # First element of keyword argument \'players\'.\n\nThe *conversion* field causes a type coercion before formatting.\nNormally, the job of formatting a value is done by the\n``__format__()`` method of the value itself.  However, in some cases\nit is desirable to force a type to be formatted as a string,\noverriding its own definition of formatting.  By converting the value\nto a string before calling ``__format__()``, the normal formatting\nlogic is bypassed.\n\nThree conversion flags are currently supported: ``\'!s\'`` which calls\n``str()`` on the value, ``\'!r\'`` which calls ``repr()`` and ``\'!a\'``\nwhich calls ``ascii()``.\n\nSome examples:\n\n   "Harold\'s a clever {0!s}"        # Calls str() on the argument first\n   "Bring out the holy {name!r}"    # Calls repr() on the argument first\n   "More {!a}"                      # Calls ascii() on the argument first\n\nThe *format_spec* field contains a specification of how the value\nshould be presented, including such details as field width, alignment,\npadding, decimal precision and so on.  Each value type can define its\nown "formatting mini-language" or interpretation of the *format_spec*.\n\nMost built-in types support a common formatting mini-language, which\nis described in the next section.\n\nA *format_spec* field can also include nested replacement fields\nwithin it. These nested replacement fields can contain only a field\nname; conversion flags and format specifications are not allowed.  The\nreplacement fields within the format_spec are substituted before the\n*format_spec* string is interpreted. This allows the formatting of a\nvalue to be dynamically specified.\n\nSee the *Format examples* section for some examples.\n\n\nFormat Specification Mini-Language\n==================================\n\n"Format specifications" are used within replacement fields contained\nwithin a format string to define how individual values are presented\n(see *Format String Syntax*).  They can also be passed directly to the\nbuilt-in ``format()`` function.  Each formattable type may define how\nthe format specification is to be interpreted.\n\nMost built-in types implement the following options for format\nspecifications, although some of the formatting options are only\nsupported by the numeric types.\n\nA general convention is that an empty format string (``""``) produces\nthe same result as if you had called ``str()`` on the value. A non-\nempty format string typically modifies the result.\n\nThe general form of a *standard format specifier* is:\n\n   format_spec ::= [[fill]align][sign][#][0][width][,][.precision][type]\n   fill        ::= <a character other than \'}\'>\n   align       ::= "<" | ">" | "=" | "^"\n   sign        ::= "+" | "-" | " "\n   width       ::= integer\n   precision   ::= integer\n   type        ::= "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"\n\nThe *fill* character can be any character other than \'{\' or \'}\'.  The\npresence of a fill character is signaled by the character following\nit, which must be one of the alignment options.  If the second\ncharacter of *format_spec* is not a valid alignment option, then it is\nassumed that both the fill character and the alignment option are\nabsent.\n\nThe meaning of the various alignment options is as follows:\n\n   +-----------+------------------------------------------------------------+\n   | Option    | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'<\'``   | Forces the field to be left-aligned within the available   |\n   |           | space (this is the default for most objects).              |\n   +-----------+------------------------------------------------------------+\n   | ``\'>\'``   | Forces the field to be right-aligned within the available  |\n   |           | space (this is the default for numbers).                   |\n   +-----------+------------------------------------------------------------+\n   | ``\'=\'``   | Forces the padding to be placed after the sign (if any)    |\n   |           | but before the digits.  This is used for printing fields   |\n   |           | in the form \'+000000120\'. This alignment option is only    |\n   |           | valid for numeric types.                                   |\n   +-----------+------------------------------------------------------------+\n   | ``\'^\'``   | Forces the field to be centered within the available       |\n   |           | space.                                                     |\n   +-----------+------------------------------------------------------------+\n\nNote that unless a minimum field width is defined, the field width\nwill always be the same size as the data to fill it, so that the\nalignment option has no meaning in this case.\n\nThe *sign* option is only valid for number types, and can be one of\nthe following:\n\n   +-----------+------------------------------------------------------------+\n   | Option    | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'+\'``   | indicates that a sign should be used for both positive as  |\n   |           | well as negative numbers.                                  |\n   +-----------+------------------------------------------------------------+\n   | ``\'-\'``   | indicates that a sign should be used only for negative     |\n   |           | numbers (this is the default behavior).                    |\n   +-----------+------------------------------------------------------------+\n   | space     | indicates that a leading space should be used on positive  |\n   |           | numbers, and a minus sign on negative numbers.             |\n   +-----------+------------------------------------------------------------+\n\nThe ``\'#\'`` option causes the "alternate form" to be used for the\nconversion.  The alternate form is defined differently for different\ntypes.  This option is only valid for integer, float, complex and\nDecimal types. For integers, when binary, octal, or hexadecimal output\nis used, this option adds the prefix respective ``\'0b\'``, ``\'0o\'``, or\n``\'0x\'`` to the output value. For floats, complex and Decimal the\nalternate form causes the result of the conversion to always contain a\ndecimal-point character, even if no digits follow it. Normally, a\ndecimal-point character appears in the result of these conversions\nonly if a digit follows it. In addition, for ``\'g\'`` and ``\'G\'``\nconversions, trailing zeros are not removed from the result.\n\nThe ``\',\'`` option signals the use of a comma for a thousands\nseparator. For a locale aware separator, use the ``\'n\'`` integer\npresentation type instead.\n\nChanged in version 3.1: Added the ``\',\'`` option (see also **PEP\n378**).\n\n*width* is a decimal integer defining the minimum field width.  If not\nspecified, then the field width will be determined by the content.\n\nIf the *width* field is preceded by a zero (``\'0\'``) character, this\nenables zero-padding.  This is equivalent to an *alignment* type of\n``\'=\'`` and a *fill* character of ``\'0\'``.\n\nThe *precision* is a decimal number indicating how many digits should\nbe displayed after the decimal point for a floating point value\nformatted with ``\'f\'`` and ``\'F\'``, or before and after the decimal\npoint for a floating point value formatted with ``\'g\'`` or ``\'G\'``.\nFor non-number types the field indicates the maximum field size - in\nother words, how many characters will be used from the field content.\nThe *precision* is not allowed for integer values.\n\nFinally, the *type* determines how the data should be presented.\n\nThe available string presentation types are:\n\n   +-----------+------------------------------------------------------------+\n   | Type      | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'s\'``   | String format. This is the default type for strings and    |\n   |           | may be omitted.                                            |\n   +-----------+------------------------------------------------------------+\n   | None      | The same as ``\'s\'``.                                       |\n   +-----------+------------------------------------------------------------+\n\nThe available integer presentation types are:\n\n   +-----------+------------------------------------------------------------+\n   | Type      | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'b\'``   | Binary format. Outputs the number in base 2.               |\n   +-----------+------------------------------------------------------------+\n   | ``\'c\'``   | Character. Converts the integer to the corresponding       |\n   |           | unicode character before printing.                         |\n   +-----------+------------------------------------------------------------+\n   | ``\'d\'``   | Decimal Integer. Outputs the number in base 10.            |\n   +-----------+------------------------------------------------------------+\n   | ``\'o\'``   | Octal format. Outputs the number in base 8.                |\n   +-----------+------------------------------------------------------------+\n   | ``\'x\'``   | Hex format. Outputs the number in base 16, using lower-    |\n   |           | case letters for the digits above 9.                       |\n   +-----------+------------------------------------------------------------+\n   | ``\'X\'``   | Hex format. Outputs the number in base 16, using upper-    |\n   |           | case letters for the digits above 9.                       |\n   +-----------+------------------------------------------------------------+\n   | ``\'n\'``   | Number. This is the same as ``\'d\'``, except that it uses   |\n   |           | the current locale setting to insert the appropriate       |\n   |           | number separator characters.                               |\n   +-----------+------------------------------------------------------------+\n   | None      | The same as ``\'d\'``.                                       |\n   +-----------+------------------------------------------------------------+\n\nIn addition to the above presentation types, integers can be formatted\nwith the floating point presentation types listed below (except\n``\'n\'`` and None). When doing so, ``float()`` is used to convert the\ninteger to a floating point number before formatting.\n\nThe available presentation types for floating point and decimal values\nare:\n\n   +-----------+------------------------------------------------------------+\n   | Type      | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'e\'``   | Exponent notation. Prints the number in scientific         |\n   |           | notation using the letter \'e\' to indicate the exponent.    |\n   +-----------+------------------------------------------------------------+\n   | ``\'E\'``   | Exponent notation. Same as ``\'e\'`` except it uses an upper |\n   |           | case \'E\' as the separator character.                       |\n   +-----------+------------------------------------------------------------+\n   | ``\'f\'``   | Fixed point. Displays the number as a fixed-point number.  |\n   +-----------+------------------------------------------------------------+\n   | ``\'F\'``   | Fixed point. Same as ``\'f\'``, but converts ``nan`` to      |\n   |           | ``NAN`` and ``inf`` to ``INF``.                            |\n   +-----------+------------------------------------------------------------+\n   | ``\'g\'``   | General format.  For a given precision ``p >= 1``, this    |\n   |           | rounds the number to ``p`` significant digits and then     |\n   |           | formats the result in either fixed-point format or in      |\n   |           | scientific notation, depending on its magnitude.  The      |\n   |           | precise rules are as follows: suppose that the result      |\n   |           | formatted with presentation type ``\'e\'`` and precision     |\n   |           | ``p-1`` would have exponent ``exp``.  Then if ``-4 <= exp  |\n   |           | < p``, the number is formatted with presentation type      |\n   |           | ``\'f\'`` and precision ``p-1-exp``. Otherwise, the number   |\n   |           | is formatted with presentation type ``\'e\'`` and precision  |\n   |           | ``p-1``. In both cases insignificant trailing zeros are    |\n   |           | removed from the significand, and the decimal point is     |\n   |           | also removed if there are no remaining digits following    |\n   |           | it.  Positive and negative infinity, positive and negative |\n   |           | zero, and nans, are formatted as ``inf``, ``-inf``, ``0``, |\n   |           | ``-0`` and ``nan`` respectively, regardless of the         |\n   |           | precision.  A precision of ``0`` is treated as equivalent  |\n   |           | to a precision of ``1``.                                   |\n   +-----------+------------------------------------------------------------+\n   | ``\'G\'``   | General format. Same as ``\'g\'`` except switches to ``\'E\'`` |\n   |           | if the number gets too large. The representations of       |\n   |           | infinity and NaN are uppercased, too.                      |\n   +-----------+------------------------------------------------------------+\n   | ``\'n\'``   | Number. This is the same as ``\'g\'``, except that it uses   |\n   |           | the current locale setting to insert the appropriate       |\n   |           | number separator characters.                               |\n   +-----------+------------------------------------------------------------+\n   | ``\'%\'``   | Percentage. Multiplies the number by 100 and displays in   |\n   |           | fixed (``\'f\'``) format, followed by a percent sign.        |\n   +-----------+------------------------------------------------------------+\n   | None      | Similar to ``\'g\'``, except with at least one digit past    |\n   |           | the decimal point and a default precision of 12. This is   |\n   |           | intended to match ``str()``, except you can add the other  |\n   |           | format modifiers.                                          |\n   +-----------+------------------------------------------------------------+\n\n\nFormat examples\n===============\n\nThis section contains examples of the new format syntax and comparison\nwith the old ``%``-formatting.\n\nIn most of the cases the syntax is similar to the old\n``%``-formatting, with the addition of the ``{}`` and with ``:`` used\ninstead of ``%``. For example, ``\'%03.2f\'`` can be translated to\n``\'{:03.2f}\'``.\n\nThe new format syntax also supports new and different options, shown\nin the follow examples.\n\nAccessing arguments by position:\n\n   >>> \'{0}, {1}, {2}\'.format(\'a\', \'b\', \'c\')\n   \'a, b, c\'\n   >>> \'{}, {}, {}\'.format(\'a\', \'b\', \'c\')  # 3.1+ only\n   \'a, b, c\'\n   >>> \'{2}, {1}, {0}\'.format(\'a\', \'b\', \'c\')\n   \'c, b, a\'\n   >>> \'{2}, {1}, {0}\'.format(*\'abc\')      # unpacking argument sequence\n   \'c, b, a\'\n   >>> \'{0}{1}{0}\'.format(\'abra\', \'cad\')   # arguments\' indices can be repeated\n   \'abracadabra\'\n\nAccessing arguments by name:\n\n   >>> \'Coordinates: {latitude}, {longitude}\'.format(latitude=\'37.24N\', longitude=\'-115.81W\')\n   \'Coordinates: 37.24N, -115.81W\'\n   >>> coord = {\'latitude\': \'37.24N\', \'longitude\': \'-115.81W\'}\n   >>> \'Coordinates: {latitude}, {longitude}\'.format(**coord)\n   \'Coordinates: 37.24N, -115.81W\'\n\nAccessing arguments\' attributes:\n\n   >>> c = 3-5j\n   >>> (\'The complex number {0} is formed from the real part {0.real} \'\n   ...  \'and the imaginary part {0.imag}.\').format(c)\n   \'The complex number (3-5j) is formed from the real part 3.0 and the imaginary part -5.0.\'\n   >>> class Point:\n   ...     def __init__(self, x, y):\n   ...         self.x, self.y = x, y\n   ...     def __str__(self):\n   ...         return \'Point({self.x}, {self.y})\'.format(self=self)\n   ...\n   >>> str(Point(4, 2))\n   \'Point(4, 2)\'\n\nAccessing arguments\' items:\n\n   >>> coord = (3, 5)\n   >>> \'X: {0[0]};  Y: {0[1]}\'.format(coord)\n   \'X: 3;  Y: 5\'\n\nReplacing ``%s`` and ``%r``:\n\n   >>> "repr() shows quotes: {!r}; str() doesn\'t: {!s}".format(\'test1\', \'test2\')\n   "repr() shows quotes: \'test1\'; str() doesn\'t: test2"\n\nAligning the text and specifying a width:\n\n   >>> \'{:<30}\'.format(\'left aligned\')\n   \'left aligned                  \'\n   >>> \'{:>30}\'.format(\'right aligned\')\n   \'                 right aligned\'\n   >>> \'{:^30}\'.format(\'centered\')\n   \'           centered           \'\n   >>> \'{:*^30}\'.format(\'centered\')  # use \'*\' as a fill char\n   \'***********centered***********\'\n\nReplacing ``%+f``, ``%-f``, and ``% f`` and specifying a sign:\n\n   >>> \'{:+f}; {:+f}\'.format(3.14, -3.14)  # show it always\n   \'+3.140000; -3.140000\'\n   >>> \'{: f}; {: f}\'.format(3.14, -3.14)  # show a space for positive numbers\n   \' 3.140000; -3.140000\'\n   >>> \'{:-f}; {:-f}\'.format(3.14, -3.14)  # show only the minus -- same as \'{:f}; {:f}\'\n   \'3.140000; -3.140000\'\n\nReplacing ``%x`` and ``%o`` and converting the value to different\nbases:\n\n   >>> # format also supports binary numbers\n   >>> "int: {0:d};  hex: {0:x};  oct: {0:o};  bin: {0:b}".format(42)\n   \'int: 42;  hex: 2a;  oct: 52;  bin: 101010\'\n   >>> # with 0x, 0o, or 0b as prefix:\n   >>> "int: {0:d};  hex: {0:#x};  oct: {0:#o};  bin: {0:#b}".format(42)\n   \'int: 42;  hex: 0x2a;  oct: 0o52;  bin: 0b101010\'\n\nUsing the comma as a thousands separator:\n\n   >>> \'{:,}\'.format(1234567890)\n   \'1,234,567,890\'\n\nExpressing a percentage:\n\n   >>> points = 19\n   >>> total = 22\n   >>> \'Correct answers: {:.2%}.\'.format(points/total)\n   \'Correct answers: 86.36%\'\n\nUsing type-specific formatting:\n\n   >>> import datetime\n   >>> d = datetime.datetime(2010, 7, 4, 12, 15, 58)\n   >>> \'{:%Y-%m-%d %H:%M:%S}\'.format(d)\n   \'2010-07-04 12:15:58\'\n\nNesting arguments and more complex examples:\n\n   >>> for align, text in zip(\'<^>\', [\'left\', \'center\', \'right\']):\n   ...     \'{0:{fill}{align}16}\'.format(text, fill=align, align=align)\n   ...\n   \'left<<<<<<<<<<<<\'\n   \'^^^^^center^^^^^\'\n   \'>>>>>>>>>>>right\'\n   >>>\n   >>> octets = [192, 168, 0, 1]\n   >>> \'{:02X}{:02X}{:02X}{:02X}\'.format(*octets)\n   \'C0A80001\'\n   >>> int(_, 16)\n   3232235521\n   >>>\n   >>> width = 5\n   >>> for num in range(5,12):\n   ...     for base in \'dXob\':\n   ...         print(\'{0:{width}{base}}\'.format(num, base=base, width=width), end=\' \')\n   ...     print()\n   ...\n       5     5     5   101\n       6     6     6   110\n       7     7     7   111\n       8     8    10  1000\n       9     9    11  1001\n      10     A    12  1010\n      11     B    13  1011\n',
+ 'formatstrings': '\nFormat String Syntax\n********************\n\nThe ``str.format()`` method and the ``Formatter`` class share the same\nsyntax for format strings (although in the case of ``Formatter``,\nsubclasses can define their own format string syntax).\n\nFormat strings contain "replacement fields" surrounded by curly braces\n``{}``. Anything that is not contained in braces is considered literal\ntext, which is copied unchanged to the output.  If you need to include\na brace character in the literal text, it can be escaped by doubling:\n``{{`` and ``}}``.\n\nThe grammar for a replacement field is as follows:\n\n      replacement_field ::= "{" [field_name] ["!" conversion] [":" format_spec] "}"\n      field_name        ::= arg_name ("." attribute_name | "[" element_index "]")*\n      arg_name          ::= [identifier | integer]\n      attribute_name    ::= identifier\n      element_index     ::= integer | index_string\n      index_string      ::= <any source character except "]"> +\n      conversion        ::= "r" | "s" | "a"\n      format_spec       ::= <described in the next section>\n\nIn less formal terms, the replacement field can start with a\n*field_name* that specifies the object whose value is to be formatted\nand inserted into the output instead of the replacement field. The\n*field_name* is optionally followed by a  *conversion* field, which is\npreceded by an exclamation point ``\'!\'``, and a *format_spec*, which\nis preceded by a colon ``\':\'``.  These specify a non-default format\nfor the replacement value.\n\nSee also the *Format Specification Mini-Language* section.\n\nThe *field_name* itself begins with an *arg_name* that is either\neither a number or a keyword.  If it\'s a number, it refers to a\npositional argument, and if it\'s a keyword, it refers to a named\nkeyword argument.  If the numerical arg_names in a format string are\n0, 1, 2, ... in sequence, they can all be omitted (not just some) and\nthe numbers 0, 1, 2, ... will be automatically inserted in that order.\nBecause *arg_name* is not quote-delimited, it is not possible to\nspecify arbitrary dictionary keys (e.g., the strings ``\'10\'`` or\n``\':-]\'``) within a format string. The *arg_name* can be followed by\nany number of index or attribute expressions. An expression of the\nform ``\'.name\'`` selects the named attribute using ``getattr()``,\nwhile an expression of the form ``\'[index]\'`` does an index lookup\nusing ``__getitem__()``.\n\nChanged in version 3.1: The positional argument specifiers can be\nomitted, so ``\'{} {}\'`` is equivalent to ``\'{0} {1}\'``.\n\nSome simple format string examples:\n\n   "First, thou shalt count to {0}" # References first positional argument\n   "Bring me a {}"                  # Implicitly references the first positional argument\n   "From {} to {}"                  # Same as "From {0} to {1}"\n   "My quest is {name}"             # References keyword argument \'name\'\n   "Weight in tons {0.weight}"      # \'weight\' attribute of first positional arg\n   "Units destroyed: {players[0]}"  # First element of keyword argument \'players\'.\n\nThe *conversion* field causes a type coercion before formatting.\nNormally, the job of formatting a value is done by the\n``__format__()`` method of the value itself.  However, in some cases\nit is desirable to force a type to be formatted as a string,\noverriding its own definition of formatting.  By converting the value\nto a string before calling ``__format__()``, the normal formatting\nlogic is bypassed.\n\nThree conversion flags are currently supported: ``\'!s\'`` which calls\n``str()`` on the value, ``\'!r\'`` which calls ``repr()`` and ``\'!a\'``\nwhich calls ``ascii()``.\n\nSome examples:\n\n   "Harold\'s a clever {0!s}"        # Calls str() on the argument first\n   "Bring out the holy {name!r}"    # Calls repr() on the argument first\n   "More {!a}"                      # Calls ascii() on the argument first\n\nThe *format_spec* field contains a specification of how the value\nshould be presented, including such details as field width, alignment,\npadding, decimal precision and so on.  Each value type can define its\nown "formatting mini-language" or interpretation of the *format_spec*.\n\nMost built-in types support a common formatting mini-language, which\nis described in the next section.\n\nA *format_spec* field can also include nested replacement fields\nwithin it. These nested replacement fields can contain only a field\nname; conversion flags and format specifications are not allowed.  The\nreplacement fields within the format_spec are substituted before the\n*format_spec* string is interpreted. This allows the formatting of a\nvalue to be dynamically specified.\n\nSee the *Format examples* section for some examples.\n\n\nFormat Specification Mini-Language\n==================================\n\n"Format specifications" are used within replacement fields contained\nwithin a format string to define how individual values are presented\n(see *Format String Syntax*).  They can also be passed directly to the\nbuilt-in ``format()`` function.  Each formattable type may define how\nthe format specification is to be interpreted.\n\nMost built-in types implement the following options for format\nspecifications, although some of the formatting options are only\nsupported by the numeric types.\n\nA general convention is that an empty format string (``""``) produces\nthe same result as if you had called ``str()`` on the value. A non-\nempty format string typically modifies the result.\n\nThe general form of a *standard format specifier* is:\n\n   format_spec ::= [[fill]align][sign][#][0][width][,][.precision][type]\n   fill        ::= <a character other than \'}\'>\n   align       ::= "<" | ">" | "=" | "^"\n   sign        ::= "+" | "-" | " "\n   width       ::= integer\n   precision   ::= integer\n   type        ::= "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"\n\nThe *fill* character can be any character other than \'{\' or \'}\'.  The\npresence of a fill character is signaled by the character following\nit, which must be one of the alignment options.  If the second\ncharacter of *format_spec* is not a valid alignment option, then it is\nassumed that both the fill character and the alignment option are\nabsent.\n\nThe meaning of the various alignment options is as follows:\n\n   +-----------+------------------------------------------------------------+\n   | Option    | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'<\'``   | Forces the field to be left-aligned within the available   |\n   |           | space (this is the default for most objects).              |\n   +-----------+------------------------------------------------------------+\n   | ``\'>\'``   | Forces the field to be right-aligned within the available  |\n   |           | space (this is the default for numbers).                   |\n   +-----------+------------------------------------------------------------+\n   | ``\'=\'``   | Forces the padding to be placed after the sign (if any)    |\n   |           | but before the digits.  This is used for printing fields   |\n   |           | in the form \'+000000120\'. This alignment option is only    |\n   |           | valid for numeric types.                                   |\n   +-----------+------------------------------------------------------------+\n   | ``\'^\'``   | Forces the field to be centered within the available       |\n   |           | space.                                                     |\n   +-----------+------------------------------------------------------------+\n\nNote that unless a minimum field width is defined, the field width\nwill always be the same size as the data to fill it, so that the\nalignment option has no meaning in this case.\n\nThe *sign* option is only valid for number types, and can be one of\nthe following:\n\n   +-----------+------------------------------------------------------------+\n   | Option    | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'+\'``   | indicates that a sign should be used for both positive as  |\n   |           | well as negative numbers.                                  |\n   +-----------+------------------------------------------------------------+\n   | ``\'-\'``   | indicates that a sign should be used only for negative     |\n   |           | numbers (this is the default behavior).                    |\n   +-----------+------------------------------------------------------------+\n   | space     | indicates that a leading space should be used on positive  |\n   |           | numbers, and a minus sign on negative numbers.             |\n   +-----------+------------------------------------------------------------+\n\nThe ``\'#\'`` option causes the "alternate form" to be used for the\nconversion.  The alternate form is defined differently for different\ntypes.  This option is only valid for integer, float, complex and\nDecimal types. For integers, when binary, octal, or hexadecimal output\nis used, this option adds the prefix respective ``\'0b\'``, ``\'0o\'``, or\n``\'0x\'`` to the output value. For floats, complex and Decimal the\nalternate form causes the result of the conversion to always contain a\ndecimal-point character, even if no digits follow it. Normally, a\ndecimal-point character appears in the result of these conversions\nonly if a digit follows it. In addition, for ``\'g\'`` and ``\'G\'``\nconversions, trailing zeros are not removed from the result.\n\nThe ``\',\'`` option signals the use of a comma for a thousands\nseparator. For a locale aware separator, use the ``\'n\'`` integer\npresentation type instead.\n\nChanged in version 3.1: Added the ``\',\'`` option (see also **PEP\n378**).\n\n*width* is a decimal integer defining the minimum field width.  If not\nspecified, then the field width will be determined by the content.\n\nIf the *width* field is preceded by a zero (``\'0\'``) character, this\nenables zero-padding.  This is equivalent to an *alignment* type of\n``\'=\'`` and a *fill* character of ``\'0\'``.\n\nThe *precision* is a decimal number indicating how many digits should\nbe displayed after the decimal point for a floating point value\nformatted with ``\'f\'`` and ``\'F\'``, or before and after the decimal\npoint for a floating point value formatted with ``\'g\'`` or ``\'G\'``.\nFor non-number types the field indicates the maximum field size - in\nother words, how many characters will be used from the field content.\nThe *precision* is not allowed for integer values.\n\nFinally, the *type* determines how the data should be presented.\n\nThe available string presentation types are:\n\n   +-----------+------------------------------------------------------------+\n   | Type      | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'s\'``   | String format. This is the default type for strings and    |\n   |           | may be omitted.                                            |\n   +-----------+------------------------------------------------------------+\n   | None      | The same as ``\'s\'``.                                       |\n   +-----------+------------------------------------------------------------+\n\nThe available integer presentation types are:\n\n   +-----------+------------------------------------------------------------+\n   | Type      | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'b\'``   | Binary format. Outputs the number in base 2.               |\n   +-----------+------------------------------------------------------------+\n   | ``\'c\'``   | Character. Converts the integer to the corresponding       |\n   |           | unicode character before printing.                         |\n   +-----------+------------------------------------------------------------+\n   | ``\'d\'``   | Decimal Integer. Outputs the number in base 10.            |\n   +-----------+------------------------------------------------------------+\n   | ``\'o\'``   | Octal format. Outputs the number in base 8.                |\n   +-----------+------------------------------------------------------------+\n   | ``\'x\'``   | Hex format. Outputs the number in base 16, using lower-    |\n   |           | case letters for the digits above 9.                       |\n   +-----------+------------------------------------------------------------+\n   | ``\'X\'``   | Hex format. Outputs the number in base 16, using upper-    |\n   |           | case letters for the digits above 9.                       |\n   +-----------+------------------------------------------------------------+\n   | ``\'n\'``   | Number. This is the same as ``\'d\'``, except that it uses   |\n   |           | the current locale setting to insert the appropriate       |\n   |           | number separator characters.                               |\n   +-----------+------------------------------------------------------------+\n   | None      | The same as ``\'d\'``.                                       |\n   +-----------+------------------------------------------------------------+\n\nIn addition to the above presentation types, integers can be formatted\nwith the floating point presentation types listed below (except\n``\'n\'`` and None). When doing so, ``float()`` is used to convert the\ninteger to a floating point number before formatting.\n\nThe available presentation types for floating point and decimal values\nare:\n\n   +-----------+------------------------------------------------------------+\n   | Type      | Meaning                                                    |\n   +===========+============================================================+\n   | ``\'e\'``   | Exponent notation. Prints the number in scientific         |\n   |           | notation using the letter \'e\' to indicate the exponent.    |\n   +-----------+------------------------------------------------------------+\n   | ``\'E\'``   | Exponent notation. Same as ``\'e\'`` except it uses an upper |\n   |           | case \'E\' as the separator character.                       |\n   +-----------+------------------------------------------------------------+\n   | ``\'f\'``   | Fixed point. Displays the number as a fixed-point number.  |\n   +-----------+------------------------------------------------------------+\n   | ``\'F\'``   | Fixed point. Same as ``\'f\'``, but converts ``nan`` to      |\n   |           | ``NAN`` and ``inf`` to ``INF``.                            |\n   +-----------+------------------------------------------------------------+\n   | ``\'g\'``   | General format.  For a given precision ``p >= 1``, this    |\n   |           | rounds the number to ``p`` significant digits and then     |\n   |           | formats the result in either fixed-point format or in      |\n   |           | scientific notation, depending on its magnitude.  The      |\n   |           | precise rules are as follows: suppose that the result      |\n   |           | formatted with presentation type ``\'e\'`` and precision     |\n   |           | ``p-1`` would have exponent ``exp``.  Then if ``-4 <= exp  |\n   |           | < p``, the number is formatted with presentation type      |\n   |           | ``\'f\'`` and precision ``p-1-exp``. Otherwise, the number   |\n   |           | is formatted with presentation type ``\'e\'`` and precision  |\n   |           | ``p-1``. In both cases insignificant trailing zeros are    |\n   |           | removed from the significand, and the decimal point is     |\n   |           | also removed if there are no remaining digits following    |\n   |           | it.  Positive and negative infinity, positive and negative |\n   |           | zero, and nans, are formatted as ``inf``, ``-inf``, ``0``, |\n   |           | ``-0`` and ``nan`` respectively, regardless of the         |\n   |           | precision.  A precision of ``0`` is treated as equivalent  |\n   |           | to a precision of ``1``.                                   |\n   +-----------+------------------------------------------------------------+\n   | ``\'G\'``   | General format. Same as ``\'g\'`` except switches to ``\'E\'`` |\n   |           | if the number gets too large. The representations of       |\n   |           | infinity and NaN are uppercased, too.                      |\n   +-----------+------------------------------------------------------------+\n   | ``\'n\'``   | Number. This is the same as ``\'g\'``, except that it uses   |\n   |           | the current locale setting to insert the appropriate       |\n   |           | number separator characters.                               |\n   +-----------+------------------------------------------------------------+\n   | ``\'%\'``   | Percentage. Multiplies the number by 100 and displays in   |\n   |           | fixed (``\'f\'``) format, followed by a percent sign.        |\n   +-----------+------------------------------------------------------------+\n   | None      | Similar to ``\'g\'``, except with at least one digit past    |\n   |           | the decimal point and a default precision of 12. This is   |\n   |           | intended to match ``str()``, except you can add the other  |\n   |           | format modifiers.                                          |\n   +-----------+------------------------------------------------------------+\n\n\nFormat examples\n===============\n\nThis section contains examples of the new format syntax and comparison\nwith the old ``%``-formatting.\n\nIn most of the cases the syntax is similar to the old\n``%``-formatting, with the addition of the ``{}`` and with ``:`` used\ninstead of ``%``. For example, ``\'%03.2f\'`` can be translated to\n``\'{:03.2f}\'``.\n\nThe new format syntax also supports new and different options, shown\nin the follow examples.\n\nAccessing arguments by position:\n\n   >>> \'{0}, {1}, {2}\'.format(\'a\', \'b\', \'c\')\n   \'a, b, c\'\n   >>> \'{}, {}, {}\'.format(\'a\', \'b\', \'c\')  # 3.1+ only\n   \'a, b, c\'\n   >>> \'{2}, {1}, {0}\'.format(\'a\', \'b\', \'c\')\n   \'c, b, a\'\n   >>> \'{2}, {1}, {0}\'.format(*\'abc\')      # unpacking argument sequence\n   \'c, b, a\'\n   >>> \'{0}{1}{0}\'.format(\'abra\', \'cad\')   # arguments\' indices can be repeated\n   \'abracadabra\'\n\nAccessing arguments by name:\n\n   >>> \'Coordinates: {latitude}, {longitude}\'.format(latitude=\'37.24N\', longitude=\'-115.81W\')\n   \'Coordinates: 37.24N, -115.81W\'\n   >>> coord = {\'latitude\': \'37.24N\', \'longitude\': \'-115.81W\'}\n   >>> \'Coordinates: {latitude}, {longitude}\'.format(**coord)\n   \'Coordinates: 37.24N, -115.81W\'\n\nAccessing arguments\' attributes:\n\n   >>> c = 3-5j\n   >>> (\'The complex number {0} is formed from the real part {0.real} \'\n   ...  \'and the imaginary part {0.imag}.\').format(c)\n   \'The complex number (3-5j) is formed from the real part 3.0 and the imaginary part -5.0.\'\n   >>> class Point:\n   ...     def __init__(self, x, y):\n   ...         self.x, self.y = x, y\n   ...     def __str__(self):\n   ...         return \'Point({self.x}, {self.y})\'.format(self=self)\n   ...\n   >>> str(Point(4, 2))\n   \'Point(4, 2)\'\n\nAccessing arguments\' items:\n\n   >>> coord = (3, 5)\n   >>> \'X: {0[0]};  Y: {0[1]}\'.format(coord)\n   \'X: 3;  Y: 5\'\n\nReplacing ``%s`` and ``%r``:\n\n   >>> "repr() shows quotes: {!r}; str() doesn\'t: {!s}".format(\'test1\', \'test2\')\n   "repr() shows quotes: \'test1\'; str() doesn\'t: test2"\n\nAligning the text and specifying a width:\n\n   >>> \'{:<30}\'.format(\'left aligned\')\n   \'left aligned                  \'\n   >>> \'{:>30}\'.format(\'right aligned\')\n   \'                 right aligned\'\n   >>> \'{:^30}\'.format(\'centered\')\n   \'           centered           \'\n   >>> \'{:*^30}\'.format(\'centered\')  # use \'*\' as a fill char\n   \'***********centered***********\'\n\nReplacing ``%+f``, ``%-f``, and ``% f`` and specifying a sign:\n\n   >>> \'{:+f}; {:+f}\'.format(3.14, -3.14)  # show it always\n   \'+3.140000; -3.140000\'\n   >>> \'{: f}; {: f}\'.format(3.14, -3.14)  # show a space for positive numbers\n   \' 3.140000; -3.140000\'\n   >>> \'{:-f}; {:-f}\'.format(3.14, -3.14)  # show only the minus -- same as \'{:f}; {:f}\'\n   \'3.140000; -3.140000\'\n\nReplacing ``%x`` and ``%o`` and converting the value to different\nbases:\n\n   >>> # format also supports binary numbers\n   >>> "int: {0:d};  hex: {0:x};  oct: {0:o};  bin: {0:b}".format(42)\n   \'int: 42;  hex: 2a;  oct: 52;  bin: 101010\'\n   >>> # with 0x, 0o, or 0b as prefix:\n   >>> "int: {0:d};  hex: {0:#x};  oct: {0:#o};  bin: {0:#b}".format(42)\n   \'int: 42;  hex: 0x2a;  oct: 0o52;  bin: 0b101010\'\n\nUsing the comma as a thousands separator:\n\n   >>> \'{:,}\'.format(1234567890)\n   \'1,234,567,890\'\n\nExpressing a percentage:\n\n   >>> points = 19\n   >>> total = 22\n   >>> \'Correct answers: {:.2%}.\'.format(points/total)\n   \'Correct answers: 86.36%\'\n\nUsing type-specific formatting:\n\n   >>> import datetime\n   >>> d = datetime.datetime(2010, 7, 4, 12, 15, 58)\n   >>> \'{:%Y-%m-%d %H:%M:%S}\'.format(d)\n   \'2010-07-04 12:15:58\'\n\nNesting arguments and more complex examples:\n\n   >>> for align, text in zip(\'<^>\', [\'left\', \'center\', \'right\']):\n   ...     \'{0:{fill}{align}16}\'.format(text, fill=align, align=align)\n   ...\n   \'left<<<<<<<<<<<<\'\n   \'^^^^^center^^^^^\'\n   \'>>>>>>>>>>>right\'\n   >>>\n   >>> octets = [192, 168, 0, 1]\n   >>> \'{:02X}{:02X}{:02X}{:02X}\'.format(*octets)\n   \'C0A80001\'\n   >>> int(_, 16)\n   3232235521\n   >>>\n   >>> width = 5\n   >>> for num in range(5,12):\n   ...     for base in \'dXob\':\n   ...         print(\'{0:{width}{base}}\'.format(num, base=base, width=width), end=\' \')\n   ...     print()\n   ...\n       5     5     5   101\n       6     6     6   110\n       7     7     7   111\n       8     8    10  1000\n       9     9    11  1001\n      10     A    12  1010\n      11     B    13  1011\n',
  'function': '\nFunction definitions\n********************\n\nA function definition defines a user-defined function object (see\nsection *The standard type hierarchy*):\n\n   funcdef        ::= [decorators] "def" funcname "(" [parameter_list] ")" ["->" expression] ":" suite\n   decorators     ::= decorator+\n   decorator      ::= "@" dotted_name ["(" [argument_list [","]] ")"] NEWLINE\n   dotted_name    ::= identifier ("." identifier)*\n   parameter_list ::= (defparameter ",")*\n                      (  "*" [parameter] ("," defparameter)*\n                      [, "**" parameter]\n                      | "**" parameter\n                      | defparameter [","] )\n   parameter      ::= identifier [":" expression]\n   defparameter   ::= parameter ["=" expression]\n   funcname       ::= identifier\n\nA function definition is an executable statement.  Its execution binds\nthe function name in the current local namespace to a function object\n(a wrapper around the executable code for the function).  This\nfunction object contains a reference to the current global namespace\nas the global namespace to be used when the function is called.\n\nThe function definition does not execute the function body; this gets\nexecuted only when the function is called. [3]\n\nA function definition may be wrapped by one or more *decorator*\nexpressions. Decorator expressions are evaluated when the function is\ndefined, in the scope that contains the function definition.  The\nresult must be a callable, which is invoked with the function object\nas the only argument. The returned value is bound to the function name\ninstead of the function object.  Multiple decorators are applied in\nnested fashion. For example, the following code\n\n   @f1(arg)\n   @f2\n   def func(): pass\n\nis equivalent to\n\n   def func(): pass\n   func = f1(arg)(f2(func))\n\nWhen one or more parameters have the form *parameter* ``=``\n*expression*, the function is said to have "default parameter values."\nFor a parameter with a default value, the corresponding argument may\nbe omitted from a call, in which case the parameter\'s default value is\nsubstituted.  If a parameter has a default value, all following\nparameters up until the "``*``" must also have a default value ---\nthis is a syntactic restriction that is not expressed by the grammar.\n\n**Default parameter values are evaluated when the function definition\nis executed.** This means that the expression is evaluated once, when\nthe function is defined, and that that same "pre-computed" value is\nused for each call.  This is especially important to understand when a\ndefault parameter is a mutable object, such as a list or a dictionary:\nif the function modifies the object (e.g. by appending an item to a\nlist), the default value is in effect modified. This is generally not\nwhat was intended.  A way around this is to use ``None`` as the\ndefault, and explicitly test for it in the body of the function, e.g.:\n\n   def whats_on_the_telly(penguin=None):\n       if penguin is None:\n           penguin = []\n       penguin.append("property of the zoo")\n       return penguin\n\nFunction call semantics are described in more detail in section\n*Calls*. A function call always assigns values to all parameters\nmentioned in the parameter list, either from position arguments, from\nkeyword arguments, or from default values.  If the form\n"``*identifier``" is present, it is initialized to a tuple receiving\nany excess positional parameters, defaulting to the empty tuple.  If\nthe form "``**identifier``" is present, it is initialized to a new\ndictionary receiving any excess keyword arguments, defaulting to a new\nempty dictionary. Parameters after "``*``" or "``*identifier``" are\nkeyword-only parameters and may only be passed used keyword arguments.\n\nParameters may have annotations of the form "``: expression``"\nfollowing the parameter name.  Any parameter may have an annotation\neven those of the form ``*identifier`` or ``**identifier``.  Functions\nmay have "return" annotation of the form "``-> expression``" after the\nparameter list.  These annotations can be any valid Python expression\nand are evaluated when the function definition is executed.\nAnnotations may be evaluated in a different order than they appear in\nthe source code.  The presence of annotations does not change the\nsemantics of a function.  The annotation values are available as\nvalues of a dictionary keyed by the parameters\' names in the\n``__annotations__`` attribute of the function object.\n\nIt is also possible to create anonymous functions (functions not bound\nto a name), for immediate use in expressions.  This uses lambda forms,\ndescribed in section *Lambdas*.  Note that the lambda form is merely a\nshorthand for a simplified function definition; a function defined in\na "``def``" statement can be passed around or assigned to another name\njust like a function defined by a lambda form.  The "``def``" form is\nactually more powerful since it allows the execution of multiple\nstatements and annotations.\n\n**Programmer\'s note:** Functions are first-class objects.  A "``def``"\nform executed inside a function definition defines a local function\nthat can be returned or passed around.  Free variables used in the\nnested function can access the local variables of the function\ncontaining the def.  See section *Naming and binding* for details.\n',
  'global': '\nThe ``global`` statement\n************************\n\n   global_stmt ::= "global" identifier ("," identifier)*\n\nThe ``global`` statement is a declaration which holds for the entire\ncurrent code block.  It means that the listed identifiers are to be\ninterpreted as globals.  It would be impossible to assign to a global\nvariable without ``global``, although free variables may refer to\nglobals without being declared global.\n\nNames listed in a ``global`` statement must not be used in the same\ncode block textually preceding that ``global`` statement.\n\nNames listed in a ``global`` statement must not be defined as formal\nparameters or in a ``for`` loop control target, ``class`` definition,\nfunction definition, or ``import`` statement.\n\n**CPython implementation detail:** The current implementation does not\nenforce the latter two restrictions, but programs should not abuse\nthis freedom, as future implementations may enforce them or silently\nchange the meaning of the program.\n\n**Programmer\'s note:** the ``global`` is a directive to the parser.\nIt applies only to code parsed at the same time as the ``global``\nstatement. In particular, a ``global`` statement contained in a string\nor code object supplied to the built-in ``exec()`` function does not\naffect the code block *containing* the function call, and code\ncontained in such a string is unaffected by ``global`` statements in\nthe code containing the function call.  The same applies to the\n``eval()`` and ``compile()`` functions.\n',
  'id-classes': '\nReserved classes of identifiers\n*******************************\n\nCertain classes of identifiers (besides keywords) have special\nmeanings.  These classes are identified by the patterns of leading and\ntrailing underscore characters:\n\n``_*``\n   Not imported by ``from module import *``.  The special identifier\n   ``_`` is used in the interactive interpreter to store the result of\n   the last evaluation; it is stored in the ``builtins`` module.  When\n   not in interactive mode, ``_`` has no special meaning and is not\n   defined. See section *The import statement*.\n\n   Note: The name ``_`` is often used in conjunction with\n     internationalization; refer to the documentation for the\n     ``gettext`` module for more information on this convention.\n\n``__*__``\n   System-defined names. These names are defined by the interpreter\n   and its implementation (including the standard library).  Current\n   system names are discussed in the *Special method names* section\n   and elsewhere.  More will likely be defined in future versions of\n   Python.  *Any* use of ``__*__`` names, in any context, that does\n   not follow explicitly documented use, is subject to breakage\n   without warning.\n\n``__*``\n   Class-private names.  Names in this category, when used within the\n   context of a class definition, are re-written to use a mangled form\n   to help avoid name clashes between "private" attributes of base and\n   derived classes. See section *Identifiers (Names)*.\n',
@@ -67,10 +67,10 @@
  'try': '\nThe ``try`` statement\n*********************\n\nThe ``try`` statement specifies exception handlers and/or cleanup code\nfor a group of statements:\n\n   try_stmt  ::= try1_stmt | try2_stmt\n   try1_stmt ::= "try" ":" suite\n                 ("except" [expression ["as" target]] ":" suite)+\n                 ["else" ":" suite]\n                 ["finally" ":" suite]\n   try2_stmt ::= "try" ":" suite\n                 "finally" ":" suite\n\nThe ``except`` clause(s) specify one or more exception handlers. When\nno exception occurs in the ``try`` clause, no exception handler is\nexecuted. When an exception occurs in the ``try`` suite, a search for\nan exception handler is started.  This search inspects the except\nclauses in turn until one is found that matches the exception.  An\nexpression-less except clause, if present, must be last; it matches\nany exception.  For an except clause with an expression, that\nexpression is evaluated, and the clause matches the exception if the\nresulting object is "compatible" with the exception.  An object is\ncompatible with an exception if it is the class or a base class of the\nexception object or a tuple containing an item compatible with the\nexception.\n\nIf no except clause matches the exception, the search for an exception\nhandler continues in the surrounding code and on the invocation stack.\n[1]\n\nIf the evaluation of an expression in the header of an except clause\nraises an exception, the original search for a handler is canceled and\na search starts for the new exception in the surrounding code and on\nthe call stack (it is treated as if the entire ``try`` statement\nraised the exception).\n\nWhen a matching except clause is found, the exception is assigned to\nthe target specified after the ``as`` keyword in that except clause,\nif present, and the except clause\'s suite is executed.  All except\nclauses must have an executable block.  When the end of this block is\nreached, execution continues normally after the entire try statement.\n(This means that if two nested handlers exist for the same exception,\nand the exception occurs in the try clause of the inner handler, the\nouter handler will not handle the exception.)\n\nWhen an exception has been assigned using ``as target``, it is cleared\nat the end of the except clause.  This is as if\n\n   except E as N:\n       foo\n\nwas translated to\n\n   except E as N:\n       try:\n           foo\n       finally:\n           del N\n\nThis means the exception must be assigned to a different name to be\nable to refer to it after the except clause.  Exceptions are cleared\nbecause with the traceback attached to them, they form a reference\ncycle with the stack frame, keeping all locals in that frame alive\nuntil the next garbage collection occurs.\n\nBefore an except clause\'s suite is executed, details about the\nexception are stored in the ``sys`` module and can be access via\n``sys.exc_info()``. ``sys.exc_info()`` returns a 3-tuple consisting of\nthe exception class, the exception instance and a traceback object\n(see section *The standard type hierarchy*) identifying the point in\nthe program where the exception occurred.  ``sys.exc_info()`` values\nare restored to their previous values (before the call) when returning\nfrom a function that handled an exception.\n\nThe optional ``else`` clause is executed if and when control flows off\nthe end of the ``try`` clause. [2] Exceptions in the ``else`` clause\nare not handled by the preceding ``except`` clauses.\n\nIf ``finally`` is present, it specifies a \'cleanup\' handler.  The\n``try`` clause is executed, including any ``except`` and ``else``\nclauses.  If an exception occurs in any of the clauses and is not\nhandled, the exception is temporarily saved. The ``finally`` clause is\nexecuted.  If there is a saved exception, it is re-raised at the end\nof the ``finally`` clause. If the ``finally`` clause raises another\nexception or executes a ``return`` or ``break`` statement, the saved\nexception is lost.  The exception information is not available to the\nprogram during execution of the ``finally`` clause.\n\nWhen a ``return``, ``break`` or ``continue`` statement is executed in\nthe ``try`` suite of a ``try``...``finally`` statement, the\n``finally`` clause is also executed \'on the way out.\' A ``continue``\nstatement is illegal in the ``finally`` clause. (The reason is a\nproblem with the current implementation --- this restriction may be\nlifted in the future).\n\nAdditional information on exceptions can be found in section\n*Exceptions*, and information on using the ``raise`` statement to\ngenerate exceptions may be found in section *The raise statement*.\n',
  'types': '\nThe standard type hierarchy\n***************************\n\nBelow is a list of the types that are built into Python.  Extension\nmodules (written in C, Java, or other languages, depending on the\nimplementation) can define additional types.  Future versions of\nPython may add types to the type hierarchy (e.g., rational numbers,\nefficiently stored arrays of integers, etc.), although such additions\nwill often be provided via the standard library instead.\n\nSome of the type descriptions below contain a paragraph listing\n\'special attributes.\'  These are attributes that provide access to the\nimplementation and are not intended for general use.  Their definition\nmay change in the future.\n\nNone\n   This type has a single value.  There is a single object with this\n   value. This object is accessed through the built-in name ``None``.\n   It is used to signify the absence of a value in many situations,\n   e.g., it is returned from functions that don\'t explicitly return\n   anything. Its truth value is false.\n\nNotImplemented\n   This type has a single value.  There is a single object with this\n   value. This object is accessed through the built-in name\n   ``NotImplemented``. Numeric methods and rich comparison methods may\n   return this value if they do not implement the operation for the\n   operands provided.  (The interpreter will then try the reflected\n   operation, or some other fallback, depending on the operator.)  Its\n   truth value is true.\n\nEllipsis\n   This type has a single value.  There is a single object with this\n   value. This object is accessed through the literal ``...`` or the\n   built-in name ``Ellipsis``.  Its truth value is true.\n\n``numbers.Number``\n   These are created by numeric literals and returned as results by\n   arithmetic operators and arithmetic built-in functions.  Numeric\n   objects are immutable; once created their value never changes.\n   Python numbers are of course strongly related to mathematical\n   numbers, but subject to the limitations of numerical representation\n   in computers.\n\n   Python distinguishes between integers, floating point numbers, and\n   complex numbers:\n\n   ``numbers.Integral``\n      These represent elements from the mathematical set of integers\n      (positive and negative).\n\n      There are two types of integers:\n\n      Integers (``int``)\n\n         These represent numbers in an unlimited range, subject to\n         available (virtual) memory only.  For the purpose of shift\n         and mask operations, a binary representation is assumed, and\n         negative numbers are represented in a variant of 2\'s\n         complement which gives the illusion of an infinite string of\n         sign bits extending to the left.\n\n      Booleans (``bool``)\n         These represent the truth values False and True.  The two\n         objects representing the values False and True are the only\n         Boolean objects. The Boolean type is a subtype of the integer\n         type, and Boolean values behave like the values 0 and 1,\n         respectively, in almost all contexts, the exception being\n         that when converted to a string, the strings ``"False"`` or\n         ``"True"`` are returned, respectively.\n\n      The rules for integer representation are intended to give the\n      most meaningful interpretation of shift and mask operations\n      involving negative integers.\n\n   ``numbers.Real`` (``float``)\n      These represent machine-level double precision floating point\n      numbers. You are at the mercy of the underlying machine\n      architecture (and C or Java implementation) for the accepted\n      range and handling of overflow. Python does not support single-\n      precision floating point numbers; the savings in processor and\n      memory usage that are usually the reason for using these is\n      dwarfed by the overhead of using objects in Python, so there is\n      no reason to complicate the language with two kinds of floating\n      point numbers.\n\n   ``numbers.Complex`` (``complex``)\n      These represent complex numbers as a pair of machine-level\n      double precision floating point numbers.  The same caveats apply\n      as for floating point numbers. The real and imaginary parts of a\n      complex number ``z`` can be retrieved through the read-only\n      attributes ``z.real`` and ``z.imag``.\n\nSequences\n   These represent finite ordered sets indexed by non-negative\n   numbers. The built-in function ``len()`` returns the number of\n   items of a sequence. When the length of a sequence is *n*, the\n   index set contains the numbers 0, 1, ..., *n*-1.  Item *i* of\n   sequence *a* is selected by ``a[i]``.\n\n   Sequences also support slicing: ``a[i:j]`` selects all items with\n   index *k* such that *i* ``<=`` *k* ``<`` *j*.  When used as an\n   expression, a slice is a sequence of the same type.  This implies\n   that the index set is renumbered so that it starts at 0.\n\n   Some sequences also support "extended slicing" with a third "step"\n   parameter: ``a[i:j:k]`` selects all items of *a* with index *x*\n   where ``x = i + n*k``, *n* ``>=`` ``0`` and *i* ``<=`` *x* ``<``\n   *j*.\n\n   Sequences are distinguished according to their mutability:\n\n   Immutable sequences\n      An object of an immutable sequence type cannot change once it is\n      created.  (If the object contains references to other objects,\n      these other objects may be mutable and may be changed; however,\n      the collection of objects directly referenced by an immutable\n      object cannot change.)\n\n      The following types are immutable sequences:\n\n      Strings\n         The items of a string object are Unicode code units.  A\n         Unicode code unit is represented by a string object of one\n         item and can hold either a 16-bit or 32-bit value\n         representing a Unicode ordinal (the maximum value for the\n         ordinal is given in ``sys.maxunicode``, and depends on how\n         Python is configured at compile time).  Surrogate pairs may\n         be present in the Unicode object, and will be reported as two\n         separate items.  The built-in functions ``chr()`` and\n         ``ord()`` convert between code units and nonnegative integers\n         representing the Unicode ordinals as defined in the Unicode\n         Standard 3.0. Conversion from and to other encodings are\n         possible through the string method ``encode()``.\n\n      Tuples\n         The items of a tuple are arbitrary Python objects. Tuples of\n         two or more items are formed by comma-separated lists of\n         expressions.  A tuple of one item (a \'singleton\') can be\n         formed by affixing a comma to an expression (an expression by\n         itself does not create a tuple, since parentheses must be\n         usable for grouping of expressions).  An empty tuple can be\n         formed by an empty pair of parentheses.\n\n      Bytes\n         A bytes object is an immutable array.  The items are 8-bit\n         bytes, represented by integers in the range 0 <= x < 256.\n         Bytes literals (like ``b\'abc\'`` and the built-in function\n         ``bytes()`` can be used to construct bytes objects.  Also,\n         bytes objects can be decoded to strings via the ``decode()``\n         method.\n\n   Mutable sequences\n      Mutable sequences can be changed after they are created.  The\n      subscription and slicing notations can be used as the target of\n      assignment and ``del`` (delete) statements.\n\n      There are currently two intrinsic mutable sequence types:\n\n      Lists\n         The items of a list are arbitrary Python objects.  Lists are\n         formed by placing a comma-separated list of expressions in\n         square brackets. (Note that there are no special cases needed\n         to form lists of length 0 or 1.)\n\n      Byte Arrays\n         A bytearray object is a mutable array. They are created by\n         the built-in ``bytearray()`` constructor.  Aside from being\n         mutable (and hence unhashable), byte arrays otherwise provide\n         the same interface and functionality as immutable bytes\n         objects.\n\n      The extension module ``array`` provides an additional example of\n      a mutable sequence type, as does the ``collections`` module.\n\nSet types\n   These represent unordered, finite sets of unique, immutable\n   objects. As such, they cannot be indexed by any subscript. However,\n   they can be iterated over, and the built-in function ``len()``\n   returns the number of items in a set. Common uses for sets are fast\n   membership testing, removing duplicates from a sequence, and\n   computing mathematical operations such as intersection, union,\n   difference, and symmetric difference.\n\n   For set elements, the same immutability rules apply as for\n   dictionary keys. Note that numeric types obey the normal rules for\n   numeric comparison: if two numbers compare equal (e.g., ``1`` and\n   ``1.0``), only one of them can be contained in a set.\n\n   There are currently two intrinsic set types:\n\n   Sets\n      These represent a mutable set. They are created by the built-in\n      ``set()`` constructor and can be modified afterwards by several\n      methods, such as ``add()``.\n\n   Frozen sets\n      These represent an immutable set.  They are created by the\n      built-in ``frozenset()`` constructor.  As a frozenset is\n      immutable and *hashable*, it can be used again as an element of\n      another set, or as a dictionary key.\n\nMappings\n   These represent finite sets of objects indexed by arbitrary index\n   sets. The subscript notation ``a[k]`` selects the item indexed by\n   ``k`` from the mapping ``a``; this can be used in expressions and\n   as the target of assignments or ``del`` statements. The built-in\n   function ``len()`` returns the number of items in a mapping.\n\n   There is currently a single intrinsic mapping type:\n\n   Dictionaries\n      These represent finite sets of objects indexed by nearly\n      arbitrary values.  The only types of values not acceptable as\n      keys are values containing lists or dictionaries or other\n      mutable types that are compared by value rather than by object\n      identity, the reason being that the efficient implementation of\n      dictionaries requires a key\'s hash value to remain constant.\n      Numeric types used for keys obey the normal rules for numeric\n      comparison: if two numbers compare equal (e.g., ``1`` and\n      ``1.0``) then they can be used interchangeably to index the same\n      dictionary entry.\n\n      Dictionaries are mutable; they can be created by the ``{...}``\n      notation (see section *Dictionary displays*).\n\n      The extension modules ``dbm.ndbm`` and ``dbm.gnu`` provide\n      additional examples of mapping types, as does the\n      ``collections`` module.\n\nCallable types\n   These are the types to which the function call operation (see\n   section *Calls*) can be applied:\n\n   User-defined functions\n      A user-defined function object is created by a function\n      definition (see section *Function definitions*).  It should be\n      called with an argument list containing the same number of items\n      as the function\'s formal parameter list.\n\n      Special attributes:\n\n      +---------------------------+---------------------------------+-------------+\n      | Attribute                 | Meaning                         |             |\n      +===========================+=================================+=============+\n      | ``__doc__``               | The function\'s documentation    | Writable    |\n      |                           | string, or ``None`` if          |             |\n      |                           | unavailable                     |             |\n      +---------------------------+---------------------------------+-------------+\n      | ``__name__``              | The function\'s name             | Writable    |\n      +---------------------------+---------------------------------+-------------+\n      | ``__module__``            | The name of the module the      | Writable    |\n      |                           | function was defined in, or     |             |\n      |                           | ``None`` if unavailable.        |             |\n      +---------------------------+---------------------------------+-------------+\n      | ``__defaults__``          | A tuple containing default      | Writable    |\n      |                           | argument values for those       |             |\n      |                           | arguments that have defaults,   |             |\n      |                           | or ``None`` if no arguments     |             |\n      |                           | have a default value            |             |\n      +---------------------------+---------------------------------+-------------+\n      | ``__code__``              | The code object representing    | Writable    |\n      |                           | the compiled function body.     |             |\n      +---------------------------+---------------------------------+-------------+\n      | ``__globals__``           | A reference to the dictionary   | Read-only   |\n      |                           | that holds the function\'s       |             |\n      |                           | global variables --- the global |             |\n      |                           | namespace of the module in      |             |\n      |                           | which the function was defined. |             |\n      +---------------------------+---------------------------------+-------------+\n      | ``__dict__``              | The namespace supporting        | Writable    |\n      |                           | arbitrary function attributes.  |             |\n      +---------------------------+---------------------------------+-------------+\n      | ``__closure__``           | ``None`` or a tuple of cells    | Read-only   |\n      |                           | that contain bindings for the   |             |\n      |                           | function\'s free variables.      |             |\n      +---------------------------+---------------------------------+-------------+\n      | ``__annotations__``       | A dict containing annotations   | Writable    |\n      |                           | of parameters.  The keys of the |             |\n      |                           | dict are the parameter names,   |             |\n      |                           | or ``\'return\'`` for the return  |             |\n      |                           | annotation, if provided.        |             |\n      +---------------------------+---------------------------------+-------------+\n      | ``__kwdefaults__``        | A dict containing defaults for  | Writable    |\n      |                           | keyword-only parameters.        |             |\n      +---------------------------+---------------------------------+-------------+\n\n      Most of the attributes labelled "Writable" check the type of the\n      assigned value.\n\n      Function objects also support getting and setting arbitrary\n      attributes, which can be used, for example, to attach metadata\n      to functions.  Regular attribute dot-notation is used to get and\n      set such attributes. *Note that the current implementation only\n      supports function attributes on user-defined functions. Function\n      attributes on built-in functions may be supported in the\n      future.*\n\n      Additional information about a function\'s definition can be\n      retrieved from its code object; see the description of internal\n      types below.\n\n   Instance methods\n      An instance method object combines a class, a class instance and\n      any callable object (normally a user-defined function).\n\n      Special read-only attributes: ``__self__`` is the class instance\n      object, ``__func__`` is the function object; ``__doc__`` is the\n      method\'s documentation (same as ``__func__.__doc__``);\n      ``__name__`` is the method name (same as ``__func__.__name__``);\n      ``__module__`` is the name of the module the method was defined\n      in, or ``None`` if unavailable.\n\n      Methods also support accessing (but not setting) the arbitrary\n      function attributes on the underlying function object.\n\n      User-defined method objects may be created when getting an\n      attribute of a class (perhaps via an instance of that class), if\n      that attribute is a user-defined function object or a class\n      method object.\n\n      When an instance method object is created by retrieving a user-\n      defined function object from a class via one of its instances,\n      its ``__self__`` attribute is the instance, and the method\n      object is said to be bound.  The new method\'s ``__func__``\n      attribute is the original function object.\n\n      When a user-defined method object is created by retrieving\n      another method object from a class or instance, the behaviour is\n      the same as for a function object, except that the ``__func__``\n      attribute of the new instance is not the original method object\n      but its ``__func__`` attribute.\n\n      When an instance method object is created by retrieving a class\n      method object from a class or instance, its ``__self__``\n      attribute is the class itself, and its ``__func__`` attribute is\n      the function object underlying the class method.\n\n      When an instance method object is called, the underlying\n      function (``__func__``) is called, inserting the class instance\n      (``__self__``) in front of the argument list.  For instance,\n      when ``C`` is a class which contains a definition for a function\n      ``f()``, and ``x`` is an instance of ``C``, calling ``x.f(1)``\n      is equivalent to calling ``C.f(x, 1)``.\n\n      When an instance method object is derived from a class method\n      object, the "class instance" stored in ``__self__`` will\n      actually be the class itself, so that calling either ``x.f(1)``\n      or ``C.f(1)`` is equivalent to calling ``f(C,1)`` where ``f`` is\n      the underlying function.\n\n      Note that the transformation from function object to instance\n      method object happens each time the attribute is retrieved from\n      the instance.  In some cases, a fruitful optimization is to\n      assign the attribute to a local variable and call that local\n      variable. Also notice that this transformation only happens for\n      user-defined functions; other callable objects (and all non-\n      callable objects) are retrieved without transformation.  It is\n      also important to note that user-defined functions which are\n      attributes of a class instance are not converted to bound\n      methods; this *only* happens when the function is an attribute\n      of the class.\n\n   Generator functions\n      A function or method which uses the ``yield`` statement (see\n      section *The yield statement*) is called a *generator function*.\n      Such a function, when called, always returns an iterator object\n      which can be used to execute the body of the function:  calling\n      the iterator\'s ``__next__()`` method will cause the function to\n      execute until it provides a value using the ``yield`` statement.\n      When the function executes a ``return`` statement or falls off\n      the end, a ``StopIteration`` exception is raised and the\n      iterator will have reached the end of the set of values to be\n      returned.\n\n   Built-in functions\n      A built-in function object is a wrapper around a C function.\n      Examples of built-in functions are ``len()`` and ``math.sin()``\n      (``math`` is a standard built-in module). The number and type of\n      the arguments are determined by the C function. Special read-\n      only attributes: ``__doc__`` is the function\'s documentation\n      string, or ``None`` if unavailable; ``__name__`` is the\n      function\'s name; ``__self__`` is set to ``None`` (but see the\n      next item); ``__module__`` is the name of the module the\n      function was defined in or ``None`` if unavailable.\n\n   Built-in methods\n      This is really a different disguise of a built-in function, this\n      time containing an object passed to the C function as an\n      implicit extra argument.  An example of a built-in method is\n      ``alist.append()``, assuming *alist* is a list object. In this\n      case, the special read-only attribute ``__self__`` is set to the\n      object denoted by *alist*.\n\n   Classes\n      Classes are callable.  These objects normally act as factories\n      for new instances of themselves, but variations are possible for\n      class types that override ``__new__()``.  The arguments of the\n      call are passed to ``__new__()`` and, in the typical case, to\n      ``__init__()`` to initialize the new instance.\n\n   Class Instances\n      Instances of arbitrary classes can be made callable by defining\n      a ``__call__()`` method in their class.\n\nModules\n   Modules are imported by the ``import`` statement (see section *The\n   import statement*). A module object has a namespace implemented by\n   a dictionary object (this is the dictionary referenced by the\n   __globals__ attribute of functions defined in the module).\n   Attribute references are translated to lookups in this dictionary,\n   e.g., ``m.x`` is equivalent to ``m.__dict__["x"]``. A module object\n   does not contain the code object used to initialize the module\n   (since it isn\'t needed once the initialization is done).\n\n   Attribute assignment updates the module\'s namespace dictionary,\n   e.g., ``m.x = 1`` is equivalent to ``m.__dict__["x"] = 1``.\n\n   Special read-only attribute: ``__dict__`` is the module\'s namespace\n   as a dictionary object.\n\n   **CPython implementation detail:** Because of the way CPython\n   clears module dictionaries, the module dictionary will be cleared\n   when the module falls out of scope even if the dictionary still has\n   live references.  To avoid this, copy the dictionary or keep the\n   module around while using its dictionary directly.\n\n   Predefined (writable) attributes: ``__name__`` is the module\'s\n   name; ``__doc__`` is the module\'s documentation string, or ``None``\n   if unavailable; ``__file__`` is the pathname of the file from which\n   the module was loaded, if it was loaded from a file. The\n   ``__file__`` attribute is not present for C modules that are\n   statically linked into the interpreter; for extension modules\n   loaded dynamically from a shared library, it is the pathname of the\n   shared library file.\n\nCustom classes\n   Custom class types are typically created by class definitions (see\n   section *Class definitions*).  A class has a namespace implemented\n   by a dictionary object. Class attribute references are translated\n   to lookups in this dictionary, e.g., ``C.x`` is translated to\n   ``C.__dict__["x"]`` (although there are a number of hooks which\n   allow for other means of locating attributes). When the attribute\n   name is not found there, the attribute search continues in the base\n   classes. This search of the base classes uses the C3 method\n   resolution order which behaves correctly even in the presence of\n   \'diamond\' inheritance structures where there are multiple\n   inheritance paths leading back to a common ancestor. Additional\n   details on the C3 MRO used by Python can be found in the\n   documentation accompanying the 2.3 release at\n   http://www.python.org/download/releases/2.3/mro/.\n\n   When a class attribute reference (for class ``C``, say) would yield\n   a class method object, it is transformed into an instance method\n   object whose ``__self__`` attributes is ``C``.  When it would yield\n   a static method object, it is transformed into the object wrapped\n   by the static method object. See section *Implementing Descriptors*\n   for another way in which attributes retrieved from a class may\n   differ from those actually contained in its ``__dict__``.\n\n   Class attribute assignments update the class\'s dictionary, never\n   the dictionary of a base class.\n\n   A class object can be called (see above) to yield a class instance\n   (see below).\n\n   Special attributes: ``__name__`` is the class name; ``__module__``\n   is the module name in which the class was defined; ``__dict__`` is\n   the dictionary containing the class\'s namespace; ``__bases__`` is a\n   tuple (possibly empty or a singleton) containing the base classes,\n   in the order of their occurrence in the base class list;\n   ``__doc__`` is the class\'s documentation string, or None if\n   undefined.\n\nClass instances\n   A class instance is created by calling a class object (see above).\n   A class instance has a namespace implemented as a dictionary which\n   is the first place in which attribute references are searched.\n   When an attribute is not found there, and the instance\'s class has\n   an attribute by that name, the search continues with the class\n   attributes.  If a class attribute is found that is a user-defined\n   function object, it is transformed into an instance method object\n   whose ``__self__`` attribute is the instance.  Static method and\n   class method objects are also transformed; see above under\n   "Classes".  See section *Implementing Descriptors* for another way\n   in which attributes of a class retrieved via its instances may\n   differ from the objects actually stored in the class\'s\n   ``__dict__``.  If no class attribute is found, and the object\'s\n   class has a ``__getattr__()`` method, that is called to satisfy the\n   lookup.\n\n   Attribute assignments and deletions update the instance\'s\n   dictionary, never a class\'s dictionary.  If the class has a\n   ``__setattr__()`` or ``__delattr__()`` method, this is called\n   instead of updating the instance dictionary directly.\n\n   Class instances can pretend to be numbers, sequences, or mappings\n   if they have methods with certain special names.  See section\n   *Special method names*.\n\n   Special attributes: ``__dict__`` is the attribute dictionary;\n   ``__class__`` is the instance\'s class.\n\nI/O objects (also known as file objects)\n   A *file object* represents an open file.  Various shortcuts are\n   available to create file objects: the ``open()`` built-in function,\n   and also ``os.popen()``, ``os.fdopen()``, and the ``makefile()``\n   method of socket objects (and perhaps by other functions or methods\n   provided by extension modules).\n\n   The objects ``sys.stdin``, ``sys.stdout`` and ``sys.stderr`` are\n   initialized to file objects corresponding to the interpreter\'s\n   standard input, output and error streams; they are all open in text\n   mode and therefore follow the interface defined by the\n   ``io.TextIOBase`` abstract class.\n\nInternal types\n   A few types used internally by the interpreter are exposed to the\n   user. Their definitions may change with future versions of the\n   interpreter, but they are mentioned here for completeness.\n\n   Code objects\n      Code objects represent *byte-compiled* executable Python code,\n      or *bytecode*. The difference between a code object and a\n      function object is that the function object contains an explicit\n      reference to the function\'s globals (the module in which it was\n      defined), while a code object contains no context; also the\n      default argument values are stored in the function object, not\n      in the code object (because they represent values calculated at\n      run-time).  Unlike function objects, code objects are immutable\n      and contain no references (directly or indirectly) to mutable\n      objects.\n\n      Special read-only attributes: ``co_name`` gives the function\n      name; ``co_argcount`` is the number of positional arguments\n      (including arguments with default values); ``co_nlocals`` is the\n      number of local variables used by the function (including\n      arguments); ``co_varnames`` is a tuple containing the names of\n      the local variables (starting with the argument names);\n      ``co_cellvars`` is a tuple containing the names of local\n      variables that are referenced by nested functions;\n      ``co_freevars`` is a tuple containing the names of free\n      variables; ``co_code`` is a string representing the sequence of\n      bytecode instructions; ``co_consts`` is a tuple containing the\n      literals used by the bytecode; ``co_names`` is a tuple\n      containing the names used by the bytecode; ``co_filename`` is\n      the filename from which the code was compiled;\n      ``co_firstlineno`` is the first line number of the function;\n      ``co_lnotab`` is a string encoding the mapping from bytecode\n      offsets to line numbers (for details see the source code of the\n      interpreter); ``co_stacksize`` is the required stack size\n      (including local variables); ``co_flags`` is an integer encoding\n      a number of flags for the interpreter.\n\n      The following flag bits are defined for ``co_flags``: bit\n      ``0x04`` is set if the function uses the ``*arguments`` syntax\n      to accept an arbitrary number of positional arguments; bit\n      ``0x08`` is set if the function uses the ``**keywords`` syntax\n      to accept arbitrary keyword arguments; bit ``0x20`` is set if\n      the function is a generator.\n\n      Future feature declarations (``from __future__ import\n      division``) also use bits in ``co_flags`` to indicate whether a\n      code object was compiled with a particular feature enabled: bit\n      ``0x2000`` is set if the function was compiled with future\n      division enabled; bits ``0x10`` and ``0x1000`` were used in\n      earlier versions of Python.\n\n      Other bits in ``co_flags`` are reserved for internal use.\n\n      If a code object represents a function, the first item in\n      ``co_consts`` is the documentation string of the function, or\n      ``None`` if undefined.\n\n   Frame objects\n      Frame objects represent execution frames.  They may occur in\n      traceback objects (see below).\n\n      Special read-only attributes: ``f_back`` is to the previous\n      stack frame (towards the caller), or ``None`` if this is the\n      bottom stack frame; ``f_code`` is the code object being executed\n      in this frame; ``f_locals`` is the dictionary used to look up\n      local variables; ``f_globals`` is used for global variables;\n      ``f_builtins`` is used for built-in (intrinsic) names;\n      ``f_lasti`` gives the precise instruction (this is an index into\n      the bytecode string of the code object).\n\n      Special writable attributes: ``f_trace``, if not ``None``, is a\n      function called at the start of each source code line (this is\n      used by the debugger); ``f_lineno`` is the current line number\n      of the frame --- writing to this from within a trace function\n      jumps to the given line (only for the bottom-most frame).  A\n      debugger can implement a Jump command (aka Set Next Statement)\n      by writing to f_lineno.\n\n   Traceback objects\n      Traceback objects represent a stack trace of an exception.  A\n      traceback object is created when an exception occurs.  When the\n      search for an exception handler unwinds the execution stack, at\n      each unwound level a traceback object is inserted in front of\n      the current traceback.  When an exception handler is entered,\n      the stack trace is made available to the program. (See section\n      *The try statement*.) It is accessible as the third item of the\n      tuple returned by ``sys.exc_info()``. When the program contains\n      no suitable handler, the stack trace is written (nicely\n      formatted) to the standard error stream; if the interpreter is\n      interactive, it is also made available to the user as\n      ``sys.last_traceback``.\n\n      Special read-only attributes: ``tb_next`` is the next level in\n      the stack trace (towards the frame where the exception\n      occurred), or ``None`` if there is no next level; ``tb_frame``\n      points to the execution frame of the current level;\n      ``tb_lineno`` gives the line number where the exception\n      occurred; ``tb_lasti`` indicates the precise instruction.  The\n      line number and last instruction in the traceback may differ\n      from the line number of its frame object if the exception\n      occurred in a ``try`` statement with no matching except clause\n      or with a finally clause.\n\n   Slice objects\n      Slice objects are used to represent slices for ``__getitem__()``\n      methods.  They are also created by the built-in ``slice()``\n      function.\n\n      Special read-only attributes: ``start`` is the lower bound;\n      ``stop`` is the upper bound; ``step`` is the step value; each is\n      ``None`` if omitted. These attributes can have any type.\n\n      Slice objects support one method:\n\n      slice.indices(self, length)\n\n         This method takes a single integer argument *length* and\n         computes information about the slice that the slice object\n         would describe if applied to a sequence of *length* items.\n         It returns a tuple of three integers; respectively these are\n         the *start* and *stop* indices and the *step* or stride\n         length of the slice. Missing or out-of-bounds indices are\n         handled in a manner consistent with regular slices.\n\n   Static method objects\n      Static method objects provide a way of defeating the\n      transformation of function objects to method objects described\n      above. A static method object is a wrapper around any other\n      object, usually a user-defined method object. When a static\n      method object is retrieved from a class or a class instance, the\n      object actually returned is the wrapped object, which is not\n      subject to any further transformation. Static method objects are\n      not themselves callable, although the objects they wrap usually\n      are. Static method objects are created by the built-in\n      ``staticmethod()`` constructor.\n\n   Class method objects\n      A class method object, like a static method object, is a wrapper\n      around another object that alters the way in which that object\n      is retrieved from classes and class instances. The behaviour of\n      class method objects upon such retrieval is described above,\n      under "User-defined methods". Class method objects are created\n      by the built-in ``classmethod()`` constructor.\n',
  'typesfunctions': '\nFunctions\n*********\n\nFunction objects are created by function definitions.  The only\noperation on a function object is to call it: ``func(argument-list)``.\n\nThere are really two flavors of function objects: built-in functions\nand user-defined functions.  Both support the same operation (to call\nthe function), but the implementation is different, hence the\ndifferent object types.\n\nSee *Function definitions* for more information.\n',
- 'typesmapping': '\nMapping Types --- ``dict``\n**************************\n\nA *mapping* object maps *hashable* values to arbitrary objects.\nMappings are mutable objects.  There is currently only one standard\nmapping type, the *dictionary*.  (For other containers see the built\nin ``list``, ``set``, and ``tuple`` classes, and the ``collections``\nmodule.)\n\nA dictionary\'s keys are *almost* arbitrary values.  Values that are\nnot *hashable*, that is, values containing lists, dictionaries or\nother mutable types (that are compared by value rather than by object\nidentity) may not be used as keys.  Numeric types used for keys obey\nthe normal rules for numeric comparison: if two numbers compare equal\n(such as ``1`` and ``1.0``) then they can be used interchangeably to\nindex the same dictionary entry.  (Note however, that since computers\nstore floating-point numbers as approximations it is usually unwise to\nuse them as dictionary keys.)\n\nDictionaries can be created by placing a comma-separated list of\n``key: value`` pairs within braces, for example: ``{\'jack\': 4098,\n\'sjoerd\': 4127}`` or ``{4098: \'jack\', 4127: \'sjoerd\'}``, or by the\n``dict`` constructor.\n\nclass class dict([arg])\n\n   Return a new dictionary initialized from an optional positional\n   argument or from a set of keyword arguments.  If no arguments are\n   given, return a new empty dictionary.  If the positional argument\n   *arg* is a mapping object, return a dictionary mapping the same\n   keys to the same values as does the mapping object.  Otherwise the\n   positional argument must be a sequence, a container that supports\n   iteration, or an iterator object.  The elements of the argument\n   must each also be of one of those kinds, and each must in turn\n   contain exactly two objects.  The first is used as a key in the new\n   dictionary, and the second as the key\'s value.  If a given key is\n   seen more than once, the last value associated with it is retained\n   in the new dictionary.\n\n   If keyword arguments are given, the keywords themselves with their\n   associated values are added as items to the dictionary.  If a key\n   is specified both in the positional argument and as a keyword\n   argument, the value associated with the keyword is retained in the\n   dictionary.  For example, these all return a dictionary equal to\n   ``{"one": 1, "two": 2}``:\n\n   * ``dict(one=1, two=2)``\n\n   * ``dict({\'one\': 1, \'two\': 2})``\n\n   * ``dict(zip((\'one\', \'two\'), (1, 2)))``\n\n   * ``dict([[\'two\', 2], [\'one\', 1]])``\n\n   The first example only works for keys that are valid Python\n   identifiers; the others work with any valid keys.\n\n   These are the operations that dictionaries support (and therefore,\n   custom mapping types should support too):\n\n   len(d)\n\n      Return the number of items in the dictionary *d*.\n\n   d[key]\n\n      Return the item of *d* with key *key*.  Raises a ``KeyError`` if\n      *key* is not in the map.\n\n      If a subclass of dict defines a method ``__missing__()``, if the\n      key *key* is not present, the ``d[key]`` operation calls that\n      method with the key *key* as argument.  The ``d[key]`` operation\n      then returns or raises whatever is returned or raised by the\n      ``__missing__(key)`` call if the key is not present. No other\n      operations or methods invoke ``__missing__()``. If\n      ``__missing__()`` is not defined, ``KeyError`` is raised.\n      ``__missing__()`` must be a method; it cannot be an instance\n      variable:\n\n         >>> class Counter(dict):\n         ...     def __missing__(self, key):\n         ...         return 0\n         >>> c = Counter()\n         >>> c[\'red\']\n         0\n         >>> c[\'red\'] += 1\n         >>> c[\'red\']\n         1\n\n      See ``collections.Counter`` for a complete implementation\n      including other methods helpful for accumulating and managing\n      tallies.\n\n   d[key] = value\n\n      Set ``d[key]`` to *value*.\n\n   del d[key]\n\n      Remove ``d[key]`` from *d*.  Raises a ``KeyError`` if *key* is\n      not in the map.\n\n   key in d\n\n      Return ``True`` if *d* has a key *key*, else ``False``.\n\n   key not in d\n\n      Equivalent to ``not key in d``.\n\n   iter(d)\n\n      Return an iterator over the keys of the dictionary.  This is a\n      shortcut for ``iter(d.keys())``.\n\n   clear()\n\n      Remove all items from the dictionary.\n\n   copy()\n\n      Return a shallow copy of the dictionary.\n\n   classmethod fromkeys(seq[, value])\n\n      Create a new dictionary with keys from *seq* and values set to\n      *value*.\n\n      ``fromkeys()`` is a class method that returns a new dictionary.\n      *value* defaults to ``None``.\n\n   get(key[, default])\n\n      Return the value for *key* if *key* is in the dictionary, else\n      *default*. If *default* is not given, it defaults to ``None``,\n      so that this method never raises a ``KeyError``.\n\n   items()\n\n      Return a new view of the dictionary\'s items (``(key, value)``\n      pairs).  See below for documentation of view objects.\n\n   keys()\n\n      Return a new view of the dictionary\'s keys.  See below for\n      documentation of view objects.\n\n   pop(key[, default])\n\n      If *key* is in the dictionary, remove it and return its value,\n      else return *default*.  If *default* is not given and *key* is\n      not in the dictionary, a ``KeyError`` is raised.\n\n   popitem()\n\n      Remove and return an arbitrary ``(key, value)`` pair from the\n      dictionary.\n\n      ``popitem()`` is useful to destructively iterate over a\n      dictionary, as often used in set algorithms.  If the dictionary\n      is empty, calling ``popitem()`` raises a ``KeyError``.\n\n   setdefault(key[, default])\n\n      If *key* is in the dictionary, return its value.  If not, insert\n      *key* with a value of *default* and return *default*.  *default*\n      defaults to ``None``.\n\n   update([other])\n\n      Update the dictionary with the key/value pairs from *other*,\n      overwriting existing keys.  Return ``None``.\n\n      ``update()`` accepts either another dictionary object or an\n      iterable of key/value pairs (as tuples or other iterables of\n      length two).  If keyword arguments are specified, the dictionary\n      is then updated with those key/value pairs: ``d.update(red=1,\n      blue=2)``.\n\n   values()\n\n      Return a new view of the dictionary\'s values.  See below for\n      documentation of view objects.\n\n\nDictionary view objects\n=======================\n\nThe objects returned by ``dict.keys()``, ``dict.values()`` and\n``dict.items()`` are *view objects*.  They provide a dynamic view on\nthe dictionary\'s entries, which means that when the dictionary\nchanges, the view reflects these changes.\n\nDictionary views can be iterated over to yield their respective data,\nand support membership tests:\n\nlen(dictview)\n\n   Return the number of entries in the dictionary.\n\niter(dictview)\n\n   Return an iterator over the keys, values or items (represented as\n   tuples of ``(key, value)``) in the dictionary.\n\n   Keys and values are iterated over in an arbitrary order which is\n   non-random, varies across Python implementations, and depends on\n   the dictionary\'s history of insertions and deletions. If keys,\n   values and items views are iterated over with no intervening\n   modifications to the dictionary, the order of items will directly\n   correspond.  This allows the creation of ``(value, key)`` pairs\n   using ``zip()``: ``pairs = zip(d.values(), d.keys())``.  Another\n   way to create the same list is ``pairs = [(v, k) for (k, v) in\n   d.items()]``.\n\n   Iterating views while adding or deleting entries in the dictionary\n   may raise a ``RuntimeError`` or fail to iterate over all entries.\n\nx in dictview\n\n   Return ``True`` if *x* is in the underlying dictionary\'s keys,\n   values or items (in the latter case, *x* should be a ``(key,\n   value)`` tuple).\n\nKeys views are set-like since their entries are unique and hashable.\nIf all values are hashable, so that ``(key, value)`` pairs are unique\nand hashable, then the items view is also set-like.  (Values views are\nnot treated as set-like since the entries are generally not unique.)\nFor set-like views, all of the operations defined for the abstract\nbase class ``collections.Set`` are available (for example, ``==``,\n``<``, or ``^``).\n\nAn example of dictionary view usage:\n\n   >>> dishes = {\'eggs\': 2, \'sausage\': 1, \'bacon\': 1, \'spam\': 500}\n   >>> keys = dishes.keys()\n   >>> values = dishes.values()\n\n   >>> # iteration\n   >>> n = 0\n   >>> for val in values:\n   ...     n += val\n   >>> print(n)\n   504\n\n   >>> # keys and values are iterated over in the same order\n   >>> list(keys)\n   [\'eggs\', \'bacon\', \'sausage\', \'spam\']\n   >>> list(values)\n   [2, 1, 1, 500]\n\n   >>> # view objects are dynamic and reflect dict changes\n   >>> del dishes[\'eggs\']\n   >>> del dishes[\'sausage\']\n   >>> list(keys)\n   [\'spam\', \'bacon\']\n\n   >>> # set operations\n   >>> keys & {\'eggs\', \'bacon\', \'salad\'}\n   {\'bacon\'}\n   >>> keys ^ {\'sausage\', \'juice\'}\n   {\'juice\', \'eggs\', \'bacon\', \'spam\'}\n',
+ 'typesmapping': '\nMapping Types --- ``dict``\n**************************\n\nA *mapping* object maps *hashable* values to arbitrary objects.\nMappings are mutable objects.  There is currently only one standard\nmapping type, the *dictionary*.  (For other containers see the built\nin ``list``, ``set``, and ``tuple`` classes, and the ``collections``\nmodule.)\n\nA dictionary\'s keys are *almost* arbitrary values.  Values that are\nnot *hashable*, that is, values containing lists, dictionaries or\nother mutable types (that are compared by value rather than by object\nidentity) may not be used as keys.  Numeric types used for keys obey\nthe normal rules for numeric comparison: if two numbers compare equal\n(such as ``1`` and ``1.0``) then they can be used interchangeably to\nindex the same dictionary entry.  (Note however, that since computers\nstore floating-point numbers as approximations it is usually unwise to\nuse them as dictionary keys.)\n\nDictionaries can be created by placing a comma-separated list of\n``key: value`` pairs within braces, for example: ``{\'jack\': 4098,\n\'sjoerd\': 4127}`` or ``{4098: \'jack\', 4127: \'sjoerd\'}``, or by the\n``dict`` constructor.\n\nclass class dict([arg])\n\n   Return a new dictionary initialized from an optional positional\n   argument or from a set of keyword arguments.  If no arguments are\n   given, return a new empty dictionary.  If the positional argument\n   *arg* is a mapping object, return a dictionary mapping the same\n   keys to the same values as does the mapping object.  Otherwise the\n   positional argument must be a sequence, a container that supports\n   iteration, or an iterator object.  The elements of the argument\n   must each also be of one of those kinds, and each must in turn\n   contain exactly two objects.  The first is used as a key in the new\n   dictionary, and the second as the key\'s value.  If a given key is\n   seen more than once, the last value associated with it is retained\n   in the new dictionary.\n\n   If keyword arguments are given, the keywords themselves with their\n   associated values are added as items to the dictionary.  If a key\n   is specified both in the positional argument and as a keyword\n   argument, the value associated with the keyword is retained in the\n   dictionary.  For example, these all return a dictionary equal to\n   ``{"one": 1, "two": 2}``:\n\n   * ``dict(one=1, two=2)``\n\n   * ``dict({\'one\': 1, \'two\': 2})``\n\n   * ``dict(zip((\'one\', \'two\'), (1, 2)))``\n\n   * ``dict([[\'two\', 2], [\'one\', 1]])``\n\n   The first example only works for keys that are valid Python\n   identifiers; the others work with any valid keys.\n\n   These are the operations that dictionaries support (and therefore,\n   custom mapping types should support too):\n\n   len(d)\n\n      Return the number of items in the dictionary *d*.\n\n   d[key]\n\n      Return the item of *d* with key *key*.  Raises a ``KeyError`` if\n      *key* is not in the map.\n\n      If a subclass of dict defines a method ``__missing__()``, if the\n      key *key* is not present, the ``d[key]`` operation calls that\n      method with the key *key* as argument.  The ``d[key]`` operation\n      then returns or raises whatever is returned or raised by the\n      ``__missing__(key)`` call if the key is not present. No other\n      operations or methods invoke ``__missing__()``. If\n      ``__missing__()`` is not defined, ``KeyError`` is raised.\n      ``__missing__()`` must be a method; it cannot be an instance\n      variable:\n\n         >>> class Counter(dict):\n         ...     def __missing__(self, key):\n         ...         return 0\n         >>> c = Counter()\n         >>> c[\'red\']\n         0\n         >>> c[\'red\'] += 1\n         >>> c[\'red\']\n         1\n\n      See ``collections.Counter`` for a complete implementation\n      including other methods helpful for accumulating and managing\n      tallies.\n\n   d[key] = value\n\n      Set ``d[key]`` to *value*.\n\n   del d[key]\n\n      Remove ``d[key]`` from *d*.  Raises a ``KeyError`` if *key* is\n      not in the map.\n\n   key in d\n\n      Return ``True`` if *d* has a key *key*, else ``False``.\n\n   key not in d\n\n      Equivalent to ``not key in d``.\n\n   iter(d)\n\n      Return an iterator over the keys of the dictionary.  This is a\n      shortcut for ``iter(d.keys())``.\n\n   clear()\n\n      Remove all items from the dictionary.\n\n   copy()\n\n      Return a shallow copy of the dictionary.\n\n   classmethod fromkeys(seq[, value])\n\n      Create a new dictionary with keys from *seq* and values set to\n      *value*.\n\n      ``fromkeys()`` is a class method that returns a new dictionary.\n      *value* defaults to ``None``.\n\n   get(key[, default])\n\n      Return the value for *key* if *key* is in the dictionary, else\n      *default*. If *default* is not given, it defaults to ``None``,\n      so that this method never raises a ``KeyError``.\n\n   items()\n\n      Return a new view of the dictionary\'s items (``(key, value)``\n      pairs).  See below for documentation of view objects.\n\n   keys()\n\n      Return a new view of the dictionary\'s keys.  See below for\n      documentation of view objects.\n\n   pop(key[, default])\n\n      If *key* is in the dictionary, remove it and return its value,\n      else return *default*.  If *default* is not given and *key* is\n      not in the dictionary, a ``KeyError`` is raised.\n\n   popitem()\n\n      Remove and return an arbitrary ``(key, value)`` pair from the\n      dictionary.\n\n      ``popitem()`` is useful to destructively iterate over a\n      dictionary, as often used in set algorithms.  If the dictionary\n      is empty, calling ``popitem()`` raises a ``KeyError``.\n\n   setdefault(key[, default])\n\n      If *key* is in the dictionary, return its value.  If not, insert\n      *key* with a value of *default* and return *default*.  *default*\n      defaults to ``None``.\n\n   update([other])\n\n      Update the dictionary with the key/value pairs from *other*,\n      overwriting existing keys.  Return ``None``.\n\n      ``update()`` accepts either another dictionary object or an\n      iterable of key/value pairs (as tuples or other iterables of\n      length two).  If keyword arguments are specified, the dictionary\n      is then updated with those key/value pairs: ``d.update(red=1,\n      blue=2)``.\n\n   values()\n\n      Return a new view of the dictionary\'s values.  See below for\n      documentation of view objects.\n\n\nDictionary view objects\n=======================\n\nThe objects returned by ``dict.keys()``, ``dict.values()`` and\n``dict.items()`` are *view objects*.  They provide a dynamic view on\nthe dictionary\'s entries, which means that when the dictionary\nchanges, the view reflects these changes.\n\nDictionary views can be iterated over to yield their respective data,\nand support membership tests:\n\nlen(dictview)\n\n   Return the number of entries in the dictionary.\n\niter(dictview)\n\n   Return an iterator over the keys, values or items (represented as\n   tuples of ``(key, value)``) in the dictionary.\n\n   Keys and values are iterated over in an arbitrary order which is\n   non-random, varies across Python implementations, and depends on\n   the dictionary\'s history of insertions and deletions. If keys,\n   values and items views are iterated over with no intervening\n   modifications to the dictionary, the order of items will directly\n   correspond.  This allows the creation of ``(value, key)`` pairs\n   using ``zip()``: ``pairs = zip(d.values(), d.keys())``.  Another\n   way to create the same list is ``pairs = [(v, k) for (k, v) in\n   d.items()]``.\n\n   Iterating views while adding or deleting entries in the dictionary\n   may raise a ``RuntimeError`` or fail to iterate over all entries.\n\nx in dictview\n\n   Return ``True`` if *x* is in the underlying dictionary\'s keys,\n   values or items (in the latter case, *x* should be a ``(key,\n   value)`` tuple).\n\nKeys views are set-like since their entries are unique and hashable.\nIf all values are hashable, so that ``(key, value)`` pairs are unique\nand hashable, then the items view is also set-like.  (Values views are\nnot treated as set-like since the entries are generally not unique.)\nFor set-like views, all of the operations defined for the abstract\nbase class ``collections.Set`` are available (for example, ``==``,\n``<``, or ``^``).\n\nAn example of dictionary view usage:\n\n   >>> dishes = {\'eggs\': 2, \'sausage\': 1, \'bacon\': 1, \'spam\': 500}\n   >>> keys = dishes.keys()\n   >>> values = dishes.values()\n\n   >>> # iteration\n   >>> n = 0\n   >>> for val in values:\n   ...     n += val\n   >>> print(n)\n   504\n\n   >>> # keys and values are iterated over in the same order\n   >>> list(keys)\n   [\'eggs\', \'bacon\', \'sausage\', \'spam\']\n   >>> list(values)\n   [2, 1, 1, 500]\n\n   >>> # view objects are dynamic and reflect dict changes\n   >>> del dishes[\'eggs\']\n   >>> del dishes[\'sausage\']\n   >>> list(keys)\n   [\'spam\', \'bacon\']\n\n   >>> # set operations\n   >>> keys & {\'eggs\', \'bacon\', \'salad\'}\n   {\'bacon\'}\n   >>> keys ^ {\'sausage\', \'juice\'}\n   {\'juice\', \'sausage\', \'bacon\', \'spam\'}\n',
  'typesmethods': "\nMethods\n*******\n\nMethods are functions that are called using the attribute notation.\nThere are two flavors: built-in methods (such as ``append()`` on\nlists) and class instance methods.  Built-in methods are described\nwith the types that support them.\n\nIf you access a method (a function defined in a class namespace)\nthrough an instance, you get a special object: a *bound method* (also\ncalled *instance method*) object. When called, it will add the\n``self`` argument to the argument list.  Bound methods have two\nspecial read-only attributes: ``m.__self__`` is the object on which\nthe method operates, and ``m.__func__`` is the function implementing\nthe method.  Calling ``m(arg-1, arg-2, ..., arg-n)`` is completely\nequivalent to calling ``m.__func__(m.__self__, arg-1, arg-2, ...,\narg-n)``.\n\nLike function objects, bound method objects support getting arbitrary\nattributes.  However, since method attributes are actually stored on\nthe underlying function object (``meth.__func__``), setting method\nattributes on bound methods is disallowed.  Attempting to set a method\nattribute results in a ``TypeError`` being raised.  In order to set a\nmethod attribute, you need to explicitly set it on the underlying\nfunction object:\n\n   class C:\n       def method(self):\n           pass\n\n   c = C()\n   c.method.__func__.whoami = 'my name is c'\n\nSee *The standard type hierarchy* for more information.\n",
- 'typesmodules': "\nModules\n*******\n\nThe only special operation on a module is attribute access:\n``m.name``, where *m* is a module and *name* accesses a name defined\nin *m*'s symbol table. Module attributes can be assigned to.  (Note\nthat the ``import`` statement is not, strictly speaking, an operation\non a module object; ``import foo`` does not require a module object\nnamed *foo* to exist, rather it requires an (external) *definition*\nfor a module named *foo* somewhere.)\n\nA special member of every module is ``__dict__``. This is the\ndictionary containing the module's symbol table. Modifying this\ndictionary will actually change the module's symbol table, but direct\nassignment to the ``__dict__`` attribute is not possible (you can\nwrite ``m.__dict__['a'] = 1``, which defines ``m.a`` to be ``1``, but\nyou can't write ``m.__dict__ = {}``).  Modifying ``__dict__`` directly\nis not recommended.\n\nModules built into the interpreter are written like this: ``<module\n'sys' (built-in)>``.  If loaded from a file, they are written as\n``<module 'os' from '/usr/local/lib/pythonX.Y/os.pyc'>``.\n",
- 'typesseq': '\nSequence Types --- ``str``, ``bytes``, ``bytearray``, ``list``, ``tuple``, ``range``\n************************************************************************************\n\nThere are six sequence types: strings, byte sequences (``bytes``\nobjects), byte arrays (``bytearray`` objects), lists, tuples, and\nrange objects.  For other containers see the built in ``dict`` and\n``set`` classes, and the ``collections`` module.\n\nStrings contain Unicode characters.  Their literals are written in\nsingle or double quotes: ``\'xyzzy\'``, ``"frobozz"``.  See *String and\nBytes literals* for more about string literals.  In addition to the\nfunctionality described here, there are also string-specific methods\ndescribed in the *String Methods* section.\n\nBytes and bytearray objects contain single bytes -- the former is\nimmutable while the latter is a mutable sequence.  Bytes objects can\nbe constructed the constructor, ``bytes()``, and from literals; use a\n``b`` prefix with normal string syntax: ``b\'xyzzy\'``.  To construct\nbyte arrays, use the ``bytearray()`` function.\n\nWhile string objects are sequences of characters (represented by\nstrings of length 1), bytes and bytearray objects are sequences of\n*integers* (between 0 and 255), representing the ASCII value of single\nbytes.  That means that for a bytes or bytearray object *b*, ``b[0]``\nwill be an integer, while ``b[0:1]`` will be a bytes or bytearray\nobject of length 1.  The representation of bytes objects uses the\nliteral format (``b\'...\'``) since it is generally more useful than\ne.g. ``bytes([50, 19, 100])``.  You can always convert a bytes object\ninto a list of integers using ``list(b)``.\n\nAlso, while in previous Python versions, byte strings and Unicode\nstrings could be exchanged for each other rather freely (barring\nencoding issues), strings and bytes are now completely separate\nconcepts.  There\'s no implicit en-/decoding if you pass an object of\nthe wrong type.  A string always compares unequal to a bytes or\nbytearray object.\n\nLists are constructed with square brackets, separating items with\ncommas: ``[a, b, c]``.  Tuples are constructed by the comma operator\n(not within square brackets), with or without enclosing parentheses,\nbut an empty tuple must have the enclosing parentheses, such as ``a,\nb, c`` or ``()``.  A single item tuple must have a trailing comma,\nsuch as ``(d,)``.\n\nObjects of type range are created using the ``range()`` function.\nThey don\'t support concatenation or repetition, and using ``min()`` or\n``max()`` on them is inefficient.\n\nMost sequence types support the following operations.  The ``in`` and\n``not in`` operations have the same priorities as the comparison\noperations.  The ``+`` and ``*`` operations have the same priority as\nthe corresponding numeric operations. [3] Additional methods are\nprovided for *Mutable Sequence Types*.\n\nThis table lists the sequence operations sorted in ascending priority\n(operations in the same box have the same priority).  In the table,\n*s* and *t* are sequences of the same type; *n*, *i*, *j* and *k* are\nintegers.\n\n+--------------------+----------------------------------+------------+\n| Operation          | Result                           | Notes      |\n+====================+==================================+============+\n| ``x in s``         | ``True`` if an item of *s* is    | (1)        |\n|                    | equal to *x*, else ``False``     |            |\n+--------------------+----------------------------------+------------+\n| ``x not in s``     | ``False`` if an item of *s* is   | (1)        |\n|                    | equal to *x*, else ``True``      |            |\n+--------------------+----------------------------------+------------+\n| ``s + t``          | the concatenation of *s* and *t* | (6)        |\n+--------------------+----------------------------------+------------+\n| ``s * n, n * s``   | *n* shallow copies of *s*        | (2)        |\n|                    | concatenated                     |            |\n+--------------------+----------------------------------+------------+\n| ``s[i]``           | *i*\'th item of *s*, origin 0     | (3)        |\n+--------------------+----------------------------------+------------+\n| ``s[i:j]``         | slice of *s* from *i* to *j*     | (3)(4)     |\n+--------------------+----------------------------------+------------+\n| ``s[i:j:k]``       | slice of *s* from *i* to *j*     | (3)(5)     |\n|                    | with step *k*                    |            |\n+--------------------+----------------------------------+------------+\n| ``len(s)``         | length of *s*                    |            |\n+--------------------+----------------------------------+------------+\n| ``min(s)``         | smallest item of *s*             |            |\n+--------------------+----------------------------------+------------+\n| ``max(s)``         | largest item of *s*              |            |\n+--------------------+----------------------------------+------------+\n| ``s.index(i)``     | index of the first occurence of  |            |\n|                    | *i* in *s*                       |            |\n+--------------------+----------------------------------+------------+\n| ``s.count(i)``     | total number of occurences of    |            |\n|                    | *i* in *s*                       |            |\n+--------------------+----------------------------------+------------+\n\nSequence types also support comparisons.  In particular, tuples and\nlists are compared lexicographically by comparing corresponding\nelements.  This means that to compare equal, every element must\ncompare equal and the two sequences must be of the same type and have\nthe same length.  (For full details see *Comparisons* in the language\nreference.)\n\nNotes:\n\n1. When *s* is a string object, the ``in`` and ``not in`` operations\n   act like a substring test.\n\n2. Values of *n* less than ``0`` are treated as ``0`` (which yields an\n   empty sequence of the same type as *s*).  Note also that the copies\n   are shallow; nested structures are not copied.  This often haunts\n   new Python programmers; consider:\n\n   >>> lists = [[]] * 3\n   >>> lists\n   [[], [], []]\n   >>> lists[0].append(3)\n   >>> lists\n   [[3], [3], [3]]\n\n   What has happened is that ``[[]]`` is a one-element list containing\n   an empty list, so all three elements of ``[[]] * 3`` are (pointers\n   to) this single empty list.  Modifying any of the elements of\n   ``lists`` modifies this single list. You can create a list of\n   different lists this way:\n\n   >>> lists = [[] for i in range(3)]\n   >>> lists[0].append(3)\n   >>> lists[1].append(5)\n   >>> lists[2].append(7)\n   >>> lists\n   [[3], [5], [7]]\n\n3. If *i* or *j* is negative, the index is relative to the end of the\n   string: ``len(s) + i`` or ``len(s) + j`` is substituted.  But note\n   that ``-0`` is still ``0``.\n\n4. The slice of *s* from *i* to *j* is defined as the sequence of\n   items with index *k* such that ``i <= k < j``.  If *i* or *j* is\n   greater than ``len(s)``, use ``len(s)``.  If *i* is omitted or\n   ``None``, use ``0``.  If *j* is omitted or ``None``, use\n   ``len(s)``.  If *i* is greater than or equal to *j*, the slice is\n   empty.\n\n5. The slice of *s* from *i* to *j* with step *k* is defined as the\n   sequence of items with index  ``x = i + n*k`` such that ``0 <= n <\n   (j-i)/k``.  In other words, the indices are ``i``, ``i+k``,\n   ``i+2*k``, ``i+3*k`` and so on, stopping when *j* is reached (but\n   never including *j*).  If *i* or *j* is greater than ``len(s)``,\n   use ``len(s)``.  If *i* or *j* are omitted or ``None``, they become\n   "end" values (which end depends on the sign of *k*).  Note, *k*\n   cannot be zero. If *k* is ``None``, it is treated like ``1``.\n\n6. **CPython implementation detail:** If *s* and *t* are both strings,\n   some Python implementations such as CPython can usually perform an\n   in-place optimization for assignments of the form ``s = s + t`` or\n   ``s += t``.  When applicable, this optimization makes quadratic\n   run-time much less likely.  This optimization is both version and\n   implementation dependent.  For performance sensitive code, it is\n   preferable to use the ``str.join()`` method which assures\n   consistent linear concatenation performance across versions and\n   implementations.\n\n\nString Methods\n==============\n\nString objects support the methods listed below.\n\nIn addition, Python\'s strings support the sequence type methods\ndescribed in the *Sequence Types --- str, bytes, bytearray, list,\ntuple, range* section. To output formatted strings, see the *String\nFormatting* section. Also, see the ``re`` module for string functions\nbased on regular expressions.\n\nstr.capitalize()\n\n   Return a copy of the string with its first character capitalized\n   and the rest lowercased.\n\nstr.center(width[, fillchar])\n\n   Return centered in a string of length *width*. Padding is done\n   using the specified *fillchar* (default is a space).\n\nstr.count(sub[, start[, end]])\n\n   Return the number of non-overlapping occurrences of substring *sub*\n   in the range [*start*, *end*].  Optional arguments *start* and\n   *end* are interpreted as in slice notation.\n\nstr.encode(encoding="utf-8", errors="strict")\n\n   Return an encoded version of the string as a bytes object. Default\n   encoding is ``\'utf-8\'``. *errors* may be given to set a different\n   error handling scheme. The default for *errors* is ``\'strict\'``,\n   meaning that encoding errors raise a ``UnicodeError``. Other\n   possible values are ``\'ignore\'``, ``\'replace\'``,\n   ``\'xmlcharrefreplace\'``, ``\'backslashreplace\'`` and any other name\n   registered via ``codecs.register_error()``, see section *Codec Base\n   Classes*. For a list of possible encodings, see section *Standard\n   Encodings*.\n\n   Changed in version 3.1: Support for keyword arguments added.\n\nstr.endswith(suffix[, start[, end]])\n\n   Return ``True`` if the string ends with the specified *suffix*,\n   otherwise return ``False``.  *suffix* can also be a tuple of\n   suffixes to look for.  With optional *start*, test beginning at\n   that position.  With optional *end*, stop comparing at that\n   position.\n\nstr.expandtabs([tabsize])\n\n   Return a copy of the string where all tab characters are replaced\n   by one or more spaces, depending on the current column and the\n   given tab size.  The column number is reset to zero after each\n   newline occurring in the string. If *tabsize* is not given, a tab\n   size of ``8`` characters is assumed.  This doesn\'t understand other\n   non-printing characters or escape sequences.\n\nstr.find(sub[, start[, end]])\n\n   Return the lowest index in the string where substring *sub* is\n   found, such that *sub* is contained in the slice ``s[start:end]``.\n   Optional arguments *start* and *end* are interpreted as in slice\n   notation.  Return ``-1`` if *sub* is not found.\n\n   Note: The ``find()`` method should be used only if you need to know the\n     position of *sub*.  To check if *sub* is a substring or not, use\n     the ``in`` operator:\n\n        >>> \'Py\' in \'Python\'\n        True\n\nstr.format(*args, **kwargs)\n\n   Perform a string formatting operation.  The string on which this\n   method is called can contain literal text or replacement fields\n   delimited by braces ``{}``.  Each replacement field contains either\n   the numeric index of a positional argument, or the name of a\n   keyword argument.  Returns a copy of the string where each\n   replacement field is replaced with the string value of the\n   corresponding argument.\n\n   >>> "The sum of 1 + 2 is {0}".format(1+2)\n   \'The sum of 1 + 2 is 3\'\n\n   See *Format String Syntax* for a description of the various\n   formatting options that can be specified in format strings.\n\nstr.format_map(mapping)\n\n   Similar to ``str.format(**mapping)``, except that ``mapping`` is\n   used directly and not copied to a ``dict`` .  This is useful if for\n   example ``mapping`` is a dict subclass:\n\n   >>> class Default(dict):\n   ...     def __missing__(self, key):\n   ...         return key\n   ...\n   >>> \'{name} was born in {country}\'.format_map(Default(name=\'Guido\'))\n   \'Guido was born in country\'\n\n   New in version 3.2.\n\nstr.index(sub[, start[, end]])\n\n   Like ``find()``, but raise ``ValueError`` when the substring is not\n   found.\n\nstr.isalnum()\n\n   Return true if all characters in the string are alphanumeric and\n   there is at least one character, false otherwise.  A character\n   ``c`` is alphanumeric if one of the following returns ``True``:\n   ``c.isalpha()``, ``c.isdecimal()``, ``c.isdigit()``, or\n   ``c.isnumeric()``.\n\nstr.isalpha()\n\n   Return true if all characters in the string are alphabetic and\n   there is at least one character, false otherwise.  Alphabetic\n   characters are those characters defined in the Unicode character\n   database as "Letter", i.e., those with general category property\n   being one of "Lm", "Lt", "Lu", "Ll", or "Lo".  Note that this is\n   different from the "Alphabetic" property defined in the Unicode\n   Standard.\n\nstr.isdecimal()\n\n   Return true if all characters in the string are decimal characters\n   and there is at least one character, false otherwise. Decimal\n   characters are those from general category "Nd". This category\n   includes digit characters, and all characters that that can be used\n   to form decimal-radix numbers, e.g. U+0660, ARABIC-INDIC DIGIT\n   ZERO.\n\nstr.isdigit()\n\n   Return true if all characters in the string are digits and there is\n   at least one character, false otherwise.  Digits include decimal\n   characters and digits that need special handling, such as the\n   compatibility superscript digits.  Formally, a digit is a character\n   that has the property value Numeric_Type=Digit or\n   Numeric_Type=Decimal.\n\nstr.isidentifier()\n\n   Return true if the string is a valid identifier according to the\n   language definition, section *Identifiers and keywords*.\n\nstr.islower()\n\n   Return true if all cased characters in the string are lowercase and\n   there is at least one cased character, false otherwise.  Cased\n   characters are those with general category property being one of\n   "Lu", "Ll", or "Lt" and lowercase characters are those with general\n   category property "Ll".\n\nstr.isnumeric()\n\n   Return true if all characters in the string are numeric characters,\n   and there is at least one character, false otherwise. Numeric\n   characters include digit characters, and all characters that have\n   the Unicode numeric value property, e.g. U+2155, VULGAR FRACTION\n   ONE FIFTH.  Formally, numeric characters are those with the\n   property value Numeric_Type=Digit, Numeric_Type=Decimal or\n   Numeric_Type=Numeric.\n\nstr.isprintable()\n\n   Return true if all characters in the string are printable or the\n   string is empty, false otherwise.  Nonprintable characters are\n   those characters defined in the Unicode character database as\n   "Other" or "Separator", excepting the ASCII space (0x20) which is\n   considered printable.  (Note that printable characters in this\n   context are those which should not be escaped when ``repr()`` is\n   invoked on a string.  It has no bearing on the handling of strings\n   written to ``sys.stdout`` or ``sys.stderr``.)\n\nstr.isspace()\n\n   Return true if there are only whitespace characters in the string\n   and there is at least one character, false otherwise.  Whitespace\n   characters  are those characters defined in the Unicode character\n   database as "Other" or "Separator" and those with bidirectional\n   property being one of "WS", "B", or "S".\n\nstr.istitle()\n\n   Return true if the string is a titlecased string and there is at\n   least one character, for example uppercase characters may only\n   follow uncased characters and lowercase characters only cased ones.\n   Return false otherwise.\n\nstr.isupper()\n\n   Return true if all cased characters in the string are uppercase and\n   there is at least one cased character, false otherwise. Cased\n   characters are those with general category property being one of\n   "Lu", "Ll", or "Lt" and uppercase characters are those with general\n   category property "Lu".\n\nstr.join(iterable)\n\n   Return a string which is the concatenation of the strings in the\n   *iterable* *iterable*.  A ``TypeError`` will be raised if there are\n   any non-string values in *seq*, including ``bytes`` objects.  The\n   separator between elements is the string providing this method.\n\nstr.ljust(width[, fillchar])\n\n   Return the string left justified in a string of length *width*.\n   Padding is done using the specified *fillchar* (default is a\n   space).  The original string is returned if *width* is less than\n   ``len(s)``.\n\nstr.lower()\n\n   Return a copy of the string converted to lowercase.\n\nstr.lstrip([chars])\n\n   Return a copy of the string with leading characters removed.  The\n   *chars* argument is a string specifying the set of characters to be\n   removed.  If omitted or ``None``, the *chars* argument defaults to\n   removing whitespace.  The *chars* argument is not a prefix; rather,\n   all combinations of its values are stripped:\n\n   >>> \'   spacious   \'.lstrip()\n   \'spacious   \'\n   >>> \'www.example.com\'.lstrip(\'cmowz.\')\n   \'example.com\'\n\nstatic str.maketrans(x[, y[, z]])\n\n   This static method returns a translation table usable for\n   ``str.translate()``.\n\n   If there is only one argument, it must be a dictionary mapping\n   Unicode ordinals (integers) or characters (strings of length 1) to\n   Unicode ordinals, strings (of arbitrary lengths) or None.\n   Character keys will then be converted to ordinals.\n\n   If there are two arguments, they must be strings of equal length,\n   and in the resulting dictionary, each character in x will be mapped\n   to the character at the same position in y.  If there is a third\n   argument, it must be a string, whose characters will be mapped to\n   None in the result.\n\nstr.partition(sep)\n\n   Split the string at the first occurrence of *sep*, and return a\n   3-tuple containing the part before the separator, the separator\n   itself, and the part after the separator.  If the separator is not\n   found, return a 3-tuple containing the string itself, followed by\n   two empty strings.\n\nstr.replace(old, new[, count])\n\n   Return a copy of the string with all occurrences of substring *old*\n   replaced by *new*.  If the optional argument *count* is given, only\n   the first *count* occurrences are replaced.\n\nstr.rfind(sub[, start[, end]])\n\n   Return the highest index in the string where substring *sub* is\n   found, such that *sub* is contained within ``s[start:end]``.\n   Optional arguments *start* and *end* are interpreted as in slice\n   notation.  Return ``-1`` on failure.\n\nstr.rindex(sub[, start[, end]])\n\n   Like ``rfind()`` but raises ``ValueError`` when the substring *sub*\n   is not found.\n\nstr.rjust(width[, fillchar])\n\n   Return the string right justified in a string of length *width*.\n   Padding is done using the specified *fillchar* (default is a\n   space). The original string is returned if *width* is less than\n   ``len(s)``.\n\nstr.rpartition(sep)\n\n   Split the string at the last occurrence of *sep*, and return a\n   3-tuple containing the part before the separator, the separator\n   itself, and the part after the separator.  If the separator is not\n   found, return a 3-tuple containing two empty strings, followed by\n   the string itself.\n\nstr.rsplit([sep[, maxsplit]])\n\n   Return a list of the words in the string, using *sep* as the\n   delimiter string. If *maxsplit* is given, at most *maxsplit* splits\n   are done, the *rightmost* ones.  If *sep* is not specified or\n   ``None``, any whitespace string is a separator.  Except for\n   splitting from the right, ``rsplit()`` behaves like ``split()``\n   which is described in detail below.\n\nstr.rstrip([chars])\n\n   Return a copy of the string with trailing characters removed.  The\n   *chars* argument is a string specifying the set of characters to be\n   removed.  If omitted or ``None``, the *chars* argument defaults to\n   removing whitespace.  The *chars* argument is not a suffix; rather,\n   all combinations of its values are stripped:\n\n   >>> \'   spacious   \'.rstrip()\n   \'   spacious\'\n   >>> \'mississippi\'.rstrip(\'ipz\')\n   \'mississ\'\n\nstr.split([sep[, maxsplit]])\n\n   Return a list of the words in the string, using *sep* as the\n   delimiter string.  If *maxsplit* is given, at most *maxsplit*\n   splits are done (thus, the list will have at most ``maxsplit+1``\n   elements).  If *maxsplit* is not specified, then there is no limit\n   on the number of splits (all possible splits are made).\n\n   If *sep* is given, consecutive delimiters are not grouped together\n   and are deemed to delimit empty strings (for example,\n   ``\'1,,2\'.split(\',\')`` returns ``[\'1\', \'\', \'2\']``).  The *sep*\n   argument may consist of multiple characters (for example,\n   ``\'1<>2<>3\'.split(\'<>\')`` returns ``[\'1\', \'2\', \'3\']``). Splitting\n   an empty string with a specified separator returns ``[\'\']``.\n\n   If *sep* is not specified or is ``None``, a different splitting\n   algorithm is applied: runs of consecutive whitespace are regarded\n   as a single separator, and the result will contain no empty strings\n   at the start or end if the string has leading or trailing\n   whitespace.  Consequently, splitting an empty string or a string\n   consisting of just whitespace with a ``None`` separator returns\n   ``[]``.\n\n   For example, ``\' 1  2   3  \'.split()`` returns ``[\'1\', \'2\', \'3\']``,\n   and ``\'  1  2   3  \'.split(None, 1)`` returns ``[\'1\', \'2   3  \']``.\n\nstr.splitlines([keepends])\n\n   Return a list of the lines in the string, breaking at line\n   boundaries.  Line breaks are not included in the resulting list\n   unless *keepends* is given and true.\n\nstr.startswith(prefix[, start[, end]])\n\n   Return ``True`` if string starts with the *prefix*, otherwise\n   return ``False``. *prefix* can also be a tuple of prefixes to look\n   for.  With optional *start*, test string beginning at that\n   position.  With optional *end*, stop comparing string at that\n   position.\n\nstr.strip([chars])\n\n   Return a copy of the string with the leading and trailing\n   characters removed. The *chars* argument is a string specifying the\n   set of characters to be removed. If omitted or ``None``, the\n   *chars* argument defaults to removing whitespace. The *chars*\n   argument is not a prefix or suffix; rather, all combinations of its\n   values are stripped:\n\n   >>> \'   spacious   \'.strip()\n   \'spacious\'\n   >>> \'www.example.com\'.strip(\'cmowz.\')\n   \'example\'\n\nstr.swapcase()\n\n   Return a copy of the string with uppercase characters converted to\n   lowercase and vice versa.\n\nstr.title()\n\n   Return a titlecased version of the string where words start with an\n   uppercase character and the remaining characters are lowercase.\n\n   The algorithm uses a simple language-independent definition of a\n   word as groups of consecutive letters.  The definition works in\n   many contexts but it means that apostrophes in contractions and\n   possessives form word boundaries, which may not be the desired\n   result:\n\n      >>> "they\'re bill\'s friends from the UK".title()\n      "They\'Re Bill\'S Friends From The Uk"\n\n   A workaround for apostrophes can be constructed using regular\n   expressions:\n\n      >>> import re\n      >>> def titlecase(s):\n              return re.sub(r"[A-Za-z]+(\'[A-Za-z]+)?",\n                            lambda mo: mo.group(0)[0].upper() +\n                                       mo.group(0)[1:].lower(),\n                            s)\n\n      >>> titlecase("they\'re bill\'s friends.")\n      "They\'re Bill\'s Friends."\n\nstr.translate(map)\n\n   Return a copy of the *s* where all characters have been mapped\n   through the *map* which must be a dictionary of Unicode ordinals\n   (integers) to Unicode ordinals, strings or ``None``.  Unmapped\n   characters are left untouched. Characters mapped to ``None`` are\n   deleted.\n\n   You can use ``str.maketrans()`` to create a translation map from\n   character-to-character mappings in different formats.\n\n   Note: An even more flexible approach is to create a custom character\n     mapping codec using the ``codecs`` module (see\n     ``encodings.cp1251`` for an example).\n\nstr.upper()\n\n   Return a copy of the string converted to uppercase.\n\nstr.zfill(width)\n\n   Return the numeric string left filled with zeros in a string of\n   length *width*.  A sign prefix is handled correctly.  The original\n   string is returned if *width* is less than ``len(s)``.\n\n\nOld String Formatting Operations\n================================\n\nNote: The formatting operations described here are obsolete and may go\n  away in future versions of Python.  Use the new *String Formatting*\n  in new code.\n\nString objects have one unique built-in operation: the ``%`` operator\n(modulo). This is also known as the string *formatting* or\n*interpolation* operator. Given ``format % values`` (where *format* is\na string), ``%`` conversion specifications in *format* are replaced\nwith zero or more elements of *values*. The effect is similar to the\nusing ``sprintf()`` in the C language.\n\nIf *format* requires a single argument, *values* may be a single non-\ntuple object. [4]  Otherwise, *values* must be a tuple with exactly\nthe number of items specified by the format string, or a single\nmapping object (for example, a dictionary).\n\nA conversion specifier contains two or more characters and has the\nfollowing components, which must occur in this order:\n\n1. The ``\'%\'`` character, which marks the start of the specifier.\n\n2. Mapping key (optional), consisting of a parenthesised sequence of\n   characters (for example, ``(somename)``).\n\n3. Conversion flags (optional), which affect the result of some\n   conversion types.\n\n4. Minimum field width (optional).  If specified as an ``\'*\'``\n   (asterisk), the actual width is read from the next element of the\n   tuple in *values*, and the object to convert comes after the\n   minimum field width and optional precision.\n\n5. Precision (optional), given as a ``\'.\'`` (dot) followed by the\n   precision.  If specified as ``\'*\'`` (an asterisk), the actual width\n   is read from the next element of the tuple in *values*, and the\n   value to convert comes after the precision.\n\n6. Length modifier (optional).\n\n7. Conversion type.\n\nWhen the right argument is a dictionary (or other mapping type), then\nthe formats in the string *must* include a parenthesised mapping key\ninto that dictionary inserted immediately after the ``\'%\'`` character.\nThe mapping key selects the value to be formatted from the mapping.\nFor example:\n\n>>> print(\'%(language)s has %(number)03d quote types.\' %\n...       {\'language\': "Python", "number": 2})\nPython has 002 quote types.\n\nIn this case no ``*`` specifiers may occur in a format (since they\nrequire a sequential parameter list).\n\nThe conversion flag characters are:\n\n+-----------+-----------------------------------------------------------------------+\n| Flag      | Meaning                                                               |\n+===========+=======================================================================+\n| ``\'#\'``   | The value conversion will use the "alternate form" (where defined     |\n|           | below).                                                               |\n+-----------+-----------------------------------------------------------------------+\n| ``\'0\'``   | The conversion will be zero padded for numeric values.                |\n+-----------+-----------------------------------------------------------------------+\n| ``\'-\'``   | The converted value is left adjusted (overrides the ``\'0\'``           |\n|           | conversion if both are given).                                        |\n+-----------+-----------------------------------------------------------------------+\n| ``\' \'``   | (a space) A blank should be left before a positive number (or empty   |\n|           | string) produced by a signed conversion.                              |\n+-----------+-----------------------------------------------------------------------+\n| ``\'+\'``   | A sign character (``\'+\'`` or ``\'-\'``) will precede the conversion     |\n|           | (overrides a "space" flag).                                           |\n+-----------+-----------------------------------------------------------------------+\n\nA length modifier (``h``, ``l``, or ``L``) may be present, but is\nignored as it is not necessary for Python -- so e.g. ``%ld`` is\nidentical to ``%d``.\n\nThe conversion types are:\n\n+--------------+-------------------------------------------------------+---------+\n| Conversion   | Meaning                                               | Notes   |\n+==============+=======================================================+=========+\n| ``\'d\'``      | Signed integer decimal.                               |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'i\'``      | Signed integer decimal.                               |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'o\'``      | Signed octal value.                                   | (1)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'u\'``      | Obsolete type -- it is identical to ``\'d\'``.          | (7)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'x\'``      | Signed hexadecimal (lowercase).                       | (2)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'X\'``      | Signed hexadecimal (uppercase).                       | (2)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'e\'``      | Floating point exponential format (lowercase).        | (3)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'E\'``      | Floating point exponential format (uppercase).        | (3)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'f\'``      | Floating point decimal format.                        | (3)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'F\'``      | Floating point decimal format.                        | (3)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'g\'``      | Floating point format. Uses lowercase exponential     | (4)     |\n|              | format if exponent is less than -4 or not less than   |         |\n|              | precision, decimal format otherwise.                  |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'G\'``      | Floating point format. Uses uppercase exponential     | (4)     |\n|              | format if exponent is less than -4 or not less than   |         |\n|              | precision, decimal format otherwise.                  |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'c\'``      | Single character (accepts integer or single character |         |\n|              | string).                                              |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'r\'``      | String (converts any Python object using ``repr()``). | (5)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'s\'``      | String (converts any Python object using ``str()``).  |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'%\'``      | No argument is converted, results in a ``\'%\'``        |         |\n|              | character in the result.                              |         |\n+--------------+-------------------------------------------------------+---------+\n\nNotes:\n\n1. The alternate form causes a leading zero (``\'0\'``) to be inserted\n   between left-hand padding and the formatting of the number if the\n   leading character of the result is not already a zero.\n\n2. The alternate form causes a leading ``\'0x\'`` or ``\'0X\'`` (depending\n   on whether the ``\'x\'`` or ``\'X\'`` format was used) to be inserted\n   between left-hand padding and the formatting of the number if the\n   leading character of the result is not already a zero.\n\n3. The alternate form causes the result to always contain a decimal\n   point, even if no digits follow it.\n\n   The precision determines the number of digits after the decimal\n   point and defaults to 6.\n\n4. The alternate form causes the result to always contain a decimal\n   point, and trailing zeroes are not removed as they would otherwise\n   be.\n\n   The precision determines the number of significant digits before\n   and after the decimal point and defaults to 6.\n\n5. The precision determines the maximal number of characters used.\n\n1. See **PEP 237**.\n\nSince Python strings have an explicit length, ``%s`` conversions do\nnot assume that ``\'\\0\'`` is the end of the string.\n\nChanged in version 3.1: ``%f`` conversions for numbers whose absolute\nvalue is over 1e50 are no longer replaced by ``%g`` conversions.\n\nAdditional string operations are defined in standard modules\n``string`` and ``re``.\n\n\nRange Type\n==========\n\nThe ``range`` type is an immutable sequence which is commonly used for\nlooping.  The advantage of the ``range`` type is that an ``range``\nobject will always take the same amount of memory, no matter the size\nof the range it represents.\n\nRange objects have relatively little behavior: they support indexing,\ncontains, iteration, the ``len()`` function, and the following\nmethods:\n\nrange.count(x)\n\n   Return the number of *i*\'s for which ``s[i] == x``.\n\n      New in version 3.2.\n\nrange.index(x)\n\n   Return the smallest *i* such that ``s[i] == x``.  Raises\n   ``ValueError`` when *x* is not in the range.\n\n      New in version 3.2.\n\n\nMutable Sequence Types\n======================\n\nList and bytearray objects support additional operations that allow\nin-place modification of the object.  Other mutable sequence types\n(when added to the language) should also support these operations.\nStrings and tuples are immutable sequence types: such objects cannot\nbe modified once created. The following operations are defined on\nmutable sequence types (where *x* is an arbitrary object).\n\nNote that while lists allow their items to be of any type, bytearray\nobject "items" are all integers in the range 0 <= x < 256.\n\n+--------------------------------+----------------------------------+-----------------------+\n| Operation                      | Result                           | Notes                 |\n+================================+==================================+=======================+\n| ``s[i] = x``                   | item *i* of *s* is replaced by   |                       |\n|                                | *x*                              |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j] = t``                 | slice of *s* from *i* to *j* is  |                       |\n|                                | replaced by the contents of the  |                       |\n|                                | iterable *t*                     |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j]``                 | same as ``s[i:j] = []``          |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j:k] = t``               | the elements of ``s[i:j:k]`` are | (1)                   |\n|                                | replaced by those of *t*         |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j:k]``               | removes the elements of          |                       |\n|                                | ``s[i:j:k]`` from the list       |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.append(x)``                | same as ``s[len(s):len(s)] =     |                       |\n|                                | [x]``                            |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.extend(x)``                | same as ``s[len(s):len(s)] = x`` | (2)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.count(x)``                 | return number of *i*\'s for which |                       |\n|                                | ``s[i] == x``                    |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.index(x[, i[, j]])``       | return smallest *k* such that    | (3)                   |\n|                                | ``s[k] == x`` and ``i <= k < j`` |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.insert(i, x)``             | same as ``s[i:i] = [x]``         | (4)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.pop([i])``                 | same as ``x = s[i]; del s[i];    | (5)                   |\n|                                | return x``                       |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.remove(x)``                | same as ``del s[s.index(x)]``    | (3)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.reverse()``                | reverses the items of *s* in     | (6)                   |\n|                                | place                            |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.sort([key[, reverse]])``   | sort the items of *s* in place   | (6), (7), (8)         |\n+--------------------------------+----------------------------------+-----------------------+\n\nNotes:\n\n1. *t* must have the same length as the slice it is replacing.\n\n2. *x* can be any iterable object.\n\n3. Raises ``ValueError`` when *x* is not found in *s*. When a negative\n   index is passed as the second or third parameter to the ``index()``\n   method, the sequence length is added, as for slice indices.  If it\n   is still negative, it is truncated to zero, as for slice indices.\n\n4. When a negative index is passed as the first parameter to the\n   ``insert()`` method, the sequence length is added, as for slice\n   indices.  If it is still negative, it is truncated to zero, as for\n   slice indices.\n\n5. The optional argument *i* defaults to ``-1``, so that by default\n   the last item is removed and returned.\n\n6. The ``sort()`` and ``reverse()`` methods modify the sequence in\n   place for economy of space when sorting or reversing a large\n   sequence.  To remind you that they operate by side effect, they\n   don\'t return the sorted or reversed sequence.\n\n7. The ``sort()`` method takes optional arguments for controlling the\n   comparisons.  Each must be specified as a keyword argument.\n\n   *key* specifies a function of one argument that is used to extract\n   a comparison key from each list element: ``key=str.lower``.  The\n   default value is ``None``. Use ``functools.cmp_to_key()`` to\n   convert an old-style *cmp* function to a *key* function.\n\n   *reverse* is a boolean value.  If set to ``True``, then the list\n   elements are sorted as if each comparison were reversed.\n\n   The ``sort()`` method is guaranteed to be stable.  A sort is stable\n   if it guarantees not to change the relative order of elements that\n   compare equal --- this is helpful for sorting in multiple passes\n   (for example, sort by department, then by salary grade).\n\n   **CPython implementation detail:** While a list is being sorted,\n   the effect of attempting to mutate, or even inspect, the list is\n   undefined.  The C implementation of Python makes the list appear\n   empty for the duration, and raises ``ValueError`` if it can detect\n   that the list has been mutated during a sort.\n\n8. ``sort()`` is not supported by ``bytearray`` objects.\n\n\nBytes and Byte Array Methods\n============================\n\nBytes and bytearray objects, being "strings of bytes", have all\nmethods found on strings, with the exception of ``encode()``,\n``format()`` and ``isidentifier()``, which do not make sense with\nthese types.  For converting the objects to strings, they have a\n``decode()`` method.\n\nWherever one of these methods needs to interpret the bytes as\ncharacters (e.g. the ``is...()`` methods), the ASCII character set is\nassumed.\n\nNote: The methods on bytes and bytearray objects don\'t accept strings as\n  their arguments, just as the methods on strings don\'t accept bytes\n  as their arguments.  For example, you have to write\n\n     a = "abc"\n     b = a.replace("a", "f")\n\n  and\n\n     a = b"abc"\n     b = a.replace(b"a", b"f")\n\nbytes.decode(encoding="utf-8", errors="strict")\nbytearray.decode(encoding="utf-8", errors="strict")\n\n   Return a string decoded from the given bytes.  Default encoding is\n   ``\'utf-8\'``. *errors* may be given to set a different error\n   handling scheme.  The default for *errors* is ``\'strict\'``, meaning\n   that encoding errors raise a ``UnicodeError``.  Other possible\n   values are ``\'ignore\'``, ``\'replace\'`` and any other name\n   registered via ``codecs.register_error()``, see section *Codec Base\n   Classes*. For a list of possible encodings, see section *Standard\n   Encodings*.\n\n   Changed in version 3.1: Added support for keyword arguments.\n\nThe bytes and bytearray types have an additional class method:\n\nclassmethod bytes.fromhex(string)\nclassmethod bytearray.fromhex(string)\n\n   This ``bytes`` class method returns a bytes or bytearray object,\n   decoding the given string object.  The string must contain two\n   hexadecimal digits per byte, spaces are ignored.\n\n   >>> bytes.fromhex(\'f0 f1f2  \')\n   b\'\\xf0\\xf1\\xf2\'\n\nThe maketrans and translate methods differ in semantics from the\nversions available on strings:\n\nbytes.translate(table[, delete])\nbytearray.translate(table[, delete])\n\n   Return a copy of the bytes or bytearray object where all bytes\n   occurring in the optional argument *delete* are removed, and the\n   remaining bytes have been mapped through the given translation\n   table, which must be a bytes object of length 256.\n\n   You can use the ``bytes.maketrans()`` method to create a\n   translation table.\n\n   Set the *table* argument to ``None`` for translations that only\n   delete characters:\n\n      >>> b\'read this short text\'.translate(None, b\'aeiou\')\n      b\'rd ths shrt txt\'\n\nstatic bytes.maketrans(from, to)\nstatic bytearray.maketrans(from, to)\n\n   This static method returns a translation table usable for\n   ``bytes.translate()`` that will map each character in *from* into\n   the character at the same position in *to*; *from* and *to* must be\n   bytes objects and have the same length.\n\n   New in version 3.1.\n',
+ 'typesmodules': "\nModules\n*******\n\nThe only special operation on a module is attribute access:\n``m.name``, where *m* is a module and *name* accesses a name defined\nin *m*'s symbol table. Module attributes can be assigned to.  (Note\nthat the ``import`` statement is not, strictly speaking, an operation\non a module object; ``import foo`` does not require a module object\nnamed *foo* to exist, rather it requires an (external) *definition*\nfor a module named *foo* somewhere.)\n\nA special attribute of every module is ``__dict__``. This is the\ndictionary containing the module's symbol table. Modifying this\ndictionary will actually change the module's symbol table, but direct\nassignment to the ``__dict__`` attribute is not possible (you can\nwrite ``m.__dict__['a'] = 1``, which defines ``m.a`` to be ``1``, but\nyou can't write ``m.__dict__ = {}``).  Modifying ``__dict__`` directly\nis not recommended.\n\nModules built into the interpreter are written like this: ``<module\n'sys' (built-in)>``.  If loaded from a file, they are written as\n``<module 'os' from '/usr/local/lib/pythonX.Y/os.pyc'>``.\n",
+ 'typesseq': '\nSequence Types --- ``str``, ``bytes``, ``bytearray``, ``list``, ``tuple``, ``range``\n************************************************************************************\n\nThere are six sequence types: strings, byte sequences (``bytes``\nobjects), byte arrays (``bytearray`` objects), lists, tuples, and\nrange objects.  For other containers see the built in ``dict`` and\n``set`` classes, and the ``collections`` module.\n\nStrings contain Unicode characters.  Their literals are written in\nsingle or double quotes: ``\'xyzzy\'``, ``"frobozz"``.  See *String and\nBytes literals* for more about string literals.  In addition to the\nfunctionality described here, there are also string-specific methods\ndescribed in the *String Methods* section.\n\nBytes and bytearray objects contain single bytes -- the former is\nimmutable while the latter is a mutable sequence.  Bytes objects can\nbe constructed the constructor, ``bytes()``, and from literals; use a\n``b`` prefix with normal string syntax: ``b\'xyzzy\'``.  To construct\nbyte arrays, use the ``bytearray()`` function.\n\nWhile string objects are sequences of characters (represented by\nstrings of length 1), bytes and bytearray objects are sequences of\n*integers* (between 0 and 255), representing the ASCII value of single\nbytes.  That means that for a bytes or bytearray object *b*, ``b[0]``\nwill be an integer, while ``b[0:1]`` will be a bytes or bytearray\nobject of length 1.  The representation of bytes objects uses the\nliteral format (``b\'...\'``) since it is generally more useful than\ne.g. ``bytes([50, 19, 100])``.  You can always convert a bytes object\ninto a list of integers using ``list(b)``.\n\nAlso, while in previous Python versions, byte strings and Unicode\nstrings could be exchanged for each other rather freely (barring\nencoding issues), strings and bytes are now completely separate\nconcepts.  There\'s no implicit en-/decoding if you pass an object of\nthe wrong type.  A string always compares unequal to a bytes or\nbytearray object.\n\nLists are constructed with square brackets, separating items with\ncommas: ``[a, b, c]``.  Tuples are constructed by the comma operator\n(not within square brackets), with or without enclosing parentheses,\nbut an empty tuple must have the enclosing parentheses, such as ``a,\nb, c`` or ``()``.  A single item tuple must have a trailing comma,\nsuch as ``(d,)``.\n\nObjects of type range are created using the ``range()`` function.\nThey don\'t support concatenation or repetition, and using ``min()`` or\n``max()`` on them is inefficient.\n\nMost sequence types support the following operations.  The ``in`` and\n``not in`` operations have the same priorities as the comparison\noperations.  The ``+`` and ``*`` operations have the same priority as\nthe corresponding numeric operations. [3] Additional methods are\nprovided for *Mutable Sequence Types*.\n\nThis table lists the sequence operations sorted in ascending priority\n(operations in the same box have the same priority).  In the table,\n*s* and *t* are sequences of the same type; *n*, *i*, *j* and *k* are\nintegers.\n\n+--------------------+----------------------------------+------------+\n| Operation          | Result                           | Notes      |\n+====================+==================================+============+\n| ``x in s``         | ``True`` if an item of *s* is    | (1)        |\n|                    | equal to *x*, else ``False``     |            |\n+--------------------+----------------------------------+------------+\n| ``x not in s``     | ``False`` if an item of *s* is   | (1)        |\n|                    | equal to *x*, else ``True``      |            |\n+--------------------+----------------------------------+------------+\n| ``s + t``          | the concatenation of *s* and *t* | (6)        |\n+--------------------+----------------------------------+------------+\n| ``s * n, n * s``   | *n* shallow copies of *s*        | (2)        |\n|                    | concatenated                     |            |\n+--------------------+----------------------------------+------------+\n| ``s[i]``           | *i*\'th item of *s*, origin 0     | (3)        |\n+--------------------+----------------------------------+------------+\n| ``s[i:j]``         | slice of *s* from *i* to *j*     | (3)(4)     |\n+--------------------+----------------------------------+------------+\n| ``s[i:j:k]``       | slice of *s* from *i* to *j*     | (3)(5)     |\n|                    | with step *k*                    |            |\n+--------------------+----------------------------------+------------+\n| ``len(s)``         | length of *s*                    |            |\n+--------------------+----------------------------------+------------+\n| ``min(s)``         | smallest item of *s*             |            |\n+--------------------+----------------------------------+------------+\n| ``max(s)``         | largest item of *s*              |            |\n+--------------------+----------------------------------+------------+\n| ``s.index(i)``     | index of the first occurence of  |            |\n|                    | *i* in *s*                       |            |\n+--------------------+----------------------------------+------------+\n| ``s.count(i)``     | total number of occurences of    |            |\n|                    | *i* in *s*                       |            |\n+--------------------+----------------------------------+------------+\n\nSequence types also support comparisons.  In particular, tuples and\nlists are compared lexicographically by comparing corresponding\nelements.  This means that to compare equal, every element must\ncompare equal and the two sequences must be of the same type and have\nthe same length.  (For full details see *Comparisons* in the language\nreference.)\n\nNotes:\n\n1. When *s* is a string object, the ``in`` and ``not in`` operations\n   act like a substring test.\n\n2. Values of *n* less than ``0`` are treated as ``0`` (which yields an\n   empty sequence of the same type as *s*).  Note also that the copies\n   are shallow; nested structures are not copied.  This often haunts\n   new Python programmers; consider:\n\n   >>> lists = [[]] * 3\n   >>> lists\n   [[], [], []]\n   >>> lists[0].append(3)\n   >>> lists\n   [[3], [3], [3]]\n\n   What has happened is that ``[[]]`` is a one-element list containing\n   an empty list, so all three elements of ``[[]] * 3`` are (pointers\n   to) this single empty list.  Modifying any of the elements of\n   ``lists`` modifies this single list. You can create a list of\n   different lists this way:\n\n   >>> lists = [[] for i in range(3)]\n   >>> lists[0].append(3)\n   >>> lists[1].append(5)\n   >>> lists[2].append(7)\n   >>> lists\n   [[3], [5], [7]]\n\n3. If *i* or *j* is negative, the index is relative to the end of the\n   string: ``len(s) + i`` or ``len(s) + j`` is substituted.  But note\n   that ``-0`` is still ``0``.\n\n4. The slice of *s* from *i* to *j* is defined as the sequence of\n   items with index *k* such that ``i <= k < j``.  If *i* or *j* is\n   greater than ``len(s)``, use ``len(s)``.  If *i* is omitted or\n   ``None``, use ``0``.  If *j* is omitted or ``None``, use\n   ``len(s)``.  If *i* is greater than or equal to *j*, the slice is\n   empty.\n\n5. The slice of *s* from *i* to *j* with step *k* is defined as the\n   sequence of items with index  ``x = i + n*k`` such that ``0 <= n <\n   (j-i)/k``.  In other words, the indices are ``i``, ``i+k``,\n   ``i+2*k``, ``i+3*k`` and so on, stopping when *j* is reached (but\n   never including *j*).  If *i* or *j* is greater than ``len(s)``,\n   use ``len(s)``.  If *i* or *j* are omitted or ``None``, they become\n   "end" values (which end depends on the sign of *k*).  Note, *k*\n   cannot be zero. If *k* is ``None``, it is treated like ``1``.\n\n6. **CPython implementation detail:** If *s* and *t* are both strings,\n   some Python implementations such as CPython can usually perform an\n   in-place optimization for assignments of the form ``s = s + t`` or\n   ``s += t``.  When applicable, this optimization makes quadratic\n   run-time much less likely.  This optimization is both version and\n   implementation dependent.  For performance sensitive code, it is\n   preferable to use the ``str.join()`` method which assures\n   consistent linear concatenation performance across versions and\n   implementations.\n\n\nString Methods\n==============\n\nString objects support the methods listed below.\n\nIn addition, Python\'s strings support the sequence type methods\ndescribed in the *Sequence Types --- str, bytes, bytearray, list,\ntuple, range* section. To output formatted strings, see the *String\nFormatting* section. Also, see the ``re`` module for string functions\nbased on regular expressions.\n\nstr.capitalize()\n\n   Return a copy of the string with its first character capitalized\n   and the rest lowercased.\n\nstr.center(width[, fillchar])\n\n   Return centered in a string of length *width*. Padding is done\n   using the specified *fillchar* (default is a space).\n\nstr.count(sub[, start[, end]])\n\n   Return the number of non-overlapping occurrences of substring *sub*\n   in the range [*start*, *end*].  Optional arguments *start* and\n   *end* are interpreted as in slice notation.\n\nstr.encode(encoding="utf-8", errors="strict")\n\n   Return an encoded version of the string as a bytes object. Default\n   encoding is ``\'utf-8\'``. *errors* may be given to set a different\n   error handling scheme. The default for *errors* is ``\'strict\'``,\n   meaning that encoding errors raise a ``UnicodeError``. Other\n   possible values are ``\'ignore\'``, ``\'replace\'``,\n   ``\'xmlcharrefreplace\'``, ``\'backslashreplace\'`` and any other name\n   registered via ``codecs.register_error()``, see section *Codec Base\n   Classes*. For a list of possible encodings, see section *Standard\n   Encodings*.\n\n   Changed in version 3.1: Support for keyword arguments added.\n\nstr.endswith(suffix[, start[, end]])\n\n   Return ``True`` if the string ends with the specified *suffix*,\n   otherwise return ``False``.  *suffix* can also be a tuple of\n   suffixes to look for.  With optional *start*, test beginning at\n   that position.  With optional *end*, stop comparing at that\n   position.\n\nstr.expandtabs([tabsize])\n\n   Return a copy of the string where all tab characters are replaced\n   by one or more spaces, depending on the current column and the\n   given tab size.  The column number is reset to zero after each\n   newline occurring in the string. If *tabsize* is not given, a tab\n   size of ``8`` characters is assumed.  This doesn\'t understand other\n   non-printing characters or escape sequences.\n\nstr.find(sub[, start[, end]])\n\n   Return the lowest index in the string where substring *sub* is\n   found, such that *sub* is contained in the slice ``s[start:end]``.\n   Optional arguments *start* and *end* are interpreted as in slice\n   notation.  Return ``-1`` if *sub* is not found.\n\n   Note: The ``find()`` method should be used only if you need to know the\n     position of *sub*.  To check if *sub* is a substring or not, use\n     the ``in`` operator:\n\n        >>> \'Py\' in \'Python\'\n        True\n\nstr.format(*args, **kwargs)\n\n   Perform a string formatting operation.  The string on which this\n   method is called can contain literal text or replacement fields\n   delimited by braces ``{}``.  Each replacement field contains either\n   the numeric index of a positional argument, or the name of a\n   keyword argument.  Returns a copy of the string where each\n   replacement field is replaced with the string value of the\n   corresponding argument.\n\n   >>> "The sum of 1 + 2 is {0}".format(1+2)\n   \'The sum of 1 + 2 is 3\'\n\n   See *Format String Syntax* for a description of the various\n   formatting options that can be specified in format strings.\n\nstr.format_map(mapping)\n\n   Similar to ``str.format(**mapping)``, except that ``mapping`` is\n   used directly and not copied to a ``dict`` .  This is useful if for\n   example ``mapping`` is a dict subclass:\n\n   >>> class Default(dict):\n   ...     def __missing__(self, key):\n   ...         return key\n   ...\n   >>> \'{name} was born in {country}\'.format_map(Default(name=\'Guido\'))\n   \'Guido was born in country\'\n\n   New in version 3.2.\n\nstr.index(sub[, start[, end]])\n\n   Like ``find()``, but raise ``ValueError`` when the substring is not\n   found.\n\nstr.isalnum()\n\n   Return true if all characters in the string are alphanumeric and\n   there is at least one character, false otherwise.  A character\n   ``c`` is alphanumeric if one of the following returns ``True``:\n   ``c.isalpha()``, ``c.isdecimal()``, ``c.isdigit()``, or\n   ``c.isnumeric()``.\n\nstr.isalpha()\n\n   Return true if all characters in the string are alphabetic and\n   there is at least one character, false otherwise.  Alphabetic\n   characters are those characters defined in the Unicode character\n   database as "Letter", i.e., those with general category property\n   being one of "Lm", "Lt", "Lu", "Ll", or "Lo".  Note that this is\n   different from the "Alphabetic" property defined in the Unicode\n   Standard.\n\nstr.isdecimal()\n\n   Return true if all characters in the string are decimal characters\n   and there is at least one character, false otherwise. Decimal\n   characters are those from general category "Nd". This category\n   includes digit characters, and all characters that that can be used\n   to form decimal-radix numbers, e.g. U+0660, ARABIC-INDIC DIGIT\n   ZERO.\n\nstr.isdigit()\n\n   Return true if all characters in the string are digits and there is\n   at least one character, false otherwise.  Digits include decimal\n   characters and digits that need special handling, such as the\n   compatibility superscript digits.  Formally, a digit is a character\n   that has the property value Numeric_Type=Digit or\n   Numeric_Type=Decimal.\n\nstr.isidentifier()\n\n   Return true if the string is a valid identifier according to the\n   language definition, section *Identifiers and keywords*.\n\nstr.islower()\n\n   Return true if all cased characters in the string are lowercase and\n   there is at least one cased character, false otherwise.  Cased\n   characters are those with general category property being one of\n   "Lu", "Ll", or "Lt" and lowercase characters are those with general\n   category property "Ll".\n\nstr.isnumeric()\n\n   Return true if all characters in the string are numeric characters,\n   and there is at least one character, false otherwise. Numeric\n   characters include digit characters, and all characters that have\n   the Unicode numeric value property, e.g. U+2155, VULGAR FRACTION\n   ONE FIFTH.  Formally, numeric characters are those with the\n   property value Numeric_Type=Digit, Numeric_Type=Decimal or\n   Numeric_Type=Numeric.\n\nstr.isprintable()\n\n   Return true if all characters in the string are printable or the\n   string is empty, false otherwise.  Nonprintable characters are\n   those characters defined in the Unicode character database as\n   "Other" or "Separator", excepting the ASCII space (0x20) which is\n   considered printable.  (Note that printable characters in this\n   context are those which should not be escaped when ``repr()`` is\n   invoked on a string.  It has no bearing on the handling of strings\n   written to ``sys.stdout`` or ``sys.stderr``.)\n\nstr.isspace()\n\n   Return true if there are only whitespace characters in the string\n   and there is at least one character, false otherwise.  Whitespace\n   characters  are those characters defined in the Unicode character\n   database as "Other" or "Separator" and those with bidirectional\n   property being one of "WS", "B", or "S".\n\nstr.istitle()\n\n   Return true if the string is a titlecased string and there is at\n   least one character, for example uppercase characters may only\n   follow uncased characters and lowercase characters only cased ones.\n   Return false otherwise.\n\nstr.isupper()\n\n   Return true if all cased characters in the string are uppercase and\n   there is at least one cased character, false otherwise. Cased\n   characters are those with general category property being one of\n   "Lu", "Ll", or "Lt" and uppercase characters are those with general\n   category property "Lu".\n\nstr.join(iterable)\n\n   Return a string which is the concatenation of the strings in the\n   *iterable* *iterable*.  A ``TypeError`` will be raised if there are\n   any non-string values in *seq*, including ``bytes`` objects.  The\n   separator between elements is the string providing this method.\n\nstr.ljust(width[, fillchar])\n\n   Return the string left justified in a string of length *width*.\n   Padding is done using the specified *fillchar* (default is a\n   space).  The original string is returned if *width* is less than\n   ``len(s)``.\n\nstr.lower()\n\n   Return a copy of the string converted to lowercase.\n\nstr.lstrip([chars])\n\n   Return a copy of the string with leading characters removed.  The\n   *chars* argument is a string specifying the set of characters to be\n   removed.  If omitted or ``None``, the *chars* argument defaults to\n   removing whitespace.  The *chars* argument is not a prefix; rather,\n   all combinations of its values are stripped:\n\n   >>> \'   spacious   \'.lstrip()\n   \'spacious   \'\n   >>> \'www.example.com\'.lstrip(\'cmowz.\')\n   \'example.com\'\n\nstatic str.maketrans(x[, y[, z]])\n\n   This static method returns a translation table usable for\n   ``str.translate()``.\n\n   If there is only one argument, it must be a dictionary mapping\n   Unicode ordinals (integers) or characters (strings of length 1) to\n   Unicode ordinals, strings (of arbitrary lengths) or None.\n   Character keys will then be converted to ordinals.\n\n   If there are two arguments, they must be strings of equal length,\n   and in the resulting dictionary, each character in x will be mapped\n   to the character at the same position in y.  If there is a third\n   argument, it must be a string, whose characters will be mapped to\n   None in the result.\n\nstr.partition(sep)\n\n   Split the string at the first occurrence of *sep*, and return a\n   3-tuple containing the part before the separator, the separator\n   itself, and the part after the separator.  If the separator is not\n   found, return a 3-tuple containing the string itself, followed by\n   two empty strings.\n\nstr.replace(old, new[, count])\n\n   Return a copy of the string with all occurrences of substring *old*\n   replaced by *new*.  If the optional argument *count* is given, only\n   the first *count* occurrences are replaced.\n\nstr.rfind(sub[, start[, end]])\n\n   Return the highest index in the string where substring *sub* is\n   found, such that *sub* is contained within ``s[start:end]``.\n   Optional arguments *start* and *end* are interpreted as in slice\n   notation.  Return ``-1`` on failure.\n\nstr.rindex(sub[, start[, end]])\n\n   Like ``rfind()`` but raises ``ValueError`` when the substring *sub*\n   is not found.\n\nstr.rjust(width[, fillchar])\n\n   Return the string right justified in a string of length *width*.\n   Padding is done using the specified *fillchar* (default is a\n   space). The original string is returned if *width* is less than\n   ``len(s)``.\n\nstr.rpartition(sep)\n\n   Split the string at the last occurrence of *sep*, and return a\n   3-tuple containing the part before the separator, the separator\n   itself, and the part after the separator.  If the separator is not\n   found, return a 3-tuple containing two empty strings, followed by\n   the string itself.\n\nstr.rsplit([sep[, maxsplit]])\n\n   Return a list of the words in the string, using *sep* as the\n   delimiter string. If *maxsplit* is given, at most *maxsplit* splits\n   are done, the *rightmost* ones.  If *sep* is not specified or\n   ``None``, any whitespace string is a separator.  Except for\n   splitting from the right, ``rsplit()`` behaves like ``split()``\n   which is described in detail below.\n\nstr.rstrip([chars])\n\n   Return a copy of the string with trailing characters removed.  The\n   *chars* argument is a string specifying the set of characters to be\n   removed.  If omitted or ``None``, the *chars* argument defaults to\n   removing whitespace.  The *chars* argument is not a suffix; rather,\n   all combinations of its values are stripped:\n\n   >>> \'   spacious   \'.rstrip()\n   \'   spacious\'\n   >>> \'mississippi\'.rstrip(\'ipz\')\n   \'mississ\'\n\nstr.split([sep[, maxsplit]])\n\n   Return a list of the words in the string, using *sep* as the\n   delimiter string.  If *maxsplit* is given, at most *maxsplit*\n   splits are done (thus, the list will have at most ``maxsplit+1``\n   elements).  If *maxsplit* is not specified, then there is no limit\n   on the number of splits (all possible splits are made).\n\n   If *sep* is given, consecutive delimiters are not grouped together\n   and are deemed to delimit empty strings (for example,\n   ``\'1,,2\'.split(\',\')`` returns ``[\'1\', \'\', \'2\']``).  The *sep*\n   argument may consist of multiple characters (for example,\n   ``\'1<>2<>3\'.split(\'<>\')`` returns ``[\'1\', \'2\', \'3\']``). Splitting\n   an empty string with a specified separator returns ``[\'\']``.\n\n   If *sep* is not specified or is ``None``, a different splitting\n   algorithm is applied: runs of consecutive whitespace are regarded\n   as a single separator, and the result will contain no empty strings\n   at the start or end if the string has leading or trailing\n   whitespace.  Consequently, splitting an empty string or a string\n   consisting of just whitespace with a ``None`` separator returns\n   ``[]``.\n\n   For example, ``\' 1  2   3  \'.split()`` returns ``[\'1\', \'2\', \'3\']``,\n   and ``\'  1  2   3  \'.split(None, 1)`` returns ``[\'1\', \'2   3  \']``.\n\nstr.splitlines([keepends])\n\n   Return a list of the lines in the string, breaking at line\n   boundaries.  Line breaks are not included in the resulting list\n   unless *keepends* is given and true.\n\nstr.startswith(prefix[, start[, end]])\n\n   Return ``True`` if string starts with the *prefix*, otherwise\n   return ``False``. *prefix* can also be a tuple of prefixes to look\n   for.  With optional *start*, test string beginning at that\n   position.  With optional *end*, stop comparing string at that\n   position.\n\nstr.strip([chars])\n\n   Return a copy of the string with the leading and trailing\n   characters removed. The *chars* argument is a string specifying the\n   set of characters to be removed. If omitted or ``None``, the\n   *chars* argument defaults to removing whitespace. The *chars*\n   argument is not a prefix or suffix; rather, all combinations of its\n   values are stripped:\n\n   >>> \'   spacious   \'.strip()\n   \'spacious\'\n   >>> \'www.example.com\'.strip(\'cmowz.\')\n   \'example\'\n\nstr.swapcase()\n\n   Return a copy of the string with uppercase characters converted to\n   lowercase and vice versa.\n\nstr.title()\n\n   Return a titlecased version of the string where words start with an\n   uppercase character and the remaining characters are lowercase.\n\n   The algorithm uses a simple language-independent definition of a\n   word as groups of consecutive letters.  The definition works in\n   many contexts but it means that apostrophes in contractions and\n   possessives form word boundaries, which may not be the desired\n   result:\n\n      >>> "they\'re bill\'s friends from the UK".title()\n      "They\'Re Bill\'S Friends From The Uk"\n\n   A workaround for apostrophes can be constructed using regular\n   expressions:\n\n      >>> import re\n      >>> def titlecase(s):\n              return re.sub(r"[A-Za-z]+(\'[A-Za-z]+)?",\n                            lambda mo: mo.group(0)[0].upper() +\n                                       mo.group(0)[1:].lower(),\n                            s)\n\n      >>> titlecase("they\'re bill\'s friends.")\n      "They\'re Bill\'s Friends."\n\nstr.translate(map)\n\n   Return a copy of the *s* where all characters have been mapped\n   through the *map* which must be a dictionary of Unicode ordinals\n   (integers) to Unicode ordinals, strings or ``None``.  Unmapped\n   characters are left untouched. Characters mapped to ``None`` are\n   deleted.\n\n   You can use ``str.maketrans()`` to create a translation map from\n   character-to-character mappings in different formats.\n\n   Note: An even more flexible approach is to create a custom character\n     mapping codec using the ``codecs`` module (see\n     ``encodings.cp1251`` for an example).\n\nstr.upper()\n\n   Return a copy of the string converted to uppercase.\n\nstr.zfill(width)\n\n   Return the numeric string left filled with zeros in a string of\n   length *width*.  A sign prefix is handled correctly.  The original\n   string is returned if *width* is less than ``len(s)``.\n\n\nOld String Formatting Operations\n================================\n\nNote: The formatting operations described here are obsolete and may go\n  away in future versions of Python.  Use the new *String Formatting*\n  in new code.\n\nString objects have one unique built-in operation: the ``%`` operator\n(modulo). This is also known as the string *formatting* or\n*interpolation* operator. Given ``format % values`` (where *format* is\na string), ``%`` conversion specifications in *format* are replaced\nwith zero or more elements of *values*. The effect is similar to the\nusing ``sprintf()`` in the C language.\n\nIf *format* requires a single argument, *values* may be a single non-\ntuple object. [4]  Otherwise, *values* must be a tuple with exactly\nthe number of items specified by the format string, or a single\nmapping object (for example, a dictionary).\n\nA conversion specifier contains two or more characters and has the\nfollowing components, which must occur in this order:\n\n1. The ``\'%\'`` character, which marks the start of the specifier.\n\n2. Mapping key (optional), consisting of a parenthesised sequence of\n   characters (for example, ``(somename)``).\n\n3. Conversion flags (optional), which affect the result of some\n   conversion types.\n\n4. Minimum field width (optional).  If specified as an ``\'*\'``\n   (asterisk), the actual width is read from the next element of the\n   tuple in *values*, and the object to convert comes after the\n   minimum field width and optional precision.\n\n5. Precision (optional), given as a ``\'.\'`` (dot) followed by the\n   precision.  If specified as ``\'*\'`` (an asterisk), the actual\n   precision is read from the next element of the tuple in *values*,\n   and the value to convert comes after the precision.\n\n6. Length modifier (optional).\n\n7. Conversion type.\n\nWhen the right argument is a dictionary (or other mapping type), then\nthe formats in the string *must* include a parenthesised mapping key\ninto that dictionary inserted immediately after the ``\'%\'`` character.\nThe mapping key selects the value to be formatted from the mapping.\nFor example:\n\n>>> print(\'%(language)s has %(number)03d quote types.\' %\n...       {\'language\': "Python", "number": 2})\nPython has 002 quote types.\n\nIn this case no ``*`` specifiers may occur in a format (since they\nrequire a sequential parameter list).\n\nThe conversion flag characters are:\n\n+-----------+-----------------------------------------------------------------------+\n| Flag      | Meaning                                                               |\n+===========+=======================================================================+\n| ``\'#\'``   | The value conversion will use the "alternate form" (where defined     |\n|           | below).                                                               |\n+-----------+-----------------------------------------------------------------------+\n| ``\'0\'``   | The conversion will be zero padded for numeric values.                |\n+-----------+-----------------------------------------------------------------------+\n| ``\'-\'``   | The converted value is left adjusted (overrides the ``\'0\'``           |\n|           | conversion if both are given).                                        |\n+-----------+-----------------------------------------------------------------------+\n| ``\' \'``   | (a space) A blank should be left before a positive number (or empty   |\n|           | string) produced by a signed conversion.                              |\n+-----------+-----------------------------------------------------------------------+\n| ``\'+\'``   | A sign character (``\'+\'`` or ``\'-\'``) will precede the conversion     |\n|           | (overrides a "space" flag).                                           |\n+-----------+-----------------------------------------------------------------------+\n\nA length modifier (``h``, ``l``, or ``L``) may be present, but is\nignored as it is not necessary for Python -- so e.g. ``%ld`` is\nidentical to ``%d``.\n\nThe conversion types are:\n\n+--------------+-------------------------------------------------------+---------+\n| Conversion   | Meaning                                               | Notes   |\n+==============+=======================================================+=========+\n| ``\'d\'``      | Signed integer decimal.                               |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'i\'``      | Signed integer decimal.                               |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'o\'``      | Signed octal value.                                   | (1)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'u\'``      | Obsolete type -- it is identical to ``\'d\'``.          | (7)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'x\'``      | Signed hexadecimal (lowercase).                       | (2)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'X\'``      | Signed hexadecimal (uppercase).                       | (2)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'e\'``      | Floating point exponential format (lowercase).        | (3)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'E\'``      | Floating point exponential format (uppercase).        | (3)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'f\'``      | Floating point decimal format.                        | (3)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'F\'``      | Floating point decimal format.                        | (3)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'g\'``      | Floating point format. Uses lowercase exponential     | (4)     |\n|              | format if exponent is less than -4 or not less than   |         |\n|              | precision, decimal format otherwise.                  |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'G\'``      | Floating point format. Uses uppercase exponential     | (4)     |\n|              | format if exponent is less than -4 or not less than   |         |\n|              | precision, decimal format otherwise.                  |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'c\'``      | Single character (accepts integer or single character |         |\n|              | string).                                              |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'r\'``      | String (converts any Python object using ``repr()``). | (5)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'s\'``      | String (converts any Python object using ``str()``).  | (5)     |\n+--------------+-------------------------------------------------------+---------+\n| ``\'a\'``      | String (converts any Python object using              | (5)     |\n|              | ``ascii()``).                                         |         |\n+--------------+-------------------------------------------------------+---------+\n| ``\'%\'``      | No argument is converted, results in a ``\'%\'``        |         |\n|              | character in the result.                              |         |\n+--------------+-------------------------------------------------------+---------+\n\nNotes:\n\n1. The alternate form causes a leading zero (``\'0\'``) to be inserted\n   between left-hand padding and the formatting of the number if the\n   leading character of the result is not already a zero.\n\n2. The alternate form causes a leading ``\'0x\'`` or ``\'0X\'`` (depending\n   on whether the ``\'x\'`` or ``\'X\'`` format was used) to be inserted\n   between left-hand padding and the formatting of the number if the\n   leading character of the result is not already a zero.\n\n3. The alternate form causes the result to always contain a decimal\n   point, even if no digits follow it.\n\n   The precision determines the number of digits after the decimal\n   point and defaults to 6.\n\n4. The alternate form causes the result to always contain a decimal\n   point, and trailing zeroes are not removed as they would otherwise\n   be.\n\n   The precision determines the number of significant digits before\n   and after the decimal point and defaults to 6.\n\n5. If precision is ``N``, the output is truncated to ``N`` characters.\n\n1. See **PEP 237**.\n\nSince Python strings have an explicit length, ``%s`` conversions do\nnot assume that ``\'\\0\'`` is the end of the string.\n\nChanged in version 3.1: ``%f`` conversions for numbers whose absolute\nvalue is over 1e50 are no longer replaced by ``%g`` conversions.\n\nAdditional string operations are defined in standard modules\n``string`` and ``re``.\n\n\nRange Type\n==========\n\nThe ``range`` type is an immutable sequence which is commonly used for\nlooping.  The advantage of the ``range`` type is that an ``range``\nobject will always take the same amount of memory, no matter the size\nof the range it represents.\n\nRange objects have relatively little behavior: they support indexing,\ncontains, iteration, the ``len()`` function, and the following\nmethods:\n\nrange.count(x)\n\n   Return the number of *i*\'s for which ``s[i] == x``.\n\n      New in version 3.2.\n\nrange.index(x)\n\n   Return the smallest *i* such that ``s[i] == x``.  Raises\n   ``ValueError`` when *x* is not in the range.\n\n      New in version 3.2.\n\n\nMutable Sequence Types\n======================\n\nList and bytearray objects support additional operations that allow\nin-place modification of the object.  Other mutable sequence types\n(when added to the language) should also support these operations.\nStrings and tuples are immutable sequence types: such objects cannot\nbe modified once created. The following operations are defined on\nmutable sequence types (where *x* is an arbitrary object).\n\nNote that while lists allow their items to be of any type, bytearray\nobject "items" are all integers in the range 0 <= x < 256.\n\n+--------------------------------+----------------------------------+-----------------------+\n| Operation                      | Result                           | Notes                 |\n+================================+==================================+=======================+\n| ``s[i] = x``                   | item *i* of *s* is replaced by   |                       |\n|                                | *x*                              |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j] = t``                 | slice of *s* from *i* to *j* is  |                       |\n|                                | replaced by the contents of the  |                       |\n|                                | iterable *t*                     |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j]``                 | same as ``s[i:j] = []``          |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j:k] = t``               | the elements of ``s[i:j:k]`` are | (1)                   |\n|                                | replaced by those of *t*         |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j:k]``               | removes the elements of          |                       |\n|                                | ``s[i:j:k]`` from the list       |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.append(x)``                | same as ``s[len(s):len(s)] =     |                       |\n|                                | [x]``                            |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.extend(x)``                | same as ``s[len(s):len(s)] = x`` | (2)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.count(x)``                 | return number of *i*\'s for which |                       |\n|                                | ``s[i] == x``                    |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.index(x[, i[, j]])``       | return smallest *k* such that    | (3)                   |\n|                                | ``s[k] == x`` and ``i <= k < j`` |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.insert(i, x)``             | same as ``s[i:i] = [x]``         | (4)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.pop([i])``                 | same as ``x = s[i]; del s[i];    | (5)                   |\n|                                | return x``                       |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.remove(x)``                | same as ``del s[s.index(x)]``    | (3)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.reverse()``                | reverses the items of *s* in     | (6)                   |\n|                                | place                            |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.sort([key[, reverse]])``   | sort the items of *s* in place   | (6), (7), (8)         |\n+--------------------------------+----------------------------------+-----------------------+\n\nNotes:\n\n1. *t* must have the same length as the slice it is replacing.\n\n2. *x* can be any iterable object.\n\n3. Raises ``ValueError`` when *x* is not found in *s*. When a negative\n   index is passed as the second or third parameter to the ``index()``\n   method, the sequence length is added, as for slice indices.  If it\n   is still negative, it is truncated to zero, as for slice indices.\n\n4. When a negative index is passed as the first parameter to the\n   ``insert()`` method, the sequence length is added, as for slice\n   indices.  If it is still negative, it is truncated to zero, as for\n   slice indices.\n\n5. The optional argument *i* defaults to ``-1``, so that by default\n   the last item is removed and returned.\n\n6. The ``sort()`` and ``reverse()`` methods modify the sequence in\n   place for economy of space when sorting or reversing a large\n   sequence.  To remind you that they operate by side effect, they\n   don\'t return the sorted or reversed sequence.\n\n7. The ``sort()`` method takes optional arguments for controlling the\n   comparisons.  Each must be specified as a keyword argument.\n\n   *key* specifies a function of one argument that is used to extract\n   a comparison key from each list element: ``key=str.lower``.  The\n   default value is ``None``. Use ``functools.cmp_to_key()`` to\n   convert an old-style *cmp* function to a *key* function.\n\n   *reverse* is a boolean value.  If set to ``True``, then the list\n   elements are sorted as if each comparison were reversed.\n\n   The ``sort()`` method is guaranteed to be stable.  A sort is stable\n   if it guarantees not to change the relative order of elements that\n   compare equal --- this is helpful for sorting in multiple passes\n   (for example, sort by department, then by salary grade).\n\n   **CPython implementation detail:** While a list is being sorted,\n   the effect of attempting to mutate, or even inspect, the list is\n   undefined.  The C implementation of Python makes the list appear\n   empty for the duration, and raises ``ValueError`` if it can detect\n   that the list has been mutated during a sort.\n\n8. ``sort()`` is not supported by ``bytearray`` objects.\n\n\nBytes and Byte Array Methods\n============================\n\nBytes and bytearray objects, being "strings of bytes", have all\nmethods found on strings, with the exception of ``encode()``,\n``format()`` and ``isidentifier()``, which do not make sense with\nthese types.  For converting the objects to strings, they have a\n``decode()`` method.\n\nWherever one of these methods needs to interpret the bytes as\ncharacters (e.g. the ``is...()`` methods), the ASCII character set is\nassumed.\n\nNote: The methods on bytes and bytearray objects don\'t accept strings as\n  their arguments, just as the methods on strings don\'t accept bytes\n  as their arguments.  For example, you have to write\n\n     a = "abc"\n     b = a.replace("a", "f")\n\n  and\n\n     a = b"abc"\n     b = a.replace(b"a", b"f")\n\nbytes.decode(encoding="utf-8", errors="strict")\nbytearray.decode(encoding="utf-8", errors="strict")\n\n   Return a string decoded from the given bytes.  Default encoding is\n   ``\'utf-8\'``. *errors* may be given to set a different error\n   handling scheme.  The default for *errors* is ``\'strict\'``, meaning\n   that encoding errors raise a ``UnicodeError``.  Other possible\n   values are ``\'ignore\'``, ``\'replace\'`` and any other name\n   registered via ``codecs.register_error()``, see section *Codec Base\n   Classes*. For a list of possible encodings, see section *Standard\n   Encodings*.\n\n   Changed in version 3.1: Added support for keyword arguments.\n\nThe bytes and bytearray types have an additional class method:\n\nclassmethod bytes.fromhex(string)\nclassmethod bytearray.fromhex(string)\n\n   This ``bytes`` class method returns a bytes or bytearray object,\n   decoding the given string object.  The string must contain two\n   hexadecimal digits per byte, spaces are ignored.\n\n   >>> bytes.fromhex(\'f0 f1f2  \')\n   b\'\\xf0\\xf1\\xf2\'\n\nThe maketrans and translate methods differ in semantics from the\nversions available on strings:\n\nbytes.translate(table[, delete])\nbytearray.translate(table[, delete])\n\n   Return a copy of the bytes or bytearray object where all bytes\n   occurring in the optional argument *delete* are removed, and the\n   remaining bytes have been mapped through the given translation\n   table, which must be a bytes object of length 256.\n\n   You can use the ``bytes.maketrans()`` method to create a\n   translation table.\n\n   Set the *table* argument to ``None`` for translations that only\n   delete characters:\n\n      >>> b\'read this short text\'.translate(None, b\'aeiou\')\n      b\'rd ths shrt txt\'\n\nstatic bytes.maketrans(from, to)\nstatic bytearray.maketrans(from, to)\n\n   This static method returns a translation table usable for\n   ``bytes.translate()`` that will map each character in *from* into\n   the character at the same position in *to*; *from* and *to* must be\n   bytes objects and have the same length.\n\n   New in version 3.1.\n',
  'typesseq-mutable': '\nMutable Sequence Types\n**********************\n\nList and bytearray objects support additional operations that allow\nin-place modification of the object.  Other mutable sequence types\n(when added to the language) should also support these operations.\nStrings and tuples are immutable sequence types: such objects cannot\nbe modified once created. The following operations are defined on\nmutable sequence types (where *x* is an arbitrary object).\n\nNote that while lists allow their items to be of any type, bytearray\nobject "items" are all integers in the range 0 <= x < 256.\n\n+--------------------------------+----------------------------------+-----------------------+\n| Operation                      | Result                           | Notes                 |\n+================================+==================================+=======================+\n| ``s[i] = x``                   | item *i* of *s* is replaced by   |                       |\n|                                | *x*                              |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j] = t``                 | slice of *s* from *i* to *j* is  |                       |\n|                                | replaced by the contents of the  |                       |\n|                                | iterable *t*                     |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j]``                 | same as ``s[i:j] = []``          |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j:k] = t``               | the elements of ``s[i:j:k]`` are | (1)                   |\n|                                | replaced by those of *t*         |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j:k]``               | removes the elements of          |                       |\n|                                | ``s[i:j:k]`` from the list       |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.append(x)``                | same as ``s[len(s):len(s)] =     |                       |\n|                                | [x]``                            |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.extend(x)``                | same as ``s[len(s):len(s)] = x`` | (2)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.count(x)``                 | return number of *i*\'s for which |                       |\n|                                | ``s[i] == x``                    |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.index(x[, i[, j]])``       | return smallest *k* such that    | (3)                   |\n|                                | ``s[k] == x`` and ``i <= k < j`` |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.insert(i, x)``             | same as ``s[i:i] = [x]``         | (4)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.pop([i])``                 | same as ``x = s[i]; del s[i];    | (5)                   |\n|                                | return x``                       |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.remove(x)``                | same as ``del s[s.index(x)]``    | (3)                   |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.reverse()``                | reverses the items of *s* in     | (6)                   |\n|                                | place                            |                       |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.sort([key[, reverse]])``   | sort the items of *s* in place   | (6), (7), (8)         |\n+--------------------------------+----------------------------------+-----------------------+\n\nNotes:\n\n1. *t* must have the same length as the slice it is replacing.\n\n2. *x* can be any iterable object.\n\n3. Raises ``ValueError`` when *x* is not found in *s*. When a negative\n   index is passed as the second or third parameter to the ``index()``\n   method, the sequence length is added, as for slice indices.  If it\n   is still negative, it is truncated to zero, as for slice indices.\n\n4. When a negative index is passed as the first parameter to the\n   ``insert()`` method, the sequence length is added, as for slice\n   indices.  If it is still negative, it is truncated to zero, as for\n   slice indices.\n\n5. The optional argument *i* defaults to ``-1``, so that by default\n   the last item is removed and returned.\n\n6. The ``sort()`` and ``reverse()`` methods modify the sequence in\n   place for economy of space when sorting or reversing a large\n   sequence.  To remind you that they operate by side effect, they\n   don\'t return the sorted or reversed sequence.\n\n7. The ``sort()`` method takes optional arguments for controlling the\n   comparisons.  Each must be specified as a keyword argument.\n\n   *key* specifies a function of one argument that is used to extract\n   a comparison key from each list element: ``key=str.lower``.  The\n   default value is ``None``. Use ``functools.cmp_to_key()`` to\n   convert an old-style *cmp* function to a *key* function.\n\n   *reverse* is a boolean value.  If set to ``True``, then the list\n   elements are sorted as if each comparison were reversed.\n\n   The ``sort()`` method is guaranteed to be stable.  A sort is stable\n   if it guarantees not to change the relative order of elements that\n   compare equal --- this is helpful for sorting in multiple passes\n   (for example, sort by department, then by salary grade).\n\n   **CPython implementation detail:** While a list is being sorted,\n   the effect of attempting to mutate, or even inspect, the list is\n   undefined.  The C implementation of Python makes the list appear\n   empty for the duration, and raises ``ValueError`` if it can detect\n   that the list has been mutated during a sort.\n\n8. ``sort()`` is not supported by ``bytearray`` objects.\n',
  'unary': '\nUnary arithmetic and bitwise operations\n***************************************\n\nAll unary arithmetic and bitwise operations have the same priority:\n\n   u_expr ::= power | "-" u_expr | "+" u_expr | "~" u_expr\n\nThe unary ``-`` (minus) operator yields the negation of its numeric\nargument.\n\nThe unary ``+`` (plus) operator yields its numeric argument unchanged.\n\nThe unary ``~`` (invert) operator yields the bitwise inversion of its\ninteger argument.  The bitwise inversion of ``x`` is defined as\n``-(x+1)``.  It only applies to integral numbers.\n\nIn all three cases, if the argument does not have the proper type, a\n``TypeError`` exception is raised.\n',
  'while': '\nThe ``while`` statement\n***********************\n\nThe ``while`` statement is used for repeated execution as long as an\nexpression is true:\n\n   while_stmt ::= "while" expression ":" suite\n                  ["else" ":" suite]\n\nThis repeatedly tests the expression and, if it is true, executes the\nfirst suite; if the expression is false (which may be the first time\nit is tested) the suite of the ``else`` clause, if present, is\nexecuted and the loop terminates.\n\nA ``break`` statement executed in the first suite terminates the loop\nwithout executing the ``else`` clause\'s suite.  A ``continue``\nstatement executed in the first suite skips the rest of the suite and\ngoes back to testing the expression.\n',
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -13,62 +13,81 @@
 - Issue #12802: the Windows error ERROR_DIRECTORY (numbered 267) is now
   mapped to POSIX errno ENOTDIR (previously EINVAL).
 
+- Issue #9200: The str.is* methods now work with strings that contain non-BMP
+  characters even in narrow Unicode builds.
+
+- Issue #12791: Break reference cycles early when a generator exits with
+  an exception.
+
+- Issue #12266: Fix str.capitalize() to correctly uppercase/lowercase
+  titlecased and cased non-letter characters.
+
+Library
+-------
+
+- Issue #12878: Expose a __dict__ attribute on io.IOBase and its subclasses.
+
+- Issue #12636: IDLE reads the coding cookie when executing a Python script.
+
+- Issue #12847: Fix a crash with negative PUT and LONG_BINPUT arguments in
+  the C pickle implementation.
+
+- Issue #11564: Avoid crashes when trying to pickle huge objects or containers
+  (more than 2**31 items).  Instead, in most cases, an OverflowError is raised.
+
+- Issue #12287: Fix a stack corruption in ossaudiodev module when the FD is
+  greater than FD_SETSIZE.
+
+- Issue #11657: Fix sending file descriptors over 255 over a multiprocessing
+  Pipe.
+
+- Issue #12213: Fix a buffering bug with interleaved reads and writes that
+  could appear on BufferedRandom streams.
+
+- Issue #12650: Fix a race condition where a subprocess.Popen could leak
+  resources (FD/zombie) when killed at the wrong time.
+
+Tests
+-----
+
+- Issue #12821: Fix test_fcntl failures on OpenBSD 5.
+
+
+What's New in Python 3.2.2?
+===========================
+
+*Release date: 03-Sep-2011*
+
+Core and Builtins
+-----------------
+
+- Issue #12326: sys.platform is now always 'linux2' on Linux, even if Python
+  is compiled on Linux 3.
+
 - Accept bytes for the AST string type. This is temporary until a proper fix in
   3.3.
 
-- Issue #9200: The str.is* methods now work with strings that contain non-BMP
-  characters even in narrow Unicode builds.
-
-- Issue #12791: Break reference cycles early when a generator exits with
-  an exception.
-
-- Issue #12266: Fix str.capitalize() to correctly uppercase/lowercase
-  titlecased and cased non-letter characters.
-
 Library
 -------
 
-- Issue #12878: Expose a __dict__ attribute on io.IOBase and its subclasses.
-
-- Issue #12636: IDLE reads the coding cookie when executing a Python script.
-
 - Issue #10946: The distutils commands bdist_dumb, bdist_wininst and bdist_msi
   now respect a --skip-build option given to bdist.
 
-- Issue #12847: Fix a crash with negative PUT and LONG_BINPUT arguments in
-  the C pickle implementation.
-
-- Issue #11564: Avoid crashes when trying to pickle huge objects or containers
-  (more than 2**31 items).  Instead, in most cases, an OverflowError is raised.
-
-- Issue #12287: Fix a stack corruption in ossaudiodev module when the FD is
-  greater than FD_SETSIZE.
-
 - Issue #12839: Fix crash in zlib module due to version mismatch.
   Fix by Richard M. Tew.
 
-- Issue #11657: Fix sending file descriptors over 255 over a multiprocessing
-  Pipe.
-
-- Issue #12213: Fix a buffering bug with interleaved reads and writes that
-  could appear on BufferedRandom streams.
-
-- Issue #12326: sys.platform is now always 'linux2' on Linux, even if Python
-  is compiled on Linux 3.
-
-- Issue #12650: Fix a race condition where a subprocess.Popen could leak
-  resources (FD/zombie) when killed at the wrong time.
-
-Tests
------
-
-- Issue #12821: Fix test_fcntl failures on OpenBSD 5.
-
-
-What's New in Python 3.2.2?
-===========================
-
-*Release date: XX-XXX-2011*
+Extension Modules
+-----------------
+
+- Issue #9651: Fix a crash when ctypes.create_string_buffer(0) was passed to
+  some functions like file.write().
+
+
+
+What's New in Python 3.2.2 release candidate 1?
+===============================================
+
+*Release date: 14-Aug-2011*
 
 Core and Builtins
 -----------------
@@ -200,9 +219,6 @@
 
 - Issue #11241: subclasses of ctypes.Array can now be subclassed.
 
-- Issue #9651: Fix a crash when ctypes.create_string_buffer(0) was passed to
-  some functions like file.write().
-
 - Issue #10309: Define _GNU_SOURCE so that mremap() gets the proper
   signature.  Without this, architectures where sizeof void* != sizeof int are
   broken.  Patch given by Hallvard B Furuseth.
diff --git a/Misc/RPM/python-3.2.spec b/Misc/RPM/python-3.2.spec
--- a/Misc/RPM/python-3.2.spec
+++ b/Misc/RPM/python-3.2.spec
@@ -39,7 +39,7 @@
 
 %define name python
 #--start constants--
-%define version 3.2.1
+%define version 3.2.2
 %define libvers 3.2
 #--end constants--
 %define release 1pydotorg
diff --git a/README b/README
--- a/README
+++ b/README
@@ -1,4 +1,4 @@
-This is Python version 3.2.1
+This is Python version 3.2.2
 ============================
 
 Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:48 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:48 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Post-release_ve?=
	=?utf8?q?rsion_bump=2E?=
Message-ID: <E1R06Po-0002bG-FC@dinsdale.python.org>

http://hg.python.org/cpython/rev/792606c351cf
changeset:   72266:792606c351cf
branch:      3.2
user:        Georg Brandl <georg at python.org>
date:        Sun Sep 04 08:36:22 2011 +0200
summary:
  Post-release version bump.

files:
  Include/patchlevel.h |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Include/patchlevel.h b/Include/patchlevel.h
--- a/Include/patchlevel.h
+++ b/Include/patchlevel.h
@@ -23,7 +23,7 @@
 #define PY_RELEASE_SERIAL	0
 
 /* Version as a string */
-#define PY_VERSION      	"3.2.2"
+#define PY_VERSION      	"3.2.2+"
 /*--end constants--*/
 
 /* Subversion Revision number of this file (not of the repository). Empty

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 08:42:49 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 04 Sep 2011 08:42:49 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge_with_3=2E2=2E?=
Message-ID: <E1R06Pp-0002bK-9P@dinsdale.python.org>

http://hg.python.org/cpython/rev/df673c6d3e0b
changeset:   72267:df673c6d3e0b
parent:      72219:fdd83a93ba37
parent:      72266:792606c351cf
user:        Georg Brandl <georg at python.org>
date:        Sun Sep 04 08:42:26 2011 +0200
summary:
  Merge with 3.2.

files:
  .hgtags                              |   2 ++
  Doc/library/sys.rst                  |   7 ++++++-
  Doc/license.rst                      |  10 +++++++++-
  Doc/tools/sphinxext/susp-ignored.csv |  16 ++++++++--------
  LICENSE                              |   6 +++++-
  5 files changed, 30 insertions(+), 11 deletions(-)


diff --git a/.hgtags b/.hgtags
--- a/.hgtags
+++ b/.hgtags
@@ -91,3 +91,5 @@
 cfa9364997c7f2e67b9cbb45c3a5fa3bba4e4999 v3.2.1rc1
 5df549718fb4841ff521fe051f6b54f290fad5d8 v3.2.1rc2
 ac1f7e5c05104d557d5acd922e95625ba5d1fe10 v3.2.1
+c860feaa348d663e598986894ee4680480577e15 v3.2.2rc1
+137e45f15c0bd262c9ad4c032d97425bc0589456 v3.2.2
diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -710,6 +710,8 @@
 
       if sys.platform.startswith('freebsd'):
           # FreeBSD-specific code here...
+      elif sys.platform.startswith('linux'):
+          # Linux-specific code here...
 
    For other systems, the values are:
 
@@ -726,7 +728,9 @@
 
    .. versionchanged:: 3.3
       On Linux, :attr:`sys.platform` doesn't contain the major version anymore.
-      It is always ``'linux'``, instead of ``'linux2'`` or ``'linux3'``.
+      It is always ``'linux'``, instead of ``'linux2'`` or ``'linux3'``.  Since
+      older Python versions include the version number, it is recommended to
+      always use the ``startswith`` idiom presented above.
 
    .. seealso::
       :attr:`os.name` has a coarser granularity.  :func:`os.uname` gives
@@ -735,6 +739,7 @@
       The :mod:`platform` module provides detailed checks for the
       system's identity.
 
+
 .. data:: prefix
 
    A string giving the site-specific directory prefix where the platform
diff --git a/Doc/license.rst b/Doc/license.rst
--- a/Doc/license.rst
+++ b/Doc/license.rst
@@ -106,10 +106,18 @@
 +----------------+--------------+------------+------------+-----------------+
 | 3.1.1          | 3.1          | 2009       | PSF        | yes             |
 +----------------+--------------+------------+------------+-----------------+
-| 3.1.2          | 3.1          | 2010       | PSF        | yes             |
+| 3.1.2          | 3.1.1        | 2010       | PSF        | yes             |
++----------------+--------------+------------+------------+-----------------+
+| 3.1.3          | 3.1.2        | 2010       | PSF        | yes             |
++----------------+--------------+------------+------------+-----------------+
+| 3.1.4          | 3.1.3        | 2011       | PSF        | yes             |
 +----------------+--------------+------------+------------+-----------------+
 | 3.2            | 3.1          | 2011       | PSF        | yes             |
 +----------------+--------------+------------+------------+-----------------+
+| 3.2.1          | 3.2          | 2011       | PSF        | yes             |
++----------------+--------------+------------+------------+-----------------+
+| 3.2.2          | 3.2.1        | 2011       | PSF        | yes             |
++----------------+--------------+------------+------------+-----------------+
 | 3.3            | 3.2          | 2012       | PSF        | yes             |
 +----------------+--------------+------------+------------+-----------------+
 
diff --git a/Doc/tools/sphinxext/susp-ignored.csv b/Doc/tools/sphinxext/susp-ignored.csv
--- a/Doc/tools/sphinxext/susp-ignored.csv
+++ b/Doc/tools/sphinxext/susp-ignored.csv
@@ -286,10 +286,10 @@
 documenting/rest,187,.. function:,.. function:: foo(x)
 documenting/rest,187,:bar,:bar: no
 documenting/rest,208,.. rubric:,.. rubric:: Footnotes
-faq/programming,762,:reduce,"print((lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+y,map(lambda y,"
-faq/programming,762,:reduce,"Sx=Sx,Sy=Sy:reduce(lambda x,y:x+y,map(lambda x,xc=Ru,yc=yc,Ru=Ru,Ro=Ro,"
-faq/programming,762,:chr,">=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr("
-faq/programming,1047,::,for x in sequence[::-1]:
+faq/programming,,:reduce,"print((lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+y,map(lambda y,"
+faq/programming,,:reduce,"Sx=Sx,Sy=Sy:reduce(lambda x,y:x+y,map(lambda x,xc=Ru,yc=yc,Ru=Ru,Ro=Ro,"
+faq/programming,,:chr,">=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr("
+faq/programming,,::,for x in sequence[::-1]:
 faq/windows,229,:EOF, at setlocal enableextensions & python -x %~f0 %* & goto :EOF
 faq/windows,393,:REG,.py :REG_SZ: c:\<path to python>\python.exe -u %s %s
 library/bisect,32,:hi,all(val >= x for val in a[i:hi])
@@ -310,10 +310,10 @@
 library/xmlrpc.client,103,:pass,http://user:pass at host:port/path
 library/xmlrpc.client,103,:port,http://user:pass at host:port/path
 library/xmlrpc.client,103,:pass,user:pass
-license,717,`,* THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
-license,717,`,* THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND
-license,879,`,"``Software''), to deal in the Software without restriction, including"
-license,879,`,"THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND,"
+license,,`,* THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
+license,,`,* THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND
+license,,`,"``Software''), to deal in the Software without restriction, including"
+license,,`,"THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND,"
 reference/lexical_analysis,704,`,$       ?       `
 whatsnew/2.7,735,:Sunday,'2009:4:Sunday'
 whatsnew/2.7,862,::,"export PYTHONWARNINGS=all,error:::Cookie:0"
diff --git a/LICENSE b/LICENSE
--- a/LICENSE
+++ b/LICENSE
@@ -67,8 +67,12 @@
     3.0.1           3.0         2009        PSF         yes
     3.1             3.0.1       2009        PSF         yes
     3.1.1           3.1         2009        PSF         yes
-    3.1.2           3.1         2010        PSF         yes
+    3.1.2           3.1.1       2010        PSF         yes
+    3.1.3           3.1.2       2010        PSF         yes
+    3.1.4           3.1.3       2011        PSF         yes
     3.2             3.1         2011        PSF         yes
+    3.2.1           3.2         2011        PSF         yes
+    3.2.2           3.2.1       2011        PSF         yes
     3.3             3.2         2012        PSF         yes
 
 Footnotes:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep  4 16:27:48 2011
From: python-checkins at python.org (ezio.melotti)
Date: Sun,  4 Sep 2011 16:27:48 +0200 (CEST)
Subject: [Python-checkins] r88893 - in
	tracker/instances/python-dev/extensions: local_replace.py
	test/local_replace_data.txt test/test_local_replace.py
Message-ID: <3RsGkD4XqMzNXk@mail.python.org>

Author: ezio.melotti
Date: Sun Sep  4 16:27:48 2011
New Revision: 88893

Log:
Tweak PEP regex to prevent linkification in URLs.

Modified:
   tracker/instances/python-dev/extensions/local_replace.py
   tracker/instances/python-dev/extensions/test/local_replace_data.txt
   tracker/instances/python-dev/extensions/test/test_local_replace.py

Modified: tracker/instances/python-dev/extensions/local_replace.py
==============================================================================
--- tracker/instances/python-dev/extensions/local_replace.py	(original)
+++ tracker/instances/python-dev/extensions/local_replace.py	Sun Sep  4 16:27:48 2011
@@ -98,7 +98,7 @@
      make_traceback_link),
 
     # PEP 8, PEP8, PEP 0008, ...
-    (re.compile(r'\b(?<!/)PEP\s*(\d{1,4})\b', re.I),
+    (re.compile(r'\b(?<![/=-])PEP\s*(\d{1,4})(?!/)\b', re.I),
      make_pep_link),
 
     # devguide

Modified: tracker/instances/python-dev/extensions/test/local_replace_data.txt
==============================================================================
--- tracker/instances/python-dev/extensions/test/local_replace_data.txt	(original)
+++ tracker/instances/python-dev/extensions/test/local_replace_data.txt	Sun Sep  4 16:27:48 2011
@@ -192,6 +192,14 @@
 write a PEP!
 http://wiki.python.org/moin/SummerOfCode/2011/PEP393
 <a href="http://wiki.python.org/moin/SummerOfCode/2011/PEP393">http://wiki.python.org/moin/SummerOfCode/2011/PEP393</a>
+see https://bitbucket.org/rndblnch/cpython-pep380/raw/tip/pep380
+see <a href="https://bitbucket.org/rndblnch/cpython-pep380/raw/tip/pep380">https://bitbucket.org/rndblnch/cpython-pep380/raw/tip/pep380</a>
+see https://bitbucket.org/rndblnch/cpython-pep380/qseries?apply=t&qs_apply=pep380
+see <a href="https://bitbucket.org/rndblnch/cpython-pep380/qseries?apply=t&amp;qs_apply=pep380">https://bitbucket.org/rndblnch/cpython-pep380/qseries?apply=t&amp;qs_apply=pep380</a>
+see https://bitbucket.org/rndblnch/cpython-pep380/wiki
+see <a href="https://bitbucket.org/rndblnch/cpython-pep380/wiki">https://bitbucket.org/rndblnch/cpython-pep380/wiki</a>
+see http://www.python.org/dev/peps/pep-0008/
+see <a href="http://www.python.org/dev/peps/pep-0008/">http://www.python.org/dev/peps/pep-0008/</a>
 ##
 ##
 see the devguide.

Modified: tracker/instances/python-dev/extensions/test/test_local_replace.py
==============================================================================
--- tracker/instances/python-dev/extensions/test/test_local_replace.py	(original)
+++ tracker/instances/python-dev/extensions/test/test_local_replace.py	Sun Sep  4 16:27:48 2011
@@ -59,7 +59,9 @@
                 continue  # skip the comments
             p = PyDevStringHTMLProperty(self.client, 'test', '1',
                                         None, 'test', text)
-            self.assertEqual(p.pydev_hyperlinked(), expected_result)
+            # decode the str -- Unicode strings have a better diff
+            self.assertEqual(p.pydev_hyperlinked().decode(),
+                             expected_result.decode())
 
 # run the tests
 unittest.main()

From python-checkins at python.org  Mon Sep  5 00:27:20 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Mon, 05 Sep 2011 00:27:20 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogQWRkIDMuMi4yIGFu?=
	=?utf8?q?d_3=2E2=2E3_UUIDs=2E?=
Message-ID: <E1R0L9s-0004LB-O5@dinsdale.python.org>

http://hg.python.org/cpython/rev/b38fdedb29d9
changeset:   72268:b38fdedb29d9
branch:      3.2
parent:      72266:792606c351cf
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Mon Sep 05 00:14:09 2011 +0200
summary:
  Add 3.2.2 and 3.2.3 UUIDs.

files:
  Tools/msi/uuids.py |  5 +++++
  1 files changed, 5 insertions(+), 0 deletions(-)


diff --git a/Tools/msi/uuids.py b/Tools/msi/uuids.py
--- a/Tools/msi/uuids.py
+++ b/Tools/msi/uuids.py
@@ -90,4 +90,9 @@
     '3.2.1121':'{4f90de4a-83dd-4443-b625-ca130ff361dd}', # 3.2.1rc1
     '3.2.1122':'{dc5eb04d-ff8a-4bed-8f96-23942fd59e5f}', # 3.2.1rc2
     '3.2.1150':'{34b2530c-6349-4292-9dc3-60bda4aed93c}', # 3.2.1
+    '3.2.2121':'{DFB29A53-ACC4-44e6-85A6-D0DA26FE8E4E}', # 3.2.2rc1
+    '3.2.2150':'{4CDE3168-D060-4b7c-BC74-4D8F9BB01AFD}', # 3.2.2
+    '3.2.3121':'{B8E8CFF7-E4C6-4a7c-9F06-BB3A8B75DDA8}', # 3.2.3rc1
+    '3.2.3150':'{789C9644-9F82-44d3-B4CA-AC31F46F5882}', # 3.2.3
+
 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 00:27:21 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Mon, 05 Sep 2011 00:27:21 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge_with_3=2E2?=
Message-ID: <E1R0L9t-0004LL-GM@dinsdale.python.org>

http://hg.python.org/cpython/rev/31b3218794e2
changeset:   72269:31b3218794e2
parent:      72267:df673c6d3e0b
parent:      72268:b38fdedb29d9
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Mon Sep 05 00:17:49 2011 +0200
summary:
  Merge with 3.2

files:
  Tools/msi/uuids.py |  5 +++++
  1 files changed, 5 insertions(+), 0 deletions(-)


diff --git a/Tools/msi/uuids.py b/Tools/msi/uuids.py
--- a/Tools/msi/uuids.py
+++ b/Tools/msi/uuids.py
@@ -30,4 +30,9 @@
     '3.2.1121':'{4f90de4a-83dd-4443-b625-ca130ff361dd}', # 3.2.1rc1
     '3.2.1122':'{dc5eb04d-ff8a-4bed-8f96-23942fd59e5f}', # 3.2.1rc2
     '3.2.1150':'{34b2530c-6349-4292-9dc3-60bda4aed93c}', # 3.2.1
+    '3.2.2121':'{DFB29A53-ACC4-44e6-85A6-D0DA26FE8E4E}', # 3.2.2rc1
+    '3.2.2150':'{4CDE3168-D060-4b7c-BC74-4D8F9BB01AFD}', # 3.2.2
+    '3.2.3121':'{B8E8CFF7-E4C6-4a7c-9F06-BB3A8B75DDA8}', # 3.2.3rc1
+    '3.2.3150':'{789C9644-9F82-44d3-B4CA-AC31F46F5882}', # 3.2.3
+
 }

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Mon Sep  5 05:19:45 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Mon, 05 Sep 2011 05:19:45 +0200
Subject: [Python-checkins] Daily reference leaks (31b3218794e2): sum=0
Message-ID: <E1R0Pir-0003z2-E4@ap.vmr.nerim.net>

results for 31b3218794e2 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogCfQOPU', '-x']

From python-checkins at python.org  Mon Sep  5 16:16:30 2011
From: python-checkins at python.org (ezio.melotti)
Date: Mon, 05 Sep 2011 16:16:30 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzEyODg4OiBGaXgg?=
 =?utf8?q?a_bug_in_HTMLParser=2Eunescape_that_prevented_it_to_escape_more_?=
 =?utf8?q?than?=
Message-ID: <E1R0ZyQ-0001Yd-8v@dinsdale.python.org>

http://hg.python.org/cpython/rev/9896fc2a8167
changeset:   72270:9896fc2a8167
branch:      3.2
parent:      72268:b38fdedb29d9
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Mon Sep 05 17:11:06 2011 +0300
summary:
  #12888: Fix a bug in HTMLParser.unescape that prevented it to escape more than 128 entities.  Patch by Peter Otten.

files:
  Lib/html/parser.py          |  2 +-
  Lib/test/test_htmlparser.py |  3 ++-
  Misc/ACKS                   |  1 +
  Misc/NEWS                   |  3 +++
  4 files changed, 7 insertions(+), 2 deletions(-)


diff --git a/Lib/html/parser.py b/Lib/html/parser.py
--- a/Lib/html/parser.py
+++ b/Lib/html/parser.py
@@ -458,4 +458,4 @@
                     return '&'+s+';'
 
         return re.sub(r"&(#?[xX]?(?:[0-9a-fA-F]+|\w{1,8}));",
-                      replaceEntities, s, re.ASCII)
+                      replaceEntities, s, flags=re.ASCII)
diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -377,7 +377,8 @@
         p = html.parser.HTMLParser()
         self.assertEqual(p.unescape('&#bad;'),'&#bad;')
         self.assertEqual(p.unescape('&#0038;'),'&')
-
+        # see #12888
+        self.assertEqual(p.unescape('&#123; ' * 1050), '{ ' * 1050)
 
 def test_main():
     support.run_unittest(HTMLParserTestCase, HTMLParserTolerantTestCase)
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -661,6 +661,7 @@
 Michele Orr?
 Oleg Oshmyan
 Denis S. Otkidach
+Peter Otten
 Michael Otteneder
 R. M. Oudkerk
 Russel Owen
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -25,6 +25,9 @@
 Library
 -------
 
+- Issue #12888: Fix a bug in HTMLParser.unescape that prevented it to escape
+  more than 128 entities.  Patch by Peter Otten.
+
 - Issue #12878: Expose a __dict__ attribute on io.IOBase and its subclasses.
 
 - Issue #12636: IDLE reads the coding cookie when executing a Python script.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 16:16:31 2011
From: python-checkins at python.org (ezio.melotti)
Date: Mon, 05 Sep 2011 16:16:31 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_=2312888=3A_merge_with_3=2E2=2E?=
Message-ID: <E1R0ZyR-0001Yh-73@dinsdale.python.org>

http://hg.python.org/cpython/rev/7b6096852665
changeset:   72271:7b6096852665
parent:      72269:31b3218794e2
parent:      72270:9896fc2a8167
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Mon Sep 05 17:15:32 2011 +0300
summary:
  #12888: merge with 3.2.

files:
  Lib/html/parser.py          |  2 +-
  Lib/test/test_htmlparser.py |  3 ++-
  Misc/ACKS                   |  1 +
  Misc/NEWS                   |  3 +++
  4 files changed, 7 insertions(+), 2 deletions(-)


diff --git a/Lib/html/parser.py b/Lib/html/parser.py
--- a/Lib/html/parser.py
+++ b/Lib/html/parser.py
@@ -458,4 +458,4 @@
                     return '&'+s+';'
 
         return re.sub(r"&(#?[xX]?(?:[0-9a-fA-F]+|\w{1,8}));",
-                      replaceEntities, s, re.ASCII)
+                      replaceEntities, s, flags=re.ASCII)
diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -377,7 +377,8 @@
         p = html.parser.HTMLParser()
         self.assertEqual(p.unescape('&#bad;'),'&#bad;')
         self.assertEqual(p.unescape('&#0038;'),'&')
-
+        # see #12888
+        self.assertEqual(p.unescape('&#123; ' * 1050), '{ ' * 1050)
 
 def test_main():
     support.run_unittest(HTMLParserTestCase, HTMLParserTolerantTestCase)
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -704,6 +704,7 @@
 Michele Orr?
 Oleg Oshmyan
 Denis S. Otkidach
+Peter Otten
 Michael Otteneder
 R. M. Oudkerk
 Russel Owen
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -271,6 +271,9 @@
 Library
 -------
 
+- Issue #12888: Fix a bug in HTMLParser.unescape that prevented it to escape
+  more than 128 entities.  Patch by Peter Otten.
+
 - Issue #12878: Expose a __dict__ attribute on io.IOBase and its subclasses.
 
 - Issue #12636: IDLE reads the coding cookie when executing a Python script.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 17:14:43 2011
From: python-checkins at python.org (lars.gustaebel)
Date: Mon, 05 Sep 2011 17:14:43 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyODQx?=
 =?utf8?q?=3A_Fix_tarfile_extraction_of_non-existent_uids/gids=2E?=
Message-ID: <E1R0asl-0006u5-4t@dinsdale.python.org>

http://hg.python.org/cpython/rev/2bc122347351
changeset:   72272:2bc122347351
branch:      3.2
parent:      72270:9896fc2a8167
user:        Lars Gust?bel <lars at gustaebel.de>
date:        Mon Sep 05 16:58:14 2011 +0200
summary:
  Issue #12841: Fix tarfile extraction of non-existent uids/gids.

tarfile unnecessarily checked the existence of numerical user and group ids on
extraction. If one of them did not exist the respective id of the current user
(i.e. root) was used for the file and ownership information was lost. (Patch
by Sebastien Luttringer)

files:
  Lib/tarfile.py |  10 ++--------
  Misc/NEWS      |   5 +++++
  2 files changed, 7 insertions(+), 8 deletions(-)


diff --git a/Lib/tarfile.py b/Lib/tarfile.py
--- a/Lib/tarfile.py
+++ b/Lib/tarfile.py
@@ -2368,17 +2368,11 @@
             try:
                 g = grp.getgrnam(tarinfo.gname)[2]
             except KeyError:
-                try:
-                    g = grp.getgrgid(tarinfo.gid)[2]
-                except KeyError:
-                    g = os.getgid()
+                g = tarinfo.gid
             try:
                 u = pwd.getpwnam(tarinfo.uname)[2]
             except KeyError:
-                try:
-                    u = pwd.getpwuid(tarinfo.uid)[2]
-                except KeyError:
-                    u = os.getuid()
+                u = tarinfo.uid
             try:
                 if tarinfo.issym() and hasattr(os, "lchown"):
                     os.lchown(targetpath, u, g)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -73,6 +73,11 @@
 Library
 -------
 
+- Issue #12841: tarfile unnecessarily checked the existence of numerical user
+  and group ids on extraction. If one of them did not exist the respective id
+  of the current user (i.e. root) was used for the file and ownership
+  information was lost.
+
 - Issue #10946: The distutils commands bdist_dumb, bdist_wininst and bdist_msi
   now respect a --skip-build option given to bdist.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 17:14:44 2011
From: python-checkins at python.org (lars.gustaebel)
Date: Mon, 05 Sep 2011 17:14:44 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_with_3=2E2=3A_Issue_=2312841=3A_Fix_tarfile_extraction?=
 =?utf8?q?_of_non-existent_uids/gids=2E?=
Message-ID: <E1R0asm-0006u9-2K@dinsdale.python.org>

http://hg.python.org/cpython/rev/da59abc0ce3b
changeset:   72273:da59abc0ce3b
parent:      72271:7b6096852665
parent:      72272:2bc122347351
user:        Lars Gust?bel <lars at gustaebel.de>
date:        Mon Sep 05 16:59:44 2011 +0200
summary:
  Merge with 3.2: Issue #12841: Fix tarfile extraction of non-existent uids/gids.

files:
  Lib/tarfile.py |  10 ++--------
  Misc/NEWS      |   5 +++++
  2 files changed, 7 insertions(+), 8 deletions(-)


diff --git a/Lib/tarfile.py b/Lib/tarfile.py
--- a/Lib/tarfile.py
+++ b/Lib/tarfile.py
@@ -2366,17 +2366,11 @@
             try:
                 g = grp.getgrnam(tarinfo.gname)[2]
             except KeyError:
-                try:
-                    g = grp.getgrgid(tarinfo.gid)[2]
-                except KeyError:
-                    g = os.getgid()
+                g = tarinfo.gid
             try:
                 u = pwd.getpwnam(tarinfo.uname)[2]
             except KeyError:
-                try:
-                    u = pwd.getpwuid(tarinfo.uid)[2]
-                except KeyError:
-                    u = os.getuid()
+                u = tarinfo.uid
             try:
                 if tarinfo.issym() and hasattr(os, "lchown"):
                     os.lchown(targetpath, u, g)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -271,6 +271,11 @@
 Library
 -------
 
+- Issue #12841: tarfile unnecessarily checked the existence of numerical user
+  and group ids on extraction. If one of them did not exist the respective id
+  of the current user (i.e. root) was used for the file and ownership
+  information was lost.
+
 - Issue #12888: Fix a bug in HTMLParser.unescape that prevented it to escape
   more than 128 entities.  Patch by Peter Otten.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 17:14:44 2011
From: python-checkins at python.org (lars.gustaebel)
Date: Mon, 05 Sep 2011 17:14:44 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyODQx?=
 =?utf8?q?=3A_Fix_tarfile_extraction_of_non-existent_uids/gids=2E?=
Message-ID: <E1R0asm-0006uD-Rk@dinsdale.python.org>

http://hg.python.org/cpython/rev/b64ef2951093
changeset:   72274:b64ef2951093
branch:      2.7
parent:      72209:af09b9d9be93
user:        Lars Gust?bel <lars at gustaebel.de>
date:        Mon Sep 05 17:04:18 2011 +0200
summary:
  Issue #12841: Fix tarfile extraction of non-existent uids/gids.

tarfile unnecessarily checked the existence of numerical user and group ids on
extraction. If one of them did not exist the respective id of the current user
(i.e. root) was used for the file and ownership information was lost. (Patch
by Sebastien Luttringer)

files:
  Lib/tarfile.py |  10 ++--------
  Misc/NEWS      |   5 +++++
  2 files changed, 7 insertions(+), 8 deletions(-)


diff --git a/Lib/tarfile.py b/Lib/tarfile.py
--- a/Lib/tarfile.py
+++ b/Lib/tarfile.py
@@ -2264,17 +2264,11 @@
             try:
                 g = grp.getgrnam(tarinfo.gname)[2]
             except KeyError:
-                try:
-                    g = grp.getgrgid(tarinfo.gid)[2]
-                except KeyError:
-                    g = os.getgid()
+                g = tarinfo.gid
             try:
                 u = pwd.getpwnam(tarinfo.uname)[2]
             except KeyError:
-                try:
-                    u = pwd.getpwuid(tarinfo.uid)[2]
-                except KeyError:
-                    u = os.getuid()
+                u = tarinfo.uid
             try:
                 if tarinfo.issym() and hasattr(os, "lchown"):
                     os.lchown(targetpath, u, g)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -40,6 +40,11 @@
 Library
 -------
 
+- Issue #12841: tarfile unnecessarily checked the existence of numerical user
+  and group ids on extraction. If one of them did not exist the respective id
+  of the current user (i.e. root) was used for the file and ownership
+  information was lost.
+
 - Issue #10946: The distutils commands bdist_dumb, bdist_wininst and bdist_msi
   now respect a --skip-build option given to bdist.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 17:46:06 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 05 Sep 2011 17:46:06 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Enable_catching?=
 =?utf8?q?_WARN-level_logging_messages_in_distutils=27_test=5Fsdist?=
Message-ID: <E1R0bN8-00019e-Fp@dinsdale.python.org>

http://hg.python.org/cpython/rev/a3e492ed389e
changeset:   72275:a3e492ed389e
branch:      3.2
parent:      72194:83c16d258fec
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 03 00:28:43 2011 +0200
summary:
  Enable catching WARN-level logging messages in distutils' test_sdist

files:
  Lib/distutils/tests/test_sdist.py |  9 ++++-----
  1 files changed, 4 insertions(+), 5 deletions(-)


diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -78,9 +78,6 @@
         dist.include_package_data = True
         cmd = sdist(dist)
         cmd.dist_dir = 'dist'
-        def _warn(*args):
-            pass
-        cmd.warn = _warn
         return dist, cmd
 
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
@@ -235,7 +232,8 @@
         # with the `check` subcommand
         cmd.ensure_finalized()
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 2)
 
         # trying with a complete set of metadata
@@ -244,7 +242,8 @@
         cmd.ensure_finalized()
         cmd.metadata_check = 0
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 0)
 
     def test_check_metadata_deprecated(self):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 17:46:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 05 Sep 2011 17:46:07 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Warn_instead_of?=
 =?utf8?q?_crashing_because_of_invalid_path_in_MANIFEST=2Ein_=28=238286=29?=
 =?utf8?q?=2E?=
Message-ID: <E1R0bN9-00019i-7O@dinsdale.python.org>

http://hg.python.org/cpython/rev/6cd8c1582dff
changeset:   72276:6cd8c1582dff
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 03 00:42:04 2011 +0200
summary:
  Warn instead of crashing because of invalid path in MANIFEST.in (#8286).

sdist used to crash with a full traceback dump instead of printing a
nice warning with the faulty line number.

files:
  Lib/distutils/command/sdist.py    |   5 ++-
  Lib/distutils/tests/test_sdist.py |  28 ++++++++++++++++++-
  Misc/NEWS                         |   3 ++
  3 files changed, 34 insertions(+), 2 deletions(-)


diff --git a/Lib/distutils/command/sdist.py b/Lib/distutils/command/sdist.py
--- a/Lib/distutils/command/sdist.py
+++ b/Lib/distutils/command/sdist.py
@@ -306,7 +306,10 @@
 
                 try:
                     self.filelist.process_template_line(line)
-                except DistutilsTemplateError as msg:
+                # the call above can raise a DistutilsTemplateError for
+                # malformed lines, or a ValueError from the lower-level
+                # convert_path function
+                except (DistutilsTemplateError, ValueError) as msg:
                     self.warn("%s, line %d: %s" % (template.filename,
                                                    template.current_line,
                                                    msg))
diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -15,6 +15,7 @@
 from distutils.errors import DistutilsOptionError
 from distutils.spawn import find_executable
 from distutils.log import WARN
+from distutils.filelist import FileList
 from distutils.archive_util import ARCHIVE_FORMATS
 
 SETUP_PY = """
@@ -265,7 +266,6 @@
         self.assertEqual(len(output), num_formats)
 
     def test_finalize_options(self):
-
         dist, cmd = self.get_cmd()
         cmd.finalize_options()
 
@@ -285,6 +285,32 @@
         cmd.formats = 'supazipa'
         self.assertRaises(DistutilsOptionError, cmd.finalize_options)
 
+    # the following tests make sure there is a nice error message instead
+    # of a traceback when parsing an invalid manifest template
+
+    def _test_template(self, content):
+        dist, cmd = self.get_cmd()
+        os.chdir(self.tmp_dir)
+        self.write_file('MANIFEST.in', content)
+        cmd.ensure_finalized()
+        cmd.filelist = FileList()
+        cmd.read_template()
+        warnings = self.get_logs(WARN)
+        self.assertEqual(len(warnings), 1)
+
+    def test_invalid_template_unknown_command(self):
+        self._test_template('taunt knights *')
+
+    def test_invalid_template_wrong_arguments(self):
+        # this manifest command takes one argument
+        self._test_template('prune')
+
+    @unittest.skipIf(os.name != 'nt', 'test relevant for Windows only')
+    def test_invalid_template_wrong_path(self):
+        # on Windows, trailing slashes are not allowed
+        # this used to crash instead of raising a warning: #8286
+        self._test_template('include examples/')
+
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
     def test_get_file_list(self):
         # make sure MANIFEST is recalculated
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -28,6 +28,9 @@
 Library
 -------
 
+- Issue #8286: The distutils command sdist will print a warning message instead
+  of crashing when an invalid path is given in the manifest template.
+
 - Issue #12636: IDLE reads the coding cookie when executing a Python script.
 
 - Issue #10946: The distutils commands bdist_dumb, bdist_wininst and bdist_msi

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 17:46:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 05 Sep 2011 17:46:07 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_fix_for_=238286_from_3=2E2?=
Message-ID: <E1R0bN9-00019m-W2@dinsdale.python.org>

http://hg.python.org/cpython/rev/b42661daa5cc
changeset:   72277:b42661daa5cc
parent:      72195:321b8372eef6
parent:      72276:6cd8c1582dff
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 03 00:48:17 2011 +0200
summary:
  Merge fix for #8286 from 3.2

files:
  Lib/distutils/command/sdist.py    |   5 ++-
  Lib/distutils/tests/test_sdist.py |  37 +++++++++++++++---
  Misc/NEWS                         |   3 +
  3 files changed, 38 insertions(+), 7 deletions(-)


diff --git a/Lib/distutils/command/sdist.py b/Lib/distutils/command/sdist.py
--- a/Lib/distutils/command/sdist.py
+++ b/Lib/distutils/command/sdist.py
@@ -306,7 +306,10 @@
 
                 try:
                     self.filelist.process_template_line(line)
-                except DistutilsTemplateError as msg:
+                # the call above can raise a DistutilsTemplateError for
+                # malformed lines, or a ValueError from the lower-level
+                # convert_path function
+                except (DistutilsTemplateError, ValueError) as msg:
                     self.warn("%s, line %d: %s" % (template.filename,
                                                    template.current_line,
                                                    msg))
diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -15,6 +15,7 @@
 from distutils.errors import DistutilsOptionError
 from distutils.spawn import find_executable
 from distutils.log import WARN
+from distutils.filelist import FileList
 from distutils.archive_util import ARCHIVE_FORMATS
 
 SETUP_PY = """
@@ -78,9 +79,6 @@
         dist.include_package_data = True
         cmd = sdist(dist)
         cmd.dist_dir = 'dist'
-        def _warn(*args):
-            pass
-        cmd.warn = _warn
         return dist, cmd
 
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
@@ -235,7 +233,8 @@
         # with the `check` subcommand
         cmd.ensure_finalized()
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 2)
 
         # trying with a complete set of metadata
@@ -244,7 +243,8 @@
         cmd.ensure_finalized()
         cmd.metadata_check = 0
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 0)
 
     def test_check_metadata_deprecated(self):
@@ -266,7 +266,6 @@
         self.assertEqual(len(output), num_formats)
 
     def test_finalize_options(self):
-
         dist, cmd = self.get_cmd()
         cmd.finalize_options()
 
@@ -286,6 +285,32 @@
         cmd.formats = 'supazipa'
         self.assertRaises(DistutilsOptionError, cmd.finalize_options)
 
+    # the following tests make sure there is a nice error message instead
+    # of a traceback when parsing an invalid manifest template
+
+    def _test_template(self, content):
+        dist, cmd = self.get_cmd()
+        os.chdir(self.tmp_dir)
+        self.write_file('MANIFEST.in', content)
+        cmd.ensure_finalized()
+        cmd.filelist = FileList()
+        cmd.read_template()
+        warnings = self.get_logs(WARN)
+        self.assertEqual(len(warnings), 1)
+
+    def test_invalid_template_unknown_command(self):
+        self._test_template('taunt knights *')
+
+    def test_invalid_template_wrong_arguments(self):
+        # this manifest command takes one argument
+        self._test_template('prune')
+
+    @unittest.skipIf(os.name != 'nt', 'test relevant for Windows only')
+    def test_invalid_template_wrong_path(self):
+        # on Windows, trailing slashes are not allowed
+        # this used to crash instead of raising a warning: #8286
+        self._test_template('include examples/')
+
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
     def test_get_file_list(self):
         # make sure MANIFEST is recalculated
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -271,6 +271,9 @@
 Library
 -------
 
+- Issue #8286: The distutils command sdist will print a warning message instead
+  of crashing when an invalid path is given in the manifest template.
+
 - Issue #12636: IDLE reads the coding cookie when executing a Python script.
 
 - Issue #12494: On error, call(), check_call(), check_output() and

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 17:46:08 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 05 Sep 2011 17:46:08 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMik6?=
	=?utf8?q?_Branch_merge?=
Message-ID: <E1R0bNA-00019q-OA@dinsdale.python.org>

http://hg.python.org/cpython/rev/3629063d1929
changeset:   72278:3629063d1929
branch:      3.2
parent:      72216:8ec8a4579788
parent:      72276:6cd8c1582dff
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 05 01:53:52 2011 +0200
summary:
  Branch merge

files:
  Lib/distutils/command/sdist.py    |   5 ++-
  Lib/distutils/tests/test_sdist.py |  37 +++++++++++++++---
  Misc/NEWS                         |   3 +
  3 files changed, 38 insertions(+), 7 deletions(-)


diff --git a/Lib/distutils/command/sdist.py b/Lib/distutils/command/sdist.py
--- a/Lib/distutils/command/sdist.py
+++ b/Lib/distutils/command/sdist.py
@@ -306,7 +306,10 @@
 
                 try:
                     self.filelist.process_template_line(line)
-                except DistutilsTemplateError as msg:
+                # the call above can raise a DistutilsTemplateError for
+                # malformed lines, or a ValueError from the lower-level
+                # convert_path function
+                except (DistutilsTemplateError, ValueError) as msg:
                     self.warn("%s, line %d: %s" % (template.filename,
                                                    template.current_line,
                                                    msg))
diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -15,6 +15,7 @@
 from distutils.errors import DistutilsOptionError
 from distutils.spawn import find_executable
 from distutils.log import WARN
+from distutils.filelist import FileList
 from distutils.archive_util import ARCHIVE_FORMATS
 
 SETUP_PY = """
@@ -78,9 +79,6 @@
         dist.include_package_data = True
         cmd = sdist(dist)
         cmd.dist_dir = 'dist'
-        def _warn(*args):
-            pass
-        cmd.warn = _warn
         return dist, cmd
 
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
@@ -235,7 +233,8 @@
         # with the `check` subcommand
         cmd.ensure_finalized()
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 2)
 
         # trying with a complete set of metadata
@@ -244,7 +243,8 @@
         cmd.ensure_finalized()
         cmd.metadata_check = 0
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 0)
 
     def test_check_metadata_deprecated(self):
@@ -266,7 +266,6 @@
         self.assertEqual(len(output), num_formats)
 
     def test_finalize_options(self):
-
         dist, cmd = self.get_cmd()
         cmd.finalize_options()
 
@@ -286,6 +285,32 @@
         cmd.formats = 'supazipa'
         self.assertRaises(DistutilsOptionError, cmd.finalize_options)
 
+    # the following tests make sure there is a nice error message instead
+    # of a traceback when parsing an invalid manifest template
+
+    def _test_template(self, content):
+        dist, cmd = self.get_cmd()
+        os.chdir(self.tmp_dir)
+        self.write_file('MANIFEST.in', content)
+        cmd.ensure_finalized()
+        cmd.filelist = FileList()
+        cmd.read_template()
+        warnings = self.get_logs(WARN)
+        self.assertEqual(len(warnings), 1)
+
+    def test_invalid_template_unknown_command(self):
+        self._test_template('taunt knights *')
+
+    def test_invalid_template_wrong_arguments(self):
+        # this manifest command takes one argument
+        self._test_template('prune')
+
+    @unittest.skipIf(os.name != 'nt', 'test relevant for Windows only')
+    def test_invalid_template_wrong_path(self):
+        # on Windows, trailing slashes are not allowed
+        # this used to crash instead of raising a warning: #8286
+        self._test_template('include examples/')
+
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
     def test_get_file_list(self):
         # make sure MANIFEST is recalculated
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -28,6 +28,9 @@
 Library
 -------
 
+- Issue #8286: The distutils command sdist will print a warning message instead
+  of crashing when an invalid path is given in the manifest template.
+
 - Issue #12878: Expose a __dict__ attribute on io.IOBase and its subclasses.
 
 - Issue #12636: IDLE reads the coding cookie when executing a Python script.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 17:46:09 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 05 Sep 2011 17:46:09 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Branch_merge?=
Message-ID: <E1R0bNB-00019u-Gg@dinsdale.python.org>

http://hg.python.org/cpython/rev/278c69260d0c
changeset:   72279:278c69260d0c
parent:      72217:fe0497bd7354
parent:      72277:b42661daa5cc
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 05 01:55:54 2011 +0200
summary:
  Branch merge

files:
  Lib/distutils/command/sdist.py    |   5 ++-
  Lib/distutils/tests/test_sdist.py |  37 +++++++++++++++---
  Misc/NEWS                         |   3 +
  3 files changed, 38 insertions(+), 7 deletions(-)


diff --git a/Lib/distutils/command/sdist.py b/Lib/distutils/command/sdist.py
--- a/Lib/distutils/command/sdist.py
+++ b/Lib/distutils/command/sdist.py
@@ -306,7 +306,10 @@
 
                 try:
                     self.filelist.process_template_line(line)
-                except DistutilsTemplateError as msg:
+                # the call above can raise a DistutilsTemplateError for
+                # malformed lines, or a ValueError from the lower-level
+                # convert_path function
+                except (DistutilsTemplateError, ValueError) as msg:
                     self.warn("%s, line %d: %s" % (template.filename,
                                                    template.current_line,
                                                    msg))
diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -15,6 +15,7 @@
 from distutils.errors import DistutilsOptionError
 from distutils.spawn import find_executable
 from distutils.log import WARN
+from distutils.filelist import FileList
 from distutils.archive_util import ARCHIVE_FORMATS
 
 SETUP_PY = """
@@ -78,9 +79,6 @@
         dist.include_package_data = True
         cmd = sdist(dist)
         cmd.dist_dir = 'dist'
-        def _warn(*args):
-            pass
-        cmd.warn = _warn
         return dist, cmd
 
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
@@ -235,7 +233,8 @@
         # with the `check` subcommand
         cmd.ensure_finalized()
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 2)
 
         # trying with a complete set of metadata
@@ -244,7 +243,8 @@
         cmd.ensure_finalized()
         cmd.metadata_check = 0
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 0)
 
     def test_check_metadata_deprecated(self):
@@ -266,7 +266,6 @@
         self.assertEqual(len(output), num_formats)
 
     def test_finalize_options(self):
-
         dist, cmd = self.get_cmd()
         cmd.finalize_options()
 
@@ -286,6 +285,32 @@
         cmd.formats = 'supazipa'
         self.assertRaises(DistutilsOptionError, cmd.finalize_options)
 
+    # the following tests make sure there is a nice error message instead
+    # of a traceback when parsing an invalid manifest template
+
+    def _test_template(self, content):
+        dist, cmd = self.get_cmd()
+        os.chdir(self.tmp_dir)
+        self.write_file('MANIFEST.in', content)
+        cmd.ensure_finalized()
+        cmd.filelist = FileList()
+        cmd.read_template()
+        warnings = self.get_logs(WARN)
+        self.assertEqual(len(warnings), 1)
+
+    def test_invalid_template_unknown_command(self):
+        self._test_template('taunt knights *')
+
+    def test_invalid_template_wrong_arguments(self):
+        # this manifest command takes one argument
+        self._test_template('prune')
+
+    @unittest.skipIf(os.name != 'nt', 'test relevant for Windows only')
+    def test_invalid_template_wrong_path(self):
+        # on Windows, trailing slashes are not allowed
+        # this used to crash instead of raising a warning: #8286
+        self._test_template('include examples/')
+
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
     def test_get_file_list(self):
         # make sure MANIFEST is recalculated
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -271,6 +271,9 @@
 Library
 -------
 
+- Issue #8286: The distutils command sdist will print a warning message instead
+  of crashing when an invalid path is given in the manifest template.
+
 - Issue #12878: Expose a __dict__ attribute on io.IOBase and its subclasses.
 
 - Issue #12636: IDLE reads the coding cookie when executing a Python script.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 17:46:10 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 05 Sep 2011 17:46:10 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge_3=2E2?=
Message-ID: <E1R0bNC-00019y-85@dinsdale.python.org>

http://hg.python.org/cpython/rev/783eec09e17f
changeset:   72280:783eec09e17f
parent:      72279:278c69260d0c
parent:      72278:3629063d1929
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 05 01:56:11 2011 +0200
summary:
  Merge 3.2

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 17:46:11 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 05 Sep 2011 17:46:11 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Branch_merge?=
Message-ID: <E1R0bND-0001A2-05@dinsdale.python.org>

http://hg.python.org/cpython/rev/78a7f0a1b4f3
changeset:   72281:78a7f0a1b4f3
parent:      72273:da59abc0ce3b
parent:      72280:783eec09e17f
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 05 17:44:47 2011 +0200
summary:
  Branch merge

files:
  Lib/distutils/command/sdist.py    |   5 ++-
  Lib/distutils/tests/test_sdist.py |  37 +++++++++++++++---
  Misc/NEWS                         |   3 +
  3 files changed, 38 insertions(+), 7 deletions(-)


diff --git a/Lib/distutils/command/sdist.py b/Lib/distutils/command/sdist.py
--- a/Lib/distutils/command/sdist.py
+++ b/Lib/distutils/command/sdist.py
@@ -306,7 +306,10 @@
 
                 try:
                     self.filelist.process_template_line(line)
-                except DistutilsTemplateError as msg:
+                # the call above can raise a DistutilsTemplateError for
+                # malformed lines, or a ValueError from the lower-level
+                # convert_path function
+                except (DistutilsTemplateError, ValueError) as msg:
                     self.warn("%s, line %d: %s" % (template.filename,
                                                    template.current_line,
                                                    msg))
diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -15,6 +15,7 @@
 from distutils.errors import DistutilsOptionError
 from distutils.spawn import find_executable
 from distutils.log import WARN
+from distutils.filelist import FileList
 from distutils.archive_util import ARCHIVE_FORMATS
 
 SETUP_PY = """
@@ -78,9 +79,6 @@
         dist.include_package_data = True
         cmd = sdist(dist)
         cmd.dist_dir = 'dist'
-        def _warn(*args):
-            pass
-        cmd.warn = _warn
         return dist, cmd
 
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
@@ -235,7 +233,8 @@
         # with the `check` subcommand
         cmd.ensure_finalized()
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 2)
 
         # trying with a complete set of metadata
@@ -244,7 +243,8 @@
         cmd.ensure_finalized()
         cmd.metadata_check = 0
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 0)
 
     def test_check_metadata_deprecated(self):
@@ -266,7 +266,6 @@
         self.assertEqual(len(output), num_formats)
 
     def test_finalize_options(self):
-
         dist, cmd = self.get_cmd()
         cmd.finalize_options()
 
@@ -286,6 +285,32 @@
         cmd.formats = 'supazipa'
         self.assertRaises(DistutilsOptionError, cmd.finalize_options)
 
+    # the following tests make sure there is a nice error message instead
+    # of a traceback when parsing an invalid manifest template
+
+    def _test_template(self, content):
+        dist, cmd = self.get_cmd()
+        os.chdir(self.tmp_dir)
+        self.write_file('MANIFEST.in', content)
+        cmd.ensure_finalized()
+        cmd.filelist = FileList()
+        cmd.read_template()
+        warnings = self.get_logs(WARN)
+        self.assertEqual(len(warnings), 1)
+
+    def test_invalid_template_unknown_command(self):
+        self._test_template('taunt knights *')
+
+    def test_invalid_template_wrong_arguments(self):
+        # this manifest command takes one argument
+        self._test_template('prune')
+
+    @unittest.skipIf(os.name != 'nt', 'test relevant for Windows only')
+    def test_invalid_template_wrong_path(self):
+        # on Windows, trailing slashes are not allowed
+        # this used to crash instead of raising a warning: #8286
+        self._test_template('include examples/')
+
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
     def test_get_file_list(self):
         # make sure MANIFEST is recalculated
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -271,6 +271,9 @@
 Library
 -------
 
+- Issue #8286: The distutils command sdist will print a warning message instead
+  of crashing when an invalid path is given in the manifest template.
+
 - Issue #12841: tarfile unnecessarily checked the existence of numerical user
   and group ids on extraction. If one of them did not exist the respective id
   of the current user (i.e. root) was used for the file and ownership

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 17:46:12 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 05 Sep 2011 17:46:12 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMik6?=
	=?utf8?q?_Branch_merge?=
Message-ID: <E1R0bNE-0001A6-7O@dinsdale.python.org>

http://hg.python.org/cpython/rev/c5b00467483b
changeset:   72282:c5b00467483b
branch:      3.2
parent:      72272:2bc122347351
parent:      72278:3629063d1929
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 05 17:44:07 2011 +0200
summary:
  Branch merge

files:
  Lib/distutils/command/sdist.py    |   5 ++-
  Lib/distutils/tests/test_sdist.py |  37 +++++++++++++++---
  Misc/NEWS                         |   3 +
  3 files changed, 38 insertions(+), 7 deletions(-)


diff --git a/Lib/distutils/command/sdist.py b/Lib/distutils/command/sdist.py
--- a/Lib/distutils/command/sdist.py
+++ b/Lib/distutils/command/sdist.py
@@ -306,7 +306,10 @@
 
                 try:
                     self.filelist.process_template_line(line)
-                except DistutilsTemplateError as msg:
+                # the call above can raise a DistutilsTemplateError for
+                # malformed lines, or a ValueError from the lower-level
+                # convert_path function
+                except (DistutilsTemplateError, ValueError) as msg:
                     self.warn("%s, line %d: %s" % (template.filename,
                                                    template.current_line,
                                                    msg))
diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -15,6 +15,7 @@
 from distutils.errors import DistutilsOptionError
 from distutils.spawn import find_executable
 from distutils.log import WARN
+from distutils.filelist import FileList
 from distutils.archive_util import ARCHIVE_FORMATS
 
 SETUP_PY = """
@@ -78,9 +79,6 @@
         dist.include_package_data = True
         cmd = sdist(dist)
         cmd.dist_dir = 'dist'
-        def _warn(*args):
-            pass
-        cmd.warn = _warn
         return dist, cmd
 
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
@@ -235,7 +233,8 @@
         # with the `check` subcommand
         cmd.ensure_finalized()
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 2)
 
         # trying with a complete set of metadata
@@ -244,7 +243,8 @@
         cmd.ensure_finalized()
         cmd.metadata_check = 0
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 0)
 
     def test_check_metadata_deprecated(self):
@@ -266,7 +266,6 @@
         self.assertEqual(len(output), num_formats)
 
     def test_finalize_options(self):
-
         dist, cmd = self.get_cmd()
         cmd.finalize_options()
 
@@ -286,6 +285,32 @@
         cmd.formats = 'supazipa'
         self.assertRaises(DistutilsOptionError, cmd.finalize_options)
 
+    # the following tests make sure there is a nice error message instead
+    # of a traceback when parsing an invalid manifest template
+
+    def _test_template(self, content):
+        dist, cmd = self.get_cmd()
+        os.chdir(self.tmp_dir)
+        self.write_file('MANIFEST.in', content)
+        cmd.ensure_finalized()
+        cmd.filelist = FileList()
+        cmd.read_template()
+        warnings = self.get_logs(WARN)
+        self.assertEqual(len(warnings), 1)
+
+    def test_invalid_template_unknown_command(self):
+        self._test_template('taunt knights *')
+
+    def test_invalid_template_wrong_arguments(self):
+        # this manifest command takes one argument
+        self._test_template('prune')
+
+    @unittest.skipIf(os.name != 'nt', 'test relevant for Windows only')
+    def test_invalid_template_wrong_path(self):
+        # on Windows, trailing slashes are not allowed
+        # this used to crash instead of raising a warning: #8286
+        self._test_template('include examples/')
+
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
     def test_get_file_list(self):
         # make sure MANIFEST is recalculated
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -72,6 +72,9 @@
 
 Library
 -------
+ 
+- Issue #8286: The distutils command sdist will print a warning message instead
+  of crashing when an invalid path is given in the manifest template.
 
 - Issue #12841: tarfile unnecessarily checked the existence of numerical user
   and group ids on extraction. If one of them did not exist the respective id

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 17:46:13 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 05 Sep 2011 17:46:13 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge_3=2E2?=
Message-ID: <E1R0bNF-0001AA-3y@dinsdale.python.org>

http://hg.python.org/cpython/rev/1cc83e603319
changeset:   72283:1cc83e603319
parent:      72281:78a7f0a1b4f3
parent:      72282:c5b00467483b
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 05 17:45:07 2011 +0200
summary:
  Merge 3.2

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 17:52:33 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 05 Sep 2011 17:52:33 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Enable_catching?=
 =?utf8?q?_WARN-level_logging_messages_in_distutils=27_test=5Fsdist?=
Message-ID: <E1R0bTN-0001fs-7M@dinsdale.python.org>

http://hg.python.org/cpython/rev/9b23e150d3d7
changeset:   72284:9b23e150d3d7
branch:      2.7
parent:      72202:f08d4f338b0c
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 03 00:28:43 2011 +0200
summary:
  Enable catching WARN-level logging messages in distutils' test_sdist

files:
  Lib/distutils/tests/test_sdist.py |  9 ++++-----
  1 files changed, 4 insertions(+), 5 deletions(-)


diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -85,9 +85,6 @@
         dist.include_package_data = True
         cmd = sdist(dist)
         cmd.dist_dir = 'dist'
-        def _warn(*args):
-            pass
-        cmd.warn = _warn
         return dist, cmd
 
     @unittest.skipUnless(zlib, "requires zlib")
@@ -242,7 +239,8 @@
         # with the `check` subcommand
         cmd.ensure_finalized()
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 2)
 
         # trying with a complete set of metadata
@@ -251,7 +249,8 @@
         cmd.ensure_finalized()
         cmd.metadata_check = 0
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 0)
 
     def test_check_metadata_deprecated(self):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 17:52:33 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 05 Sep 2011 17:52:33 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Warn_instead_of?=
 =?utf8?q?_crashing_because_of_invalid_path_in_MANIFEST=2Ein_=28=238286=29?=
 =?utf8?q?=2E?=
Message-ID: <E1R0bTN-0001gR-Rz@dinsdale.python.org>

http://hg.python.org/cpython/rev/9cdc845d5f2e
changeset:   72285:9cdc845d5f2e
branch:      2.7
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 03 00:47:07 2011 +0200
summary:
  Warn instead of crashing because of invalid path in MANIFEST.in (#8286).

sdist used to crash with a full traceback dump instead of printing a
nice warning with the faulty line number.

files:
  Lib/distutils/command/sdist.py    |   5 ++-
  Lib/distutils/tests/test_sdist.py |  28 ++++++++++++++++++-
  Misc/NEWS                         |   3 ++
  3 files changed, 34 insertions(+), 2 deletions(-)


diff --git a/Lib/distutils/command/sdist.py b/Lib/distutils/command/sdist.py
--- a/Lib/distutils/command/sdist.py
+++ b/Lib/distutils/command/sdist.py
@@ -320,7 +320,10 @@
 
                 try:
                     self.filelist.process_template_line(line)
-                except DistutilsTemplateError, msg:
+                # the call above can raise a DistutilsTemplateError for
+                # malformed lines, or a ValueError from the lower-level
+                # convert_path function
+                except (DistutilsTemplateError, ValueError) as msg:
                     self.warn("%s, line %d: %s" % (template.filename,
                                                    template.current_line,
                                                    msg))
diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -29,6 +29,7 @@
 from distutils.errors import DistutilsOptionError
 from distutils.spawn import find_executable
 from distutils.log import WARN
+from distutils.filelist import FileList
 from distutils.archive_util import ARCHIVE_FORMATS
 
 SETUP_PY = """
@@ -272,7 +273,6 @@
         self.assertEqual(len(output), num_formats)
 
     def test_finalize_options(self):
-
         dist, cmd = self.get_cmd()
         cmd.finalize_options()
 
@@ -342,6 +342,32 @@
         finally:
             archive.close()
 
+    # the following tests make sure there is a nice error message instead
+    # of a traceback when parsing an invalid manifest template
+
+    def _test_template(self, content):
+        dist, cmd = self.get_cmd()
+        os.chdir(self.tmp_dir)
+        self.write_file('MANIFEST.in', content)
+        cmd.ensure_finalized()
+        cmd.filelist = FileList()
+        cmd.read_template()
+        warnings = self.get_logs(WARN)
+        self.assertEqual(len(warnings), 1)
+
+    def test_invalid_template_unknown_command(self):
+        self._test_template('taunt knights *')
+
+    def test_invalid_template_wrong_arguments(self):
+        # this manifest command takes one argument
+        self._test_template('prune')
+
+    @unittest.skipIf(os.name != 'nt', 'test relevant for Windows only')
+    def test_invalid_template_wrong_path(self):
+        # on Windows, trailing slashes are not allowed
+        # this used to crash instead of raising a warning: #8286
+        self._test_template('include examples/')
+
     @unittest.skipUnless(zlib, "requires zlib")
     def test_get_file_list(self):
         # make sure MANIFEST is recalculated
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -40,6 +40,9 @@
 Library
 -------
 
+- Issue #8286: The distutils command sdist will print a warning message instead
+  of crashing when an invalid path is given in the manifest template.
+
 - Issue #10946: The distutils commands bdist_dumb, bdist_wininst and bdist_msi
   now respect a --skip-build option given to bdist.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 17:52:35 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 05 Sep 2011 17:52:35 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?=
	=?utf8?q?_Branch_merge?=
Message-ID: <E1R0bTP-0001gV-2d@dinsdale.python.org>

http://hg.python.org/cpython/rev/5da0e45519d4
changeset:   72286:5da0e45519d4
branch:      2.7
parent:      72274:b64ef2951093
parent:      72285:9cdc845d5f2e
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 05 17:45:48 2011 +0200
summary:
  Branch merge

files:
  Lib/distutils/command/sdist.py    |   5 ++-
  Lib/distutils/tests/test_sdist.py |  37 +++++++++++++++---
  Misc/NEWS                         |   3 +
  3 files changed, 38 insertions(+), 7 deletions(-)


diff --git a/Lib/distutils/command/sdist.py b/Lib/distutils/command/sdist.py
--- a/Lib/distutils/command/sdist.py
+++ b/Lib/distutils/command/sdist.py
@@ -320,7 +320,10 @@
 
                 try:
                     self.filelist.process_template_line(line)
-                except DistutilsTemplateError, msg:
+                # the call above can raise a DistutilsTemplateError for
+                # malformed lines, or a ValueError from the lower-level
+                # convert_path function
+                except (DistutilsTemplateError, ValueError) as msg:
                     self.warn("%s, line %d: %s" % (template.filename,
                                                    template.current_line,
                                                    msg))
diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -29,6 +29,7 @@
 from distutils.errors import DistutilsOptionError
 from distutils.spawn import find_executable
 from distutils.log import WARN
+from distutils.filelist import FileList
 from distutils.archive_util import ARCHIVE_FORMATS
 
 SETUP_PY = """
@@ -85,9 +86,6 @@
         dist.include_package_data = True
         cmd = sdist(dist)
         cmd.dist_dir = 'dist'
-        def _warn(*args):
-            pass
-        cmd.warn = _warn
         return dist, cmd
 
     @unittest.skipUnless(zlib, "requires zlib")
@@ -242,7 +240,8 @@
         # with the `check` subcommand
         cmd.ensure_finalized()
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 2)
 
         # trying with a complete set of metadata
@@ -251,7 +250,8 @@
         cmd.ensure_finalized()
         cmd.metadata_check = 0
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 0)
 
     def test_check_metadata_deprecated(self):
@@ -273,7 +273,6 @@
         self.assertEqual(len(output), num_formats)
 
     def test_finalize_options(self):
-
         dist, cmd = self.get_cmd()
         cmd.finalize_options()
 
@@ -343,6 +342,32 @@
         finally:
             archive.close()
 
+    # the following tests make sure there is a nice error message instead
+    # of a traceback when parsing an invalid manifest template
+
+    def _test_template(self, content):
+        dist, cmd = self.get_cmd()
+        os.chdir(self.tmp_dir)
+        self.write_file('MANIFEST.in', content)
+        cmd.ensure_finalized()
+        cmd.filelist = FileList()
+        cmd.read_template()
+        warnings = self.get_logs(WARN)
+        self.assertEqual(len(warnings), 1)
+
+    def test_invalid_template_unknown_command(self):
+        self._test_template('taunt knights *')
+
+    def test_invalid_template_wrong_arguments(self):
+        # this manifest command takes one argument
+        self._test_template('prune')
+
+    @unittest.skipIf(os.name != 'nt', 'test relevant for Windows only')
+    def test_invalid_template_wrong_path(self):
+        # on Windows, trailing slashes are not allowed
+        # this used to crash instead of raising a warning: #8286
+        self._test_template('include examples/')
+
     @unittest.skipUnless(zlib, "requires zlib")
     def test_get_file_list(self):
         # make sure MANIFEST is recalculated
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -39,6 +39,9 @@
 
 Library
 -------
+ 
+- Issue #8286: The distutils command sdist will print a warning message instead
+  of crashing when an invalid path is given in the manifest template.
 
 - Issue #12841: tarfile unnecessarily checked the existence of numerical user
   and group ids on extraction. If one of them did not exist the respective id

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 18:23:57 2011
From: python-checkins at python.org (senthil.kumaran)
Date: Mon, 05 Sep 2011 18:23:57 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_closes_Issu?=
 =?utf8?q?e11155__-_Correct_the_multiprocessing=2EQueue=2Eput=27s_arg_=28r?=
 =?utf8?q?eplace?=
Message-ID: <E1R0bxl-0004Mw-Vv@dinsdale.python.org>

http://hg.python.org/cpython/rev/c6d4d4d64405
changeset:   72287:c6d4d4d64405
branch:      3.2
parent:      72282:c5b00467483b
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Tue Sep 06 00:21:30 2011 +0800
summary:
  Fix closes Issue11155  - Correct the multiprocessing.Queue.put's arg (replace 'item' with 'obj') in the docs. Patch by Westley Mart?nez.

files:
  Doc/library/multiprocessing.rst |  12 ++++++------
  1 files changed, 6 insertions(+), 6 deletions(-)


diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst
--- a/Doc/library/multiprocessing.rst
+++ b/Doc/library/multiprocessing.rst
@@ -552,9 +552,9 @@
       Return ``True`` if the queue is full, ``False`` otherwise.  Because of
       multithreading/multiprocessing semantics, this is not reliable.
 
-   .. method:: put(item[, block[, timeout]])
-
-      Put item into the queue.  If the optional argument *block* is ``True``
+   .. method:: put(obj[, block[, timeout]])
+
+      Put obj into the queue.  If the optional argument *block* is ``True``
       (the default) and *timeout* is ``None`` (the default), block if necessary until
       a free slot is available.  If *timeout* is a positive number, it blocks at
       most *timeout* seconds and raises the :exc:`queue.Full` exception if no
@@ -563,9 +563,9 @@
       available, else raise the :exc:`queue.Full` exception (*timeout* is
       ignored in that case).
 
-   .. method:: put_nowait(item)
-
-      Equivalent to ``put(item, False)``.
+   .. method:: put_nowait(obj)
+
+      Equivalent to ``put(obj, False)``.
 
    .. method:: get([block[, timeout]])
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 18:23:58 2011
From: python-checkins at python.org (senthil.kumaran)
Date: Mon, 05 Sep 2011 18:23:58 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_merge_from_3=2E2=2E_Fix_closes_Issue11155__-_Correct_the?=
Message-ID: <E1R0bxm-0004N1-Pg@dinsdale.python.org>

http://hg.python.org/cpython/rev/8f1187288fac
changeset:   72288:8f1187288fac
parent:      72283:1cc83e603319
parent:      72287:c6d4d4d64405
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Tue Sep 06 00:22:15 2011 +0800
summary:
  merge from 3.2. Fix closes Issue11155  - Correct the multiprocessing.Queue.put's arg (replace 'item' with 'obj') in the docs. Patch by Westley Mart?nez.

files:
  Doc/library/multiprocessing.rst |  12 ++++++------
  1 files changed, 6 insertions(+), 6 deletions(-)


diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst
--- a/Doc/library/multiprocessing.rst
+++ b/Doc/library/multiprocessing.rst
@@ -571,9 +571,9 @@
       Return ``True`` if the queue is full, ``False`` otherwise.  Because of
       multithreading/multiprocessing semantics, this is not reliable.
 
-   .. method:: put(item[, block[, timeout]])
-
-      Put item into the queue.  If the optional argument *block* is ``True``
+   .. method:: put(obj[, block[, timeout]])
+
+      Put obj into the queue.  If the optional argument *block* is ``True``
       (the default) and *timeout* is ``None`` (the default), block if necessary until
       a free slot is available.  If *timeout* is a positive number, it blocks at
       most *timeout* seconds and raises the :exc:`queue.Full` exception if no
@@ -582,9 +582,9 @@
       available, else raise the :exc:`queue.Full` exception (*timeout* is
       ignored in that case).
 
-   .. method:: put_nowait(item)
-
-      Equivalent to ``put(item, False)``.
+   .. method:: put_nowait(obj)
+
+      Equivalent to ``put(obj, False)``.
 
    .. method:: get([block[, timeout]])
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 18:23:59 2011
From: python-checkins at python.org (senthil.kumaran)
Date: Mon, 05 Sep 2011 18:23:59 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_merge_from_3=2E?=
 =?utf8?q?2=2E__Fix_closes_Issue11155__-_Correct_the?=
Message-ID: <E1R0bxn-0004N5-Jl@dinsdale.python.org>

http://hg.python.org/cpython/rev/d29c9006d770
changeset:   72289:d29c9006d770
branch:      2.7
parent:      72286:5da0e45519d4
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Tue Sep 06 00:23:10 2011 +0800
summary:
  merge from 3.2.  Fix closes Issue11155  - Correct the multiprocessing.Queue.put's arg (replace 'item' with 'obj') in the docs. Patch by Westley Mart?nez.

files:
  Doc/library/multiprocessing.rst |  12 ++++++------
  1 files changed, 6 insertions(+), 6 deletions(-)


diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst
--- a/Doc/library/multiprocessing.rst
+++ b/Doc/library/multiprocessing.rst
@@ -552,9 +552,9 @@
       Return ``True`` if the queue is full, ``False`` otherwise.  Because of
       multithreading/multiprocessing semantics, this is not reliable.
 
-   .. method:: put(item[, block[, timeout]])
-
-      Put item into the queue.  If the optional argument *block* is ``True``
+   .. method:: put(obj[, block[, timeout]])
+
+      Put obj into the queue.  If the optional argument *block* is ``True``
       (the default) and *timeout* is ``None`` (the default), block if necessary until
       a free slot is available.  If *timeout* is a positive number, it blocks at
       most *timeout* seconds and raises the :exc:`Queue.Full` exception if no
@@ -563,9 +563,9 @@
       available, else raise the :exc:`Queue.Full` exception (*timeout* is
       ignored in that case).
 
-   .. method:: put_nowait(item)
-
-      Equivalent to ``put(item, False)``.
+   .. method:: put_nowait(obj)
+
+      Equivalent to ``put(obj, False)``.
 
    .. method:: get([block[, timeout]])
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 21:40:11 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 05 Sep 2011 21:40:11 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyMzI2?=
 =?utf8?q?=3A_Remove_plat-linux3_directory?=
Message-ID: <E1R0f1f-0003E7-IP@dinsdale.python.org>

http://hg.python.org/cpython/rev/d95c4b030eac
changeset:   72290:d95c4b030eac
branch:      2.7
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Sep 05 21:38:05 2011 +0200
summary:
  Issue #12326: Remove plat-linux3 directory

sys.platform is now always 'linux2' on Linux (even on Linux 3)

files:
  Lib/plat-linux3/CDROM.py |  207 ---------
  Lib/plat-linux3/DLFCN.py |   83 ---
  Lib/plat-linux3/IN.py    |  615 ---------------------------
  Lib/plat-linux3/TYPES.py |  170 -------
  Lib/plat-linux3/regen    |    8 -
  5 files changed, 0 insertions(+), 1083 deletions(-)


diff --git a/Lib/plat-linux3/CDROM.py b/Lib/plat-linux3/CDROM.py
deleted file mode 100644
--- a/Lib/plat-linux3/CDROM.py
+++ /dev/null
@@ -1,207 +0,0 @@
-# Generated by h2py from /usr/include/linux/cdrom.h
-
-CDROMPAUSE = 0x5301
-CDROMRESUME = 0x5302
-CDROMPLAYMSF = 0x5303
-CDROMPLAYTRKIND = 0x5304
-CDROMREADTOCHDR = 0x5305
-CDROMREADTOCENTRY = 0x5306
-CDROMSTOP = 0x5307
-CDROMSTART = 0x5308
-CDROMEJECT = 0x5309
-CDROMVOLCTRL = 0x530a
-CDROMSUBCHNL = 0x530b
-CDROMREADMODE2 = 0x530c
-CDROMREADMODE1 = 0x530d
-CDROMREADAUDIO = 0x530e
-CDROMEJECT_SW = 0x530f
-CDROMMULTISESSION = 0x5310
-CDROM_GET_MCN = 0x5311
-CDROM_GET_UPC = CDROM_GET_MCN
-CDROMRESET = 0x5312
-CDROMVOLREAD = 0x5313
-CDROMREADRAW = 0x5314
-CDROMREADCOOKED = 0x5315
-CDROMSEEK = 0x5316
-CDROMPLAYBLK = 0x5317
-CDROMREADALL = 0x5318
-CDROMGETSPINDOWN = 0x531d
-CDROMSETSPINDOWN = 0x531e
-CDROMCLOSETRAY = 0x5319
-CDROM_SET_OPTIONS = 0x5320
-CDROM_CLEAR_OPTIONS = 0x5321
-CDROM_SELECT_SPEED = 0x5322
-CDROM_SELECT_DISC = 0x5323
-CDROM_MEDIA_CHANGED = 0x5325
-CDROM_DRIVE_STATUS = 0x5326
-CDROM_DISC_STATUS = 0x5327
-CDROM_CHANGER_NSLOTS = 0x5328
-CDROM_LOCKDOOR = 0x5329
-CDROM_DEBUG = 0x5330
-CDROM_GET_CAPABILITY = 0x5331
-CDROMAUDIOBUFSIZ = 0x5382
-DVD_READ_STRUCT = 0x5390
-DVD_WRITE_STRUCT = 0x5391
-DVD_AUTH = 0x5392
-CDROM_SEND_PACKET = 0x5393
-CDROM_NEXT_WRITABLE = 0x5394
-CDROM_LAST_WRITTEN = 0x5395
-CDROM_PACKET_SIZE = 12
-CGC_DATA_UNKNOWN = 0
-CGC_DATA_WRITE = 1
-CGC_DATA_READ = 2
-CGC_DATA_NONE = 3
-CD_MINS = 74
-CD_SECS = 60
-CD_FRAMES = 75
-CD_SYNC_SIZE = 12
-CD_MSF_OFFSET = 150
-CD_CHUNK_SIZE = 24
-CD_NUM_OF_CHUNKS = 98
-CD_FRAMESIZE_SUB = 96
-CD_HEAD_SIZE = 4
-CD_SUBHEAD_SIZE = 8
-CD_EDC_SIZE = 4
-CD_ZERO_SIZE = 8
-CD_ECC_SIZE = 276
-CD_FRAMESIZE = 2048
-CD_FRAMESIZE_RAW = 2352
-CD_FRAMESIZE_RAWER = 2646
-CD_FRAMESIZE_RAW1 = (CD_FRAMESIZE_RAW-CD_SYNC_SIZE)
-CD_FRAMESIZE_RAW0 = (CD_FRAMESIZE_RAW-CD_SYNC_SIZE-CD_HEAD_SIZE)
-CD_XA_HEAD = (CD_HEAD_SIZE+CD_SUBHEAD_SIZE)
-CD_XA_TAIL = (CD_EDC_SIZE+CD_ECC_SIZE)
-CD_XA_SYNC_HEAD = (CD_SYNC_SIZE+CD_XA_HEAD)
-CDROM_LBA = 0x01
-CDROM_MSF = 0x02
-CDROM_DATA_TRACK = 0x04
-CDROM_LEADOUT = 0xAA
-CDROM_AUDIO_INVALID = 0x00
-CDROM_AUDIO_PLAY = 0x11
-CDROM_AUDIO_PAUSED = 0x12
-CDROM_AUDIO_COMPLETED = 0x13
-CDROM_AUDIO_ERROR = 0x14
-CDROM_AUDIO_NO_STATUS = 0x15
-CDC_CLOSE_TRAY = 0x1
-CDC_OPEN_TRAY = 0x2
-CDC_LOCK = 0x4
-CDC_SELECT_SPEED = 0x8
-CDC_SELECT_DISC = 0x10
-CDC_MULTI_SESSION = 0x20
-CDC_MCN = 0x40
-CDC_MEDIA_CHANGED = 0x80
-CDC_PLAY_AUDIO = 0x100
-CDC_RESET = 0x200
-CDC_IOCTLS = 0x400
-CDC_DRIVE_STATUS = 0x800
-CDC_GENERIC_PACKET = 0x1000
-CDC_CD_R = 0x2000
-CDC_CD_RW = 0x4000
-CDC_DVD = 0x8000
-CDC_DVD_R = 0x10000
-CDC_DVD_RAM = 0x20000
-CDS_NO_INFO = 0
-CDS_NO_DISC = 1
-CDS_TRAY_OPEN = 2
-CDS_DRIVE_NOT_READY = 3
-CDS_DISC_OK = 4
-CDS_AUDIO = 100
-CDS_DATA_1 = 101
-CDS_DATA_2 = 102
-CDS_XA_2_1 = 103
-CDS_XA_2_2 = 104
-CDS_MIXED = 105
-CDO_AUTO_CLOSE = 0x1
-CDO_AUTO_EJECT = 0x2
-CDO_USE_FFLAGS = 0x4
-CDO_LOCK = 0x8
-CDO_CHECK_TYPE = 0x10
-CD_PART_MAX = 64
-CD_PART_MASK = (CD_PART_MAX - 1)
-GPCMD_BLANK = 0xa1
-GPCMD_CLOSE_TRACK = 0x5b
-GPCMD_FLUSH_CACHE = 0x35
-GPCMD_FORMAT_UNIT = 0x04
-GPCMD_GET_CONFIGURATION = 0x46
-GPCMD_GET_EVENT_STATUS_NOTIFICATION = 0x4a
-GPCMD_GET_PERFORMANCE = 0xac
-GPCMD_INQUIRY = 0x12
-GPCMD_LOAD_UNLOAD = 0xa6
-GPCMD_MECHANISM_STATUS = 0xbd
-GPCMD_MODE_SELECT_10 = 0x55
-GPCMD_MODE_SENSE_10 = 0x5a
-GPCMD_PAUSE_RESUME = 0x4b
-GPCMD_PLAY_AUDIO_10 = 0x45
-GPCMD_PLAY_AUDIO_MSF = 0x47
-GPCMD_PLAY_AUDIO_TI = 0x48
-GPCMD_PLAY_CD = 0xbc
-GPCMD_PREVENT_ALLOW_MEDIUM_REMOVAL = 0x1e
-GPCMD_READ_10 = 0x28
-GPCMD_READ_12 = 0xa8
-GPCMD_READ_CDVD_CAPACITY = 0x25
-GPCMD_READ_CD = 0xbe
-GPCMD_READ_CD_MSF = 0xb9
-GPCMD_READ_DISC_INFO = 0x51
-GPCMD_READ_DVD_STRUCTURE = 0xad
-GPCMD_READ_FORMAT_CAPACITIES = 0x23
-GPCMD_READ_HEADER = 0x44
-GPCMD_READ_TRACK_RZONE_INFO = 0x52
-GPCMD_READ_SUBCHANNEL = 0x42
-GPCMD_READ_TOC_PMA_ATIP = 0x43
-GPCMD_REPAIR_RZONE_TRACK = 0x58
-GPCMD_REPORT_KEY = 0xa4
-GPCMD_REQUEST_SENSE = 0x03
-GPCMD_RESERVE_RZONE_TRACK = 0x53
-GPCMD_SCAN = 0xba
-GPCMD_SEEK = 0x2b
-GPCMD_SEND_DVD_STRUCTURE = 0xad
-GPCMD_SEND_EVENT = 0xa2
-GPCMD_SEND_KEY = 0xa3
-GPCMD_SEND_OPC = 0x54
-GPCMD_SET_READ_AHEAD = 0xa7
-GPCMD_SET_STREAMING = 0xb6
-GPCMD_START_STOP_UNIT = 0x1b
-GPCMD_STOP_PLAY_SCAN = 0x4e
-GPCMD_TEST_UNIT_READY = 0x00
-GPCMD_VERIFY_10 = 0x2f
-GPCMD_WRITE_10 = 0x2a
-GPCMD_WRITE_AND_VERIFY_10 = 0x2e
-GPCMD_SET_SPEED = 0xbb
-GPCMD_PLAYAUDIO_TI = 0x48
-GPCMD_GET_MEDIA_STATUS = 0xda
-GPMODE_R_W_ERROR_PAGE = 0x01
-GPMODE_WRITE_PARMS_PAGE = 0x05
-GPMODE_AUDIO_CTL_PAGE = 0x0e
-GPMODE_POWER_PAGE = 0x1a
-GPMODE_FAULT_FAIL_PAGE = 0x1c
-GPMODE_TO_PROTECT_PAGE = 0x1d
-GPMODE_CAPABILITIES_PAGE = 0x2a
-GPMODE_ALL_PAGES = 0x3f
-GPMODE_CDROM_PAGE = 0x0d
-DVD_STRUCT_PHYSICAL = 0x00
-DVD_STRUCT_COPYRIGHT = 0x01
-DVD_STRUCT_DISCKEY = 0x02
-DVD_STRUCT_BCA = 0x03
-DVD_STRUCT_MANUFACT = 0x04
-DVD_LAYERS = 4
-DVD_LU_SEND_AGID = 0
-DVD_HOST_SEND_CHALLENGE = 1
-DVD_LU_SEND_KEY1 = 2
-DVD_LU_SEND_CHALLENGE = 3
-DVD_HOST_SEND_KEY2 = 4
-DVD_AUTH_ESTABLISHED = 5
-DVD_AUTH_FAILURE = 6
-DVD_LU_SEND_TITLE_KEY = 7
-DVD_LU_SEND_ASF = 8
-DVD_INVALIDATE_AGID = 9
-DVD_LU_SEND_RPC_STATE = 10
-DVD_HOST_SEND_RPC_STATE = 11
-DVD_CPM_NO_COPYRIGHT = 0
-DVD_CPM_COPYRIGHTED = 1
-DVD_CP_SEC_NONE = 0
-DVD_CP_SEC_EXIST = 1
-DVD_CGMS_UNRESTRICTED = 0
-DVD_CGMS_SINGLE = 2
-DVD_CGMS_RESTRICTED = 3
-
-CDROM_MAX_SLOTS = 256
diff --git a/Lib/plat-linux3/DLFCN.py b/Lib/plat-linux3/DLFCN.py
deleted file mode 100644
--- a/Lib/plat-linux3/DLFCN.py
+++ /dev/null
@@ -1,83 +0,0 @@
-# Generated by h2py from /usr/include/dlfcn.h
-_DLFCN_H = 1
-
-# Included from features.h
-_FEATURES_H = 1
-__USE_ANSI = 1
-__FAVOR_BSD = 1
-_ISOC99_SOURCE = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 199506L
-_XOPEN_SOURCE = 600
-_XOPEN_SOURCE_EXTENDED = 1
-_LARGEFILE64_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-__USE_ISOC99 = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 2
-_POSIX_C_SOURCE = 199506L
-__USE_POSIX = 1
-__USE_POSIX2 = 1
-__USE_POSIX199309 = 1
-__USE_POSIX199506 = 1
-__USE_XOPEN = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_UNIX98 = 1
-_LARGEFILE_SOURCE = 1
-__USE_XOPEN2K = 1
-__USE_ISOC99 = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_FILE_OFFSET64 = 1
-__USE_MISC = 1
-__USE_BSD = 1
-__USE_SVID = 1
-__USE_GNU = 1
-__USE_REENTRANT = 1
-__STDC_IEC_559__ = 1
-__STDC_IEC_559_COMPLEX__ = 1
-__STDC_ISO_10646__ = 200009L
-__GNU_LIBRARY__ = 6
-__GLIBC__ = 2
-__GLIBC_MINOR__ = 2
-
-# Included from sys/cdefs.h
-_SYS_CDEFS_H = 1
-def __PMT(args): return args
-
-def __P(args): return args
-
-def __PMT(args): return args
-
-def __STRING(x): return #x
-
-__flexarr = []
-__flexarr = [0]
-__flexarr = []
-__flexarr = [1]
-def __ASMNAME(cname): return __ASMNAME2 (__USER_LABEL_PREFIX__, cname)
-
-def __attribute__(xyz): return
-
-def __attribute_format_arg__(x): return __attribute__ ((__format_arg__ (x)))
-
-def __attribute_format_arg__(x): return
-
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_EXTERN_INLINES = 1
-
-# Included from gnu/stubs.h
-
-# Included from bits/dlfcn.h
-RTLD_LAZY = 0x00001
-RTLD_NOW = 0x00002
-RTLD_BINDING_MASK = 0x3
-RTLD_NOLOAD = 0x00004
-RTLD_GLOBAL = 0x00100
-RTLD_LOCAL = 0
-RTLD_NODELETE = 0x01000
diff --git a/Lib/plat-linux3/IN.py b/Lib/plat-linux3/IN.py
deleted file mode 100644
--- a/Lib/plat-linux3/IN.py
+++ /dev/null
@@ -1,615 +0,0 @@
-# Generated by h2py from /usr/include/netinet/in.h
-_NETINET_IN_H = 1
-
-# Included from features.h
-_FEATURES_H = 1
-__USE_ANSI = 1
-__FAVOR_BSD = 1
-_ISOC99_SOURCE = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 199506L
-_XOPEN_SOURCE = 600
-_XOPEN_SOURCE_EXTENDED = 1
-_LARGEFILE64_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-__USE_ISOC99 = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 2
-_POSIX_C_SOURCE = 199506L
-__USE_POSIX = 1
-__USE_POSIX2 = 1
-__USE_POSIX199309 = 1
-__USE_POSIX199506 = 1
-__USE_XOPEN = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_UNIX98 = 1
-_LARGEFILE_SOURCE = 1
-__USE_XOPEN2K = 1
-__USE_ISOC99 = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_FILE_OFFSET64 = 1
-__USE_MISC = 1
-__USE_BSD = 1
-__USE_SVID = 1
-__USE_GNU = 1
-__USE_REENTRANT = 1
-__STDC_IEC_559__ = 1
-__STDC_IEC_559_COMPLEX__ = 1
-__STDC_ISO_10646__ = 200009L
-__GNU_LIBRARY__ = 6
-__GLIBC__ = 2
-__GLIBC_MINOR__ = 2
-
-# Included from sys/cdefs.h
-_SYS_CDEFS_H = 1
-def __PMT(args): return args
-
-def __P(args): return args
-
-def __PMT(args): return args
-
-def __STRING(x): return #x
-
-__flexarr = []
-__flexarr = [0]
-__flexarr = []
-__flexarr = [1]
-def __ASMNAME(cname): return __ASMNAME2 (__USER_LABEL_PREFIX__, cname)
-
-def __attribute__(xyz): return
-
-def __attribute_format_arg__(x): return __attribute__ ((__format_arg__ (x)))
-
-def __attribute_format_arg__(x): return
-
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_EXTERN_INLINES = 1
-
-# Included from gnu/stubs.h
-
-# Included from stdint.h
-_STDINT_H = 1
-
-# Included from bits/wchar.h
-_BITS_WCHAR_H = 1
-__WCHAR_MIN = (-2147483647l - 1l)
-__WCHAR_MAX = (2147483647l)
-
-# Included from bits/wordsize.h
-__WORDSIZE = 32
-def __INT64_C(c): return c ## L
-
-def __UINT64_C(c): return c ## UL
-
-def __INT64_C(c): return c ## LL
-
-def __UINT64_C(c): return c ## ULL
-
-INT8_MIN = (-128)
-INT16_MIN = (-32767-1)
-INT32_MIN = (-2147483647-1)
-INT64_MIN = (-__INT64_C(9223372036854775807)-1)
-INT8_MAX = (127)
-INT16_MAX = (32767)
-INT32_MAX = (2147483647)
-INT64_MAX = (__INT64_C(9223372036854775807))
-UINT8_MAX = (255)
-UINT16_MAX = (65535)
-UINT64_MAX = (__UINT64_C(18446744073709551615))
-INT_LEAST8_MIN = (-128)
-INT_LEAST16_MIN = (-32767-1)
-INT_LEAST32_MIN = (-2147483647-1)
-INT_LEAST64_MIN = (-__INT64_C(9223372036854775807)-1)
-INT_LEAST8_MAX = (127)
-INT_LEAST16_MAX = (32767)
-INT_LEAST32_MAX = (2147483647)
-INT_LEAST64_MAX = (__INT64_C(9223372036854775807))
-UINT_LEAST8_MAX = (255)
-UINT_LEAST16_MAX = (65535)
-UINT_LEAST64_MAX = (__UINT64_C(18446744073709551615))
-INT_FAST8_MIN = (-128)
-INT_FAST16_MIN = (-9223372036854775807L-1)
-INT_FAST32_MIN = (-9223372036854775807L-1)
-INT_FAST16_MIN = (-2147483647-1)
-INT_FAST32_MIN = (-2147483647-1)
-INT_FAST64_MIN = (-__INT64_C(9223372036854775807)-1)
-INT_FAST8_MAX = (127)
-INT_FAST16_MAX = (9223372036854775807L)
-INT_FAST32_MAX = (9223372036854775807L)
-INT_FAST16_MAX = (2147483647)
-INT_FAST32_MAX = (2147483647)
-INT_FAST64_MAX = (__INT64_C(9223372036854775807))
-UINT_FAST8_MAX = (255)
-UINT_FAST64_MAX = (__UINT64_C(18446744073709551615))
-INTPTR_MIN = (-9223372036854775807L-1)
-INTPTR_MAX = (9223372036854775807L)
-INTPTR_MIN = (-2147483647-1)
-INTPTR_MAX = (2147483647)
-INTMAX_MIN = (-__INT64_C(9223372036854775807)-1)
-INTMAX_MAX = (__INT64_C(9223372036854775807))
-UINTMAX_MAX = (__UINT64_C(18446744073709551615))
-PTRDIFF_MIN = (-9223372036854775807L-1)
-PTRDIFF_MAX = (9223372036854775807L)
-PTRDIFF_MIN = (-2147483647-1)
-PTRDIFF_MAX = (2147483647)
-SIG_ATOMIC_MIN = (-2147483647-1)
-SIG_ATOMIC_MAX = (2147483647)
-WCHAR_MIN = __WCHAR_MIN
-WCHAR_MAX = __WCHAR_MAX
-def INT8_C(c): return c
-
-def INT16_C(c): return c
-
-def INT32_C(c): return c
-
-def INT64_C(c): return c ## L
-
-def INT64_C(c): return c ## LL
-
-def UINT8_C(c): return c ## U
-
-def UINT16_C(c): return c ## U
-
-def UINT32_C(c): return c ## U
-
-def UINT64_C(c): return c ## UL
-
-def UINT64_C(c): return c ## ULL
-
-def INTMAX_C(c): return c ## L
-
-def UINTMAX_C(c): return c ## UL
-
-def INTMAX_C(c): return c ## LL
-
-def UINTMAX_C(c): return c ## ULL
-
-
-# Included from bits/types.h
-_BITS_TYPES_H = 1
-__FD_SETSIZE = 1024
-
-# Included from bits/pthreadtypes.h
-_BITS_PTHREADTYPES_H = 1
-
-# Included from bits/sched.h
-SCHED_OTHER = 0
-SCHED_FIFO = 1
-SCHED_RR = 2
-CSIGNAL = 0x000000ff
-CLONE_VM = 0x00000100
-CLONE_FS = 0x00000200
-CLONE_FILES = 0x00000400
-CLONE_SIGHAND = 0x00000800
-CLONE_PID = 0x00001000
-CLONE_PTRACE = 0x00002000
-CLONE_VFORK = 0x00004000
-__defined_schedparam = 1
-def IN_CLASSA(a): return ((((in_addr_t)(a)) & (-2147483648)) == 0)
-
-IN_CLASSA_NET = (-16777216)
-IN_CLASSA_NSHIFT = 24
-IN_CLASSA_HOST = ((-1) & ~IN_CLASSA_NET)
-IN_CLASSA_MAX = 128
-def IN_CLASSB(a): return ((((in_addr_t)(a)) & (-1073741824)) == (-2147483648))
-
-IN_CLASSB_NET = (-65536)
-IN_CLASSB_NSHIFT = 16
-IN_CLASSB_HOST = ((-1) & ~IN_CLASSB_NET)
-IN_CLASSB_MAX = 65536
-def IN_CLASSC(a): return ((((in_addr_t)(a)) & (-536870912)) == (-1073741824))
-
-IN_CLASSC_NET = (-256)
-IN_CLASSC_NSHIFT = 8
-IN_CLASSC_HOST = ((-1) & ~IN_CLASSC_NET)
-def IN_CLASSD(a): return ((((in_addr_t)(a)) & (-268435456)) == (-536870912))
-
-def IN_MULTICAST(a): return IN_CLASSD(a)
-
-def IN_EXPERIMENTAL(a): return ((((in_addr_t)(a)) & (-536870912)) == (-536870912))
-
-def IN_BADCLASS(a): return ((((in_addr_t)(a)) & (-268435456)) == (-268435456))
-
-IN_LOOPBACKNET = 127
-INET_ADDRSTRLEN = 16
-INET6_ADDRSTRLEN = 46
-
-# Included from bits/socket.h
-
-# Included from limits.h
-_LIBC_LIMITS_H_ = 1
-MB_LEN_MAX = 16
-_LIMITS_H = 1
-CHAR_BIT = 8
-SCHAR_MIN = (-128)
-SCHAR_MAX = 127
-UCHAR_MAX = 255
-CHAR_MIN = 0
-CHAR_MAX = UCHAR_MAX
-CHAR_MIN = SCHAR_MIN
-CHAR_MAX = SCHAR_MAX
-SHRT_MIN = (-32768)
-SHRT_MAX = 32767
-USHRT_MAX = 65535
-INT_MAX = 2147483647
-LONG_MAX = 9223372036854775807L
-LONG_MAX = 2147483647L
-LONG_MIN = (-LONG_MAX - 1L)
-
-# Included from bits/posix1_lim.h
-_BITS_POSIX1_LIM_H = 1
-_POSIX_AIO_LISTIO_MAX = 2
-_POSIX_AIO_MAX = 1
-_POSIX_ARG_MAX = 4096
-_POSIX_CHILD_MAX = 6
-_POSIX_DELAYTIMER_MAX = 32
-_POSIX_LINK_MAX = 8
-_POSIX_MAX_CANON = 255
-_POSIX_MAX_INPUT = 255
-_POSIX_MQ_OPEN_MAX = 8
-_POSIX_MQ_PRIO_MAX = 32
-_POSIX_NGROUPS_MAX = 0
-_POSIX_OPEN_MAX = 16
-_POSIX_FD_SETSIZE = _POSIX_OPEN_MAX
-_POSIX_NAME_MAX = 14
-_POSIX_PATH_MAX = 256
-_POSIX_PIPE_BUF = 512
-_POSIX_RTSIG_MAX = 8
-_POSIX_SEM_NSEMS_MAX = 256
-_POSIX_SEM_VALUE_MAX = 32767
-_POSIX_SIGQUEUE_MAX = 32
-_POSIX_SSIZE_MAX = 32767
-_POSIX_STREAM_MAX = 8
-_POSIX_TZNAME_MAX = 6
-_POSIX_QLIMIT = 1
-_POSIX_HIWAT = _POSIX_PIPE_BUF
-_POSIX_UIO_MAXIOV = 16
-_POSIX_TTY_NAME_MAX = 9
-_POSIX_TIMER_MAX = 32
-_POSIX_LOGIN_NAME_MAX = 9
-_POSIX_CLOCKRES_MIN = 20000000
-
-# Included from bits/local_lim.h
-
-# Included from linux/limits.h
-NR_OPEN = 1024
-NGROUPS_MAX = 32
-ARG_MAX = 131072
-CHILD_MAX = 999
-OPEN_MAX = 256
-LINK_MAX = 127
-MAX_CANON = 255
-MAX_INPUT = 255
-NAME_MAX = 255
-PATH_MAX = 4096
-PIPE_BUF = 4096
-RTSIG_MAX = 32
-_POSIX_THREAD_KEYS_MAX = 128
-PTHREAD_KEYS_MAX = 1024
-_POSIX_THREAD_DESTRUCTOR_ITERATIONS = 4
-PTHREAD_DESTRUCTOR_ITERATIONS = _POSIX_THREAD_DESTRUCTOR_ITERATIONS
-_POSIX_THREAD_THREADS_MAX = 64
-PTHREAD_THREADS_MAX = 1024
-AIO_PRIO_DELTA_MAX = 20
-PTHREAD_STACK_MIN = 16384
-TIMER_MAX = 256
-SSIZE_MAX = LONG_MAX
-NGROUPS_MAX = _POSIX_NGROUPS_MAX
-
-# Included from bits/posix2_lim.h
-_BITS_POSIX2_LIM_H = 1
-_POSIX2_BC_BASE_MAX = 99
-_POSIX2_BC_DIM_MAX = 2048
-_POSIX2_BC_SCALE_MAX = 99
-_POSIX2_BC_STRING_MAX = 1000
-_POSIX2_COLL_WEIGHTS_MAX = 2
-_POSIX2_EXPR_NEST_MAX = 32
-_POSIX2_LINE_MAX = 2048
-_POSIX2_RE_DUP_MAX = 255
-_POSIX2_CHARCLASS_NAME_MAX = 14
-BC_BASE_MAX = _POSIX2_BC_BASE_MAX
-BC_DIM_MAX = _POSIX2_BC_DIM_MAX
-BC_SCALE_MAX = _POSIX2_BC_SCALE_MAX
-BC_STRING_MAX = _POSIX2_BC_STRING_MAX
-COLL_WEIGHTS_MAX = 255
-EXPR_NEST_MAX = _POSIX2_EXPR_NEST_MAX
-LINE_MAX = _POSIX2_LINE_MAX
-CHARCLASS_NAME_MAX = 2048
-RE_DUP_MAX = (0x7fff)
-
-# Included from bits/xopen_lim.h
-_XOPEN_LIM_H = 1
-
-# Included from bits/stdio_lim.h
-L_tmpnam = 20
-TMP_MAX = 238328
-FILENAME_MAX = 4096
-L_ctermid = 9
-L_cuserid = 9
-FOPEN_MAX = 16
-IOV_MAX = 1024
-_XOPEN_IOV_MAX = _POSIX_UIO_MAXIOV
-NL_ARGMAX = _POSIX_ARG_MAX
-NL_LANGMAX = _POSIX2_LINE_MAX
-NL_MSGMAX = INT_MAX
-NL_NMAX = INT_MAX
-NL_SETMAX = INT_MAX
-NL_TEXTMAX = INT_MAX
-NZERO = 20
-WORD_BIT = 16
-WORD_BIT = 32
-WORD_BIT = 64
-WORD_BIT = 16
-WORD_BIT = 32
-WORD_BIT = 64
-WORD_BIT = 32
-LONG_BIT = 32
-LONG_BIT = 64
-LONG_BIT = 32
-LONG_BIT = 64
-LONG_BIT = 64
-LONG_BIT = 32
-from TYPES import *
-PF_UNSPEC = 0
-PF_LOCAL = 1
-PF_UNIX = PF_LOCAL
-PF_FILE = PF_LOCAL
-PF_INET = 2
-PF_AX25 = 3
-PF_IPX = 4
-PF_APPLETALK = 5
-PF_NETROM = 6
-PF_BRIDGE = 7
-PF_ATMPVC = 8
-PF_X25 = 9
-PF_INET6 = 10
-PF_ROSE = 11
-PF_DECnet = 12
-PF_NETBEUI = 13
-PF_SECURITY = 14
-PF_KEY = 15
-PF_NETLINK = 16
-PF_ROUTE = PF_NETLINK
-PF_PACKET = 17
-PF_ASH = 18
-PF_ECONET = 19
-PF_ATMSVC = 20
-PF_SNA = 22
-PF_IRDA = 23
-PF_PPPOX = 24
-PF_WANPIPE = 25
-PF_BLUETOOTH = 31
-PF_MAX = 32
-AF_UNSPEC = PF_UNSPEC
-AF_LOCAL = PF_LOCAL
-AF_UNIX = PF_UNIX
-AF_FILE = PF_FILE
-AF_INET = PF_INET
-AF_AX25 = PF_AX25
-AF_IPX = PF_IPX
-AF_APPLETALK = PF_APPLETALK
-AF_NETROM = PF_NETROM
-AF_BRIDGE = PF_BRIDGE
-AF_ATMPVC = PF_ATMPVC
-AF_X25 = PF_X25
-AF_INET6 = PF_INET6
-AF_ROSE = PF_ROSE
-AF_DECnet = PF_DECnet
-AF_NETBEUI = PF_NETBEUI
-AF_SECURITY = PF_SECURITY
-AF_KEY = PF_KEY
-AF_NETLINK = PF_NETLINK
-AF_ROUTE = PF_ROUTE
-AF_PACKET = PF_PACKET
-AF_ASH = PF_ASH
-AF_ECONET = PF_ECONET
-AF_ATMSVC = PF_ATMSVC
-AF_SNA = PF_SNA
-AF_IRDA = PF_IRDA
-AF_PPPOX = PF_PPPOX
-AF_WANPIPE = PF_WANPIPE
-AF_BLUETOOTH = PF_BLUETOOTH
-AF_MAX = PF_MAX
-SOL_RAW = 255
-SOL_DECNET = 261
-SOL_X25 = 262
-SOL_PACKET = 263
-SOL_ATM = 264
-SOL_AAL = 265
-SOL_IRDA = 266
-SOMAXCONN = 128
-
-# Included from bits/sockaddr.h
-_BITS_SOCKADDR_H = 1
-def __SOCKADDR_COMMON(sa_prefix): return \
-
-_SS_SIZE = 128
-def CMSG_FIRSTHDR(mhdr): return \
-
-
-# Included from asm/socket.h
-
-# Included from asm/sockios.h
-FIOSETOWN = 0x8901
-SIOCSPGRP = 0x8902
-FIOGETOWN = 0x8903
-SIOCGPGRP = 0x8904
-SIOCATMARK = 0x8905
-SIOCGSTAMP = 0x8906
-SOL_SOCKET = 1
-SO_DEBUG = 1
-SO_REUSEADDR = 2
-SO_TYPE = 3
-SO_ERROR = 4
-SO_DONTROUTE = 5
-SO_BROADCAST = 6
-SO_SNDBUF = 7
-SO_RCVBUF = 8
-SO_KEEPALIVE = 9
-SO_OOBINLINE = 10
-SO_NO_CHECK = 11
-SO_PRIORITY = 12
-SO_LINGER = 13
-SO_BSDCOMPAT = 14
-SO_PASSCRED = 16
-SO_PEERCRED = 17
-SO_RCVLOWAT = 18
-SO_SNDLOWAT = 19
-SO_RCVTIMEO = 20
-SO_SNDTIMEO = 21
-SO_SECURITY_AUTHENTICATION = 22
-SO_SECURITY_ENCRYPTION_TRANSPORT = 23
-SO_SECURITY_ENCRYPTION_NETWORK = 24
-SO_BINDTODEVICE = 25
-SO_ATTACH_FILTER = 26
-SO_DETACH_FILTER = 27
-SO_PEERNAME = 28
-SO_TIMESTAMP = 29
-SCM_TIMESTAMP = SO_TIMESTAMP
-SO_ACCEPTCONN = 30
-SOCK_STREAM = 1
-SOCK_DGRAM = 2
-SOCK_RAW = 3
-SOCK_RDM = 4
-SOCK_SEQPACKET = 5
-SOCK_PACKET = 10
-SOCK_MAX = (SOCK_PACKET+1)
-
-# Included from bits/in.h
-IP_TOS = 1
-IP_TTL = 2
-IP_HDRINCL = 3
-IP_OPTIONS = 4
-IP_ROUTER_ALERT = 5
-IP_RECVOPTS = 6
-IP_RETOPTS = 7
-IP_PKTINFO = 8
-IP_PKTOPTIONS = 9
-IP_PMTUDISC = 10
-IP_MTU_DISCOVER = 10
-IP_RECVERR = 11
-IP_RECVTTL = 12
-IP_RECVTOS = 13
-IP_MULTICAST_IF = 32
-IP_MULTICAST_TTL = 33
-IP_MULTICAST_LOOP = 34
-IP_ADD_MEMBERSHIP = 35
-IP_DROP_MEMBERSHIP = 36
-IP_RECVRETOPTS = IP_RETOPTS
-IP_PMTUDISC_DONT = 0
-IP_PMTUDISC_WANT = 1
-IP_PMTUDISC_DO = 2
-SOL_IP = 0
-IP_DEFAULT_MULTICAST_TTL = 1
-IP_DEFAULT_MULTICAST_LOOP = 1
-IP_MAX_MEMBERSHIPS = 20
-IPV6_ADDRFORM = 1
-IPV6_PKTINFO = 2
-IPV6_HOPOPTS = 3
-IPV6_DSTOPTS = 4
-IPV6_RTHDR = 5
-IPV6_PKTOPTIONS = 6
-IPV6_CHECKSUM = 7
-IPV6_HOPLIMIT = 8
-IPV6_NEXTHOP = 9
-IPV6_AUTHHDR = 10
-IPV6_UNICAST_HOPS = 16
-IPV6_MULTICAST_IF = 17
-IPV6_MULTICAST_HOPS = 18
-IPV6_MULTICAST_LOOP = 19
-IPV6_JOIN_GROUP = 20
-IPV6_LEAVE_GROUP = 21
-IPV6_ROUTER_ALERT = 22
-IPV6_MTU_DISCOVER = 23
-IPV6_MTU = 24
-IPV6_RECVERR = 25
-IPV6_RXHOPOPTS = IPV6_HOPOPTS
-IPV6_RXDSTOPTS = IPV6_DSTOPTS
-IPV6_ADD_MEMBERSHIP = IPV6_JOIN_GROUP
-IPV6_DROP_MEMBERSHIP = IPV6_LEAVE_GROUP
-IPV6_PMTUDISC_DONT = 0
-IPV6_PMTUDISC_WANT = 1
-IPV6_PMTUDISC_DO = 2
-SOL_IPV6 = 41
-SOL_ICMPV6 = 58
-IPV6_RTHDR_LOOSE = 0
-IPV6_RTHDR_STRICT = 1
-IPV6_RTHDR_TYPE_0 = 0
-
-# Included from endian.h
-_ENDIAN_H = 1
-__LITTLE_ENDIAN = 1234
-__BIG_ENDIAN = 4321
-__PDP_ENDIAN = 3412
-
-# Included from bits/endian.h
-__BYTE_ORDER = __LITTLE_ENDIAN
-__FLOAT_WORD_ORDER = __BYTE_ORDER
-LITTLE_ENDIAN = __LITTLE_ENDIAN
-BIG_ENDIAN = __BIG_ENDIAN
-PDP_ENDIAN = __PDP_ENDIAN
-BYTE_ORDER = __BYTE_ORDER
-
-# Included from bits/byteswap.h
-_BITS_BYTESWAP_H = 1
-def __bswap_constant_16(x): return \
-
-def __bswap_16(x): return \
-
-def __bswap_16(x): return __bswap_constant_16 (x)
-
-def __bswap_constant_32(x): return \
-
-def __bswap_32(x): return \
-
-def __bswap_32(x): return \
-
-def __bswap_32(x): return __bswap_constant_32 (x)
-
-def __bswap_constant_64(x): return \
-
-def __bswap_64(x): return \
-
-def ntohl(x): return (x)
-
-def ntohs(x): return (x)
-
-def htonl(x): return (x)
-
-def htons(x): return (x)
-
-def ntohl(x): return __bswap_32 (x)
-
-def ntohs(x): return __bswap_16 (x)
-
-def htonl(x): return __bswap_32 (x)
-
-def htons(x): return __bswap_16 (x)
-
-def IN6_IS_ADDR_UNSPECIFIED(a): return \
-
-def IN6_IS_ADDR_LOOPBACK(a): return \
-
-def IN6_IS_ADDR_LINKLOCAL(a): return \
-
-def IN6_IS_ADDR_SITELOCAL(a): return \
-
-def IN6_IS_ADDR_V4MAPPED(a): return \
-
-def IN6_IS_ADDR_V4COMPAT(a): return \
-
-def IN6_IS_ADDR_MC_NODELOCAL(a): return \
-
-def IN6_IS_ADDR_MC_LINKLOCAL(a): return \
-
-def IN6_IS_ADDR_MC_SITELOCAL(a): return \
-
-def IN6_IS_ADDR_MC_ORGLOCAL(a): return \
-
-def IN6_IS_ADDR_MC_GLOBAL(a): return
diff --git a/Lib/plat-linux3/TYPES.py b/Lib/plat-linux3/TYPES.py
deleted file mode 100644
--- a/Lib/plat-linux3/TYPES.py
+++ /dev/null
@@ -1,170 +0,0 @@
-# Generated by h2py from /usr/include/sys/types.h
-_SYS_TYPES_H = 1
-
-# Included from features.h
-_FEATURES_H = 1
-__USE_ANSI = 1
-__FAVOR_BSD = 1
-_ISOC99_SOURCE = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 199506L
-_XOPEN_SOURCE = 600
-_XOPEN_SOURCE_EXTENDED = 1
-_LARGEFILE64_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-__USE_ISOC99 = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 2
-_POSIX_C_SOURCE = 199506L
-__USE_POSIX = 1
-__USE_POSIX2 = 1
-__USE_POSIX199309 = 1
-__USE_POSIX199506 = 1
-__USE_XOPEN = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_UNIX98 = 1
-_LARGEFILE_SOURCE = 1
-__USE_XOPEN2K = 1
-__USE_ISOC99 = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_FILE_OFFSET64 = 1
-__USE_MISC = 1
-__USE_BSD = 1
-__USE_SVID = 1
-__USE_GNU = 1
-__USE_REENTRANT = 1
-__STDC_IEC_559__ = 1
-__STDC_IEC_559_COMPLEX__ = 1
-__STDC_ISO_10646__ = 200009L
-__GNU_LIBRARY__ = 6
-__GLIBC__ = 2
-__GLIBC_MINOR__ = 2
-
-# Included from sys/cdefs.h
-_SYS_CDEFS_H = 1
-def __PMT(args): return args
-
-def __P(args): return args
-
-def __PMT(args): return args
-
-def __STRING(x): return #x
-
-__flexarr = []
-__flexarr = [0]
-__flexarr = []
-__flexarr = [1]
-def __ASMNAME(cname): return __ASMNAME2 (__USER_LABEL_PREFIX__, cname)
-
-def __attribute__(xyz): return
-
-def __attribute_format_arg__(x): return __attribute__ ((__format_arg__ (x)))
-
-def __attribute_format_arg__(x): return
-
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_EXTERN_INLINES = 1
-
-# Included from gnu/stubs.h
-
-# Included from bits/types.h
-_BITS_TYPES_H = 1
-__FD_SETSIZE = 1024
-
-# Included from bits/pthreadtypes.h
-_BITS_PTHREADTYPES_H = 1
-
-# Included from bits/sched.h
-SCHED_OTHER = 0
-SCHED_FIFO = 1
-SCHED_RR = 2
-CSIGNAL = 0x000000ff
-CLONE_VM = 0x00000100
-CLONE_FS = 0x00000200
-CLONE_FILES = 0x00000400
-CLONE_SIGHAND = 0x00000800
-CLONE_PID = 0x00001000
-CLONE_PTRACE = 0x00002000
-CLONE_VFORK = 0x00004000
-__defined_schedparam = 1
-
-# Included from time.h
-_TIME_H = 1
-
-# Included from bits/time.h
-_BITS_TIME_H = 1
-CLOCKS_PER_SEC = 1000000l
-CLOCK_REALTIME = 0
-CLOCK_PROCESS_CPUTIME_ID = 2
-CLOCK_THREAD_CPUTIME_ID = 3
-TIMER_ABSTIME = 1
-_STRUCT_TIMEVAL = 1
-CLK_TCK = CLOCKS_PER_SEC
-__clock_t_defined = 1
-__time_t_defined = 1
-__clockid_t_defined = 1
-__timer_t_defined = 1
-__timespec_defined = 1
-def __isleap(year): return \
-
-__BIT_TYPES_DEFINED__ = 1
-
-# Included from endian.h
-_ENDIAN_H = 1
-__LITTLE_ENDIAN = 1234
-__BIG_ENDIAN = 4321
-__PDP_ENDIAN = 3412
-
-# Included from bits/endian.h
-__BYTE_ORDER = __LITTLE_ENDIAN
-__FLOAT_WORD_ORDER = __BYTE_ORDER
-LITTLE_ENDIAN = __LITTLE_ENDIAN
-BIG_ENDIAN = __BIG_ENDIAN
-PDP_ENDIAN = __PDP_ENDIAN
-BYTE_ORDER = __BYTE_ORDER
-
-# Included from sys/select.h
-_SYS_SELECT_H = 1
-
-# Included from bits/select.h
-def __FD_ZERO(fdsp): return \
-
-def __FD_ZERO(set): return \
-
-
-# Included from bits/sigset.h
-_SIGSET_H_types = 1
-_SIGSET_H_fns = 1
-def __sigmask(sig): return \
-
-def __sigemptyset(set): return \
-
-def __sigfillset(set): return \
-
-def __sigisemptyset(set): return \
-
-def __FDELT(d): return ((d) / __NFDBITS)
-
-FD_SETSIZE = __FD_SETSIZE
-def FD_ZERO(fdsetp): return __FD_ZERO (fdsetp)
-
-
-# Included from sys/sysmacros.h
-_SYS_SYSMACROS_H = 1
-def major(dev): return ((int)(((dev) >> 8) & 0xff))
-
-def minor(dev): return ((int)((dev) & 0xff))
-
-def major(dev): return (((dev).__val[1] >> 8) & 0xff)
-
-def minor(dev): return ((dev).__val[1] & 0xff)
-
-def major(dev): return (((dev).__val[0] >> 8) & 0xff)
-
-def minor(dev): return ((dev).__val[0] & 0xff)
diff --git a/Lib/plat-linux3/regen b/Lib/plat-linux3/regen
deleted file mode 100755
--- a/Lib/plat-linux3/regen
+++ /dev/null
@@ -1,8 +0,0 @@
-#! /bin/sh
-case `uname` in
-Linux*)	;;
-*)	echo Probably not on a Linux system 1>&2
-	exit 1;;
-esac
-set -v
-h2py -i '(u_long)' /usr/include/sys/types.h /usr/include/netinet/in.h /usr/include/dlfcn.h

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 21:40:12 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 05 Sep 2011 21:40:12 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyMzI2?=
 =?utf8?q?=3A_Remove_plat-linux3_directory?=
Message-ID: <E1R0f1g-0003EB-BE@dinsdale.python.org>

http://hg.python.org/cpython/rev/cb47cf5138a4
changeset:   72291:cb47cf5138a4
branch:      3.2
parent:      72287:c6d4d4d64405
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Sep 05 21:38:42 2011 +0200
summary:
  Issue #12326: Remove plat-linux3 directory

sys.platform is now always 'linux2' on Linux (even on Linux 3)

files:
  Lib/plat-linux3/CDROM.py |  207 ---------
  Lib/plat-linux3/DLFCN.py |   83 ---
  Lib/plat-linux3/IN.py    |  615 ---------------------------
  Lib/plat-linux3/TYPES.py |  170 -------
  Lib/plat-linux3/regen    |    8 -
  5 files changed, 0 insertions(+), 1083 deletions(-)


diff --git a/Lib/plat-linux3/CDROM.py b/Lib/plat-linux3/CDROM.py
deleted file mode 100644
--- a/Lib/plat-linux3/CDROM.py
+++ /dev/null
@@ -1,207 +0,0 @@
-# Generated by h2py from /usr/include/linux/cdrom.h
-
-CDROMPAUSE = 0x5301
-CDROMRESUME = 0x5302
-CDROMPLAYMSF = 0x5303
-CDROMPLAYTRKIND = 0x5304
-CDROMREADTOCHDR = 0x5305
-CDROMREADTOCENTRY = 0x5306
-CDROMSTOP = 0x5307
-CDROMSTART = 0x5308
-CDROMEJECT = 0x5309
-CDROMVOLCTRL = 0x530a
-CDROMSUBCHNL = 0x530b
-CDROMREADMODE2 = 0x530c
-CDROMREADMODE1 = 0x530d
-CDROMREADAUDIO = 0x530e
-CDROMEJECT_SW = 0x530f
-CDROMMULTISESSION = 0x5310
-CDROM_GET_MCN = 0x5311
-CDROM_GET_UPC = CDROM_GET_MCN
-CDROMRESET = 0x5312
-CDROMVOLREAD = 0x5313
-CDROMREADRAW = 0x5314
-CDROMREADCOOKED = 0x5315
-CDROMSEEK = 0x5316
-CDROMPLAYBLK = 0x5317
-CDROMREADALL = 0x5318
-CDROMGETSPINDOWN = 0x531d
-CDROMSETSPINDOWN = 0x531e
-CDROMCLOSETRAY = 0x5319
-CDROM_SET_OPTIONS = 0x5320
-CDROM_CLEAR_OPTIONS = 0x5321
-CDROM_SELECT_SPEED = 0x5322
-CDROM_SELECT_DISC = 0x5323
-CDROM_MEDIA_CHANGED = 0x5325
-CDROM_DRIVE_STATUS = 0x5326
-CDROM_DISC_STATUS = 0x5327
-CDROM_CHANGER_NSLOTS = 0x5328
-CDROM_LOCKDOOR = 0x5329
-CDROM_DEBUG = 0x5330
-CDROM_GET_CAPABILITY = 0x5331
-CDROMAUDIOBUFSIZ = 0x5382
-DVD_READ_STRUCT = 0x5390
-DVD_WRITE_STRUCT = 0x5391
-DVD_AUTH = 0x5392
-CDROM_SEND_PACKET = 0x5393
-CDROM_NEXT_WRITABLE = 0x5394
-CDROM_LAST_WRITTEN = 0x5395
-CDROM_PACKET_SIZE = 12
-CGC_DATA_UNKNOWN = 0
-CGC_DATA_WRITE = 1
-CGC_DATA_READ = 2
-CGC_DATA_NONE = 3
-CD_MINS = 74
-CD_SECS = 60
-CD_FRAMES = 75
-CD_SYNC_SIZE = 12
-CD_MSF_OFFSET = 150
-CD_CHUNK_SIZE = 24
-CD_NUM_OF_CHUNKS = 98
-CD_FRAMESIZE_SUB = 96
-CD_HEAD_SIZE = 4
-CD_SUBHEAD_SIZE = 8
-CD_EDC_SIZE = 4
-CD_ZERO_SIZE = 8
-CD_ECC_SIZE = 276
-CD_FRAMESIZE = 2048
-CD_FRAMESIZE_RAW = 2352
-CD_FRAMESIZE_RAWER = 2646
-CD_FRAMESIZE_RAW1 = (CD_FRAMESIZE_RAW-CD_SYNC_SIZE)
-CD_FRAMESIZE_RAW0 = (CD_FRAMESIZE_RAW-CD_SYNC_SIZE-CD_HEAD_SIZE)
-CD_XA_HEAD = (CD_HEAD_SIZE+CD_SUBHEAD_SIZE)
-CD_XA_TAIL = (CD_EDC_SIZE+CD_ECC_SIZE)
-CD_XA_SYNC_HEAD = (CD_SYNC_SIZE+CD_XA_HEAD)
-CDROM_LBA = 0x01
-CDROM_MSF = 0x02
-CDROM_DATA_TRACK = 0x04
-CDROM_LEADOUT = 0xAA
-CDROM_AUDIO_INVALID = 0x00
-CDROM_AUDIO_PLAY = 0x11
-CDROM_AUDIO_PAUSED = 0x12
-CDROM_AUDIO_COMPLETED = 0x13
-CDROM_AUDIO_ERROR = 0x14
-CDROM_AUDIO_NO_STATUS = 0x15
-CDC_CLOSE_TRAY = 0x1
-CDC_OPEN_TRAY = 0x2
-CDC_LOCK = 0x4
-CDC_SELECT_SPEED = 0x8
-CDC_SELECT_DISC = 0x10
-CDC_MULTI_SESSION = 0x20
-CDC_MCN = 0x40
-CDC_MEDIA_CHANGED = 0x80
-CDC_PLAY_AUDIO = 0x100
-CDC_RESET = 0x200
-CDC_IOCTLS = 0x400
-CDC_DRIVE_STATUS = 0x800
-CDC_GENERIC_PACKET = 0x1000
-CDC_CD_R = 0x2000
-CDC_CD_RW = 0x4000
-CDC_DVD = 0x8000
-CDC_DVD_R = 0x10000
-CDC_DVD_RAM = 0x20000
-CDS_NO_INFO = 0
-CDS_NO_DISC = 1
-CDS_TRAY_OPEN = 2
-CDS_DRIVE_NOT_READY = 3
-CDS_DISC_OK = 4
-CDS_AUDIO = 100
-CDS_DATA_1 = 101
-CDS_DATA_2 = 102
-CDS_XA_2_1 = 103
-CDS_XA_2_2 = 104
-CDS_MIXED = 105
-CDO_AUTO_CLOSE = 0x1
-CDO_AUTO_EJECT = 0x2
-CDO_USE_FFLAGS = 0x4
-CDO_LOCK = 0x8
-CDO_CHECK_TYPE = 0x10
-CD_PART_MAX = 64
-CD_PART_MASK = (CD_PART_MAX - 1)
-GPCMD_BLANK = 0xa1
-GPCMD_CLOSE_TRACK = 0x5b
-GPCMD_FLUSH_CACHE = 0x35
-GPCMD_FORMAT_UNIT = 0x04
-GPCMD_GET_CONFIGURATION = 0x46
-GPCMD_GET_EVENT_STATUS_NOTIFICATION = 0x4a
-GPCMD_GET_PERFORMANCE = 0xac
-GPCMD_INQUIRY = 0x12
-GPCMD_LOAD_UNLOAD = 0xa6
-GPCMD_MECHANISM_STATUS = 0xbd
-GPCMD_MODE_SELECT_10 = 0x55
-GPCMD_MODE_SENSE_10 = 0x5a
-GPCMD_PAUSE_RESUME = 0x4b
-GPCMD_PLAY_AUDIO_10 = 0x45
-GPCMD_PLAY_AUDIO_MSF = 0x47
-GPCMD_PLAY_AUDIO_TI = 0x48
-GPCMD_PLAY_CD = 0xbc
-GPCMD_PREVENT_ALLOW_MEDIUM_REMOVAL = 0x1e
-GPCMD_READ_10 = 0x28
-GPCMD_READ_12 = 0xa8
-GPCMD_READ_CDVD_CAPACITY = 0x25
-GPCMD_READ_CD = 0xbe
-GPCMD_READ_CD_MSF = 0xb9
-GPCMD_READ_DISC_INFO = 0x51
-GPCMD_READ_DVD_STRUCTURE = 0xad
-GPCMD_READ_FORMAT_CAPACITIES = 0x23
-GPCMD_READ_HEADER = 0x44
-GPCMD_READ_TRACK_RZONE_INFO = 0x52
-GPCMD_READ_SUBCHANNEL = 0x42
-GPCMD_READ_TOC_PMA_ATIP = 0x43
-GPCMD_REPAIR_RZONE_TRACK = 0x58
-GPCMD_REPORT_KEY = 0xa4
-GPCMD_REQUEST_SENSE = 0x03
-GPCMD_RESERVE_RZONE_TRACK = 0x53
-GPCMD_SCAN = 0xba
-GPCMD_SEEK = 0x2b
-GPCMD_SEND_DVD_STRUCTURE = 0xad
-GPCMD_SEND_EVENT = 0xa2
-GPCMD_SEND_KEY = 0xa3
-GPCMD_SEND_OPC = 0x54
-GPCMD_SET_READ_AHEAD = 0xa7
-GPCMD_SET_STREAMING = 0xb6
-GPCMD_START_STOP_UNIT = 0x1b
-GPCMD_STOP_PLAY_SCAN = 0x4e
-GPCMD_TEST_UNIT_READY = 0x00
-GPCMD_VERIFY_10 = 0x2f
-GPCMD_WRITE_10 = 0x2a
-GPCMD_WRITE_AND_VERIFY_10 = 0x2e
-GPCMD_SET_SPEED = 0xbb
-GPCMD_PLAYAUDIO_TI = 0x48
-GPCMD_GET_MEDIA_STATUS = 0xda
-GPMODE_R_W_ERROR_PAGE = 0x01
-GPMODE_WRITE_PARMS_PAGE = 0x05
-GPMODE_AUDIO_CTL_PAGE = 0x0e
-GPMODE_POWER_PAGE = 0x1a
-GPMODE_FAULT_FAIL_PAGE = 0x1c
-GPMODE_TO_PROTECT_PAGE = 0x1d
-GPMODE_CAPABILITIES_PAGE = 0x2a
-GPMODE_ALL_PAGES = 0x3f
-GPMODE_CDROM_PAGE = 0x0d
-DVD_STRUCT_PHYSICAL = 0x00
-DVD_STRUCT_COPYRIGHT = 0x01
-DVD_STRUCT_DISCKEY = 0x02
-DVD_STRUCT_BCA = 0x03
-DVD_STRUCT_MANUFACT = 0x04
-DVD_LAYERS = 4
-DVD_LU_SEND_AGID = 0
-DVD_HOST_SEND_CHALLENGE = 1
-DVD_LU_SEND_KEY1 = 2
-DVD_LU_SEND_CHALLENGE = 3
-DVD_HOST_SEND_KEY2 = 4
-DVD_AUTH_ESTABLISHED = 5
-DVD_AUTH_FAILURE = 6
-DVD_LU_SEND_TITLE_KEY = 7
-DVD_LU_SEND_ASF = 8
-DVD_INVALIDATE_AGID = 9
-DVD_LU_SEND_RPC_STATE = 10
-DVD_HOST_SEND_RPC_STATE = 11
-DVD_CPM_NO_COPYRIGHT = 0
-DVD_CPM_COPYRIGHTED = 1
-DVD_CP_SEC_NONE = 0
-DVD_CP_SEC_EXIST = 1
-DVD_CGMS_UNRESTRICTED = 0
-DVD_CGMS_SINGLE = 2
-DVD_CGMS_RESTRICTED = 3
-
-CDROM_MAX_SLOTS = 256
diff --git a/Lib/plat-linux3/DLFCN.py b/Lib/plat-linux3/DLFCN.py
deleted file mode 100644
--- a/Lib/plat-linux3/DLFCN.py
+++ /dev/null
@@ -1,83 +0,0 @@
-# Generated by h2py from /usr/include/dlfcn.h
-_DLFCN_H = 1
-
-# Included from features.h
-_FEATURES_H = 1
-__USE_ANSI = 1
-__FAVOR_BSD = 1
-_ISOC99_SOURCE = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 199506
-_XOPEN_SOURCE = 600
-_XOPEN_SOURCE_EXTENDED = 1
-_LARGEFILE64_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-__USE_ISOC99 = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 2
-_POSIX_C_SOURCE = 199506
-__USE_POSIX = 1
-__USE_POSIX2 = 1
-__USE_POSIX199309 = 1
-__USE_POSIX199506 = 1
-__USE_XOPEN = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_UNIX98 = 1
-_LARGEFILE_SOURCE = 1
-__USE_XOPEN2K = 1
-__USE_ISOC99 = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_FILE_OFFSET64 = 1
-__USE_MISC = 1
-__USE_BSD = 1
-__USE_SVID = 1
-__USE_GNU = 1
-__USE_REENTRANT = 1
-__STDC_IEC_559__ = 1
-__STDC_IEC_559_COMPLEX__ = 1
-__STDC_ISO_10646__ = 200009
-__GNU_LIBRARY__ = 6
-__GLIBC__ = 2
-__GLIBC_MINOR__ = 2
-
-# Included from sys/cdefs.h
-_SYS_CDEFS_H = 1
-def __PMT(args): return args
-
-def __P(args): return args
-
-def __PMT(args): return args
-
-def __STRING(x): return #x
-
-__flexarr = []
-__flexarr = [0]
-__flexarr = []
-__flexarr = [1]
-def __ASMNAME(cname): return __ASMNAME2 (__USER_LABEL_PREFIX__, cname)
-
-def __attribute__(xyz): return
-
-def __attribute_format_arg__(x): return __attribute__ ((__format_arg__ (x)))
-
-def __attribute_format_arg__(x): return
-
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_EXTERN_INLINES = 1
-
-# Included from gnu/stubs.h
-
-# Included from bits/dlfcn.h
-RTLD_LAZY = 0x00001
-RTLD_NOW = 0x00002
-RTLD_BINDING_MASK = 0x3
-RTLD_NOLOAD = 0x00004
-RTLD_GLOBAL = 0x00100
-RTLD_LOCAL = 0
-RTLD_NODELETE = 0x01000
diff --git a/Lib/plat-linux3/IN.py b/Lib/plat-linux3/IN.py
deleted file mode 100644
--- a/Lib/plat-linux3/IN.py
+++ /dev/null
@@ -1,615 +0,0 @@
-# Generated by h2py from /usr/include/netinet/in.h
-_NETINET_IN_H = 1
-
-# Included from features.h
-_FEATURES_H = 1
-__USE_ANSI = 1
-__FAVOR_BSD = 1
-_ISOC99_SOURCE = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 199506
-_XOPEN_SOURCE = 600
-_XOPEN_SOURCE_EXTENDED = 1
-_LARGEFILE64_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-__USE_ISOC99 = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 2
-_POSIX_C_SOURCE = 199506
-__USE_POSIX = 1
-__USE_POSIX2 = 1
-__USE_POSIX199309 = 1
-__USE_POSIX199506 = 1
-__USE_XOPEN = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_UNIX98 = 1
-_LARGEFILE_SOURCE = 1
-__USE_XOPEN2K = 1
-__USE_ISOC99 = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_FILE_OFFSET64 = 1
-__USE_MISC = 1
-__USE_BSD = 1
-__USE_SVID = 1
-__USE_GNU = 1
-__USE_REENTRANT = 1
-__STDC_IEC_559__ = 1
-__STDC_IEC_559_COMPLEX__ = 1
-__STDC_ISO_10646__ = 200009
-__GNU_LIBRARY__ = 6
-__GLIBC__ = 2
-__GLIBC_MINOR__ = 2
-
-# Included from sys/cdefs.h
-_SYS_CDEFS_H = 1
-def __PMT(args): return args
-
-def __P(args): return args
-
-def __PMT(args): return args
-
-def __STRING(x): return #x
-
-__flexarr = []
-__flexarr = [0]
-__flexarr = []
-__flexarr = [1]
-def __ASMNAME(cname): return __ASMNAME2 (__USER_LABEL_PREFIX__, cname)
-
-def __attribute__(xyz): return
-
-def __attribute_format_arg__(x): return __attribute__ ((__format_arg__ (x)))
-
-def __attribute_format_arg__(x): return
-
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_EXTERN_INLINES = 1
-
-# Included from gnu/stubs.h
-
-# Included from stdint.h
-_STDINT_H = 1
-
-# Included from bits/wchar.h
-_BITS_WCHAR_H = 1
-__WCHAR_MIN = (-2147483647 - 1)
-__WCHAR_MAX = (2147483647)
-
-# Included from bits/wordsize.h
-__WORDSIZE = 32
-def __INT64_C(c): return c ## L
-
-def __UINT64_C(c): return c ## UL
-
-def __INT64_C(c): return c ## LL
-
-def __UINT64_C(c): return c ## ULL
-
-INT8_MIN = (-128)
-INT16_MIN = (-32767-1)
-INT32_MIN = (-2147483647-1)
-INT64_MIN = (-__INT64_C(9223372036854775807)-1)
-INT8_MAX = (127)
-INT16_MAX = (32767)
-INT32_MAX = (2147483647)
-INT64_MAX = (__INT64_C(9223372036854775807))
-UINT8_MAX = (255)
-UINT16_MAX = (65535)
-UINT64_MAX = (__UINT64_C(18446744073709551615))
-INT_LEAST8_MIN = (-128)
-INT_LEAST16_MIN = (-32767-1)
-INT_LEAST32_MIN = (-2147483647-1)
-INT_LEAST64_MIN = (-__INT64_C(9223372036854775807)-1)
-INT_LEAST8_MAX = (127)
-INT_LEAST16_MAX = (32767)
-INT_LEAST32_MAX = (2147483647)
-INT_LEAST64_MAX = (__INT64_C(9223372036854775807))
-UINT_LEAST8_MAX = (255)
-UINT_LEAST16_MAX = (65535)
-UINT_LEAST64_MAX = (__UINT64_C(18446744073709551615))
-INT_FAST8_MIN = (-128)
-INT_FAST16_MIN = (-9223372036854775807-1)
-INT_FAST32_MIN = (-9223372036854775807-1)
-INT_FAST16_MIN = (-2147483647-1)
-INT_FAST32_MIN = (-2147483647-1)
-INT_FAST64_MIN = (-__INT64_C(9223372036854775807)-1)
-INT_FAST8_MAX = (127)
-INT_FAST16_MAX = (9223372036854775807)
-INT_FAST32_MAX = (9223372036854775807)
-INT_FAST16_MAX = (2147483647)
-INT_FAST32_MAX = (2147483647)
-INT_FAST64_MAX = (__INT64_C(9223372036854775807))
-UINT_FAST8_MAX = (255)
-UINT_FAST64_MAX = (__UINT64_C(18446744073709551615))
-INTPTR_MIN = (-9223372036854775807-1)
-INTPTR_MAX = (9223372036854775807)
-INTPTR_MIN = (-2147483647-1)
-INTPTR_MAX = (2147483647)
-INTMAX_MIN = (-__INT64_C(9223372036854775807)-1)
-INTMAX_MAX = (__INT64_C(9223372036854775807))
-UINTMAX_MAX = (__UINT64_C(18446744073709551615))
-PTRDIFF_MIN = (-9223372036854775807-1)
-PTRDIFF_MAX = (9223372036854775807)
-PTRDIFF_MIN = (-2147483647-1)
-PTRDIFF_MAX = (2147483647)
-SIG_ATOMIC_MIN = (-2147483647-1)
-SIG_ATOMIC_MAX = (2147483647)
-WCHAR_MIN = __WCHAR_MIN
-WCHAR_MAX = __WCHAR_MAX
-def INT8_C(c): return c
-
-def INT16_C(c): return c
-
-def INT32_C(c): return c
-
-def INT64_C(c): return c ## L
-
-def INT64_C(c): return c ## LL
-
-def UINT8_C(c): return c ## U
-
-def UINT16_C(c): return c ## U
-
-def UINT32_C(c): return c ## U
-
-def UINT64_C(c): return c ## UL
-
-def UINT64_C(c): return c ## ULL
-
-def INTMAX_C(c): return c ## L
-
-def UINTMAX_C(c): return c ## UL
-
-def INTMAX_C(c): return c ## LL
-
-def UINTMAX_C(c): return c ## ULL
-
-
-# Included from bits/types.h
-_BITS_TYPES_H = 1
-__FD_SETSIZE = 1024
-
-# Included from bits/pthreadtypes.h
-_BITS_PTHREADTYPES_H = 1
-
-# Included from bits/sched.h
-SCHED_OTHER = 0
-SCHED_FIFO = 1
-SCHED_RR = 2
-CSIGNAL = 0x000000ff
-CLONE_VM = 0x00000100
-CLONE_FS = 0x00000200
-CLONE_FILES = 0x00000400
-CLONE_SIGHAND = 0x00000800
-CLONE_PID = 0x00001000
-CLONE_PTRACE = 0x00002000
-CLONE_VFORK = 0x00004000
-__defined_schedparam = 1
-def IN_CLASSA(a): return ((((in_addr_t)(a)) & (-2147483648)) == 0)
-
-IN_CLASSA_NET = (-16777216)
-IN_CLASSA_NSHIFT = 24
-IN_CLASSA_HOST = ((-1) & ~IN_CLASSA_NET)
-IN_CLASSA_MAX = 128
-def IN_CLASSB(a): return ((((in_addr_t)(a)) & (-1073741824)) == (-2147483648))
-
-IN_CLASSB_NET = (-65536)
-IN_CLASSB_NSHIFT = 16
-IN_CLASSB_HOST = ((-1) & ~IN_CLASSB_NET)
-IN_CLASSB_MAX = 65536
-def IN_CLASSC(a): return ((((in_addr_t)(a)) & (-536870912)) == (-1073741824))
-
-IN_CLASSC_NET = (-256)
-IN_CLASSC_NSHIFT = 8
-IN_CLASSC_HOST = ((-1) & ~IN_CLASSC_NET)
-def IN_CLASSD(a): return ((((in_addr_t)(a)) & (-268435456)) == (-536870912))
-
-def IN_MULTICAST(a): return IN_CLASSD(a)
-
-def IN_EXPERIMENTAL(a): return ((((in_addr_t)(a)) & (-536870912)) == (-536870912))
-
-def IN_BADCLASS(a): return ((((in_addr_t)(a)) & (-268435456)) == (-268435456))
-
-IN_LOOPBACKNET = 127
-INET_ADDRSTRLEN = 16
-INET6_ADDRSTRLEN = 46
-
-# Included from bits/socket.h
-
-# Included from limits.h
-_LIBC_LIMITS_H_ = 1
-MB_LEN_MAX = 16
-_LIMITS_H = 1
-CHAR_BIT = 8
-SCHAR_MIN = (-128)
-SCHAR_MAX = 127
-UCHAR_MAX = 255
-CHAR_MIN = 0
-CHAR_MAX = UCHAR_MAX
-CHAR_MIN = SCHAR_MIN
-CHAR_MAX = SCHAR_MAX
-SHRT_MIN = (-32768)
-SHRT_MAX = 32767
-USHRT_MAX = 65535
-INT_MAX = 2147483647
-LONG_MAX = 9223372036854775807
-LONG_MAX = 2147483647
-LONG_MIN = (-LONG_MAX - 1)
-
-# Included from bits/posix1_lim.h
-_BITS_POSIX1_LIM_H = 1
-_POSIX_AIO_LISTIO_MAX = 2
-_POSIX_AIO_MAX = 1
-_POSIX_ARG_MAX = 4096
-_POSIX_CHILD_MAX = 6
-_POSIX_DELAYTIMER_MAX = 32
-_POSIX_LINK_MAX = 8
-_POSIX_MAX_CANON = 255
-_POSIX_MAX_INPUT = 255
-_POSIX_MQ_OPEN_MAX = 8
-_POSIX_MQ_PRIO_MAX = 32
-_POSIX_NGROUPS_MAX = 0
-_POSIX_OPEN_MAX = 16
-_POSIX_FD_SETSIZE = _POSIX_OPEN_MAX
-_POSIX_NAME_MAX = 14
-_POSIX_PATH_MAX = 256
-_POSIX_PIPE_BUF = 512
-_POSIX_RTSIG_MAX = 8
-_POSIX_SEM_NSEMS_MAX = 256
-_POSIX_SEM_VALUE_MAX = 32767
-_POSIX_SIGQUEUE_MAX = 32
-_POSIX_SSIZE_MAX = 32767
-_POSIX_STREAM_MAX = 8
-_POSIX_TZNAME_MAX = 6
-_POSIX_QLIMIT = 1
-_POSIX_HIWAT = _POSIX_PIPE_BUF
-_POSIX_UIO_MAXIOV = 16
-_POSIX_TTY_NAME_MAX = 9
-_POSIX_TIMER_MAX = 32
-_POSIX_LOGIN_NAME_MAX = 9
-_POSIX_CLOCKRES_MIN = 20000000
-
-# Included from bits/local_lim.h
-
-# Included from linux/limits.h
-NR_OPEN = 1024
-NGROUPS_MAX = 32
-ARG_MAX = 131072
-CHILD_MAX = 999
-OPEN_MAX = 256
-LINK_MAX = 127
-MAX_CANON = 255
-MAX_INPUT = 255
-NAME_MAX = 255
-PATH_MAX = 4096
-PIPE_BUF = 4096
-RTSIG_MAX = 32
-_POSIX_THREAD_KEYS_MAX = 128
-PTHREAD_KEYS_MAX = 1024
-_POSIX_THREAD_DESTRUCTOR_ITERATIONS = 4
-PTHREAD_DESTRUCTOR_ITERATIONS = _POSIX_THREAD_DESTRUCTOR_ITERATIONS
-_POSIX_THREAD_THREADS_MAX = 64
-PTHREAD_THREADS_MAX = 1024
-AIO_PRIO_DELTA_MAX = 20
-PTHREAD_STACK_MIN = 16384
-TIMER_MAX = 256
-SSIZE_MAX = LONG_MAX
-NGROUPS_MAX = _POSIX_NGROUPS_MAX
-
-# Included from bits/posix2_lim.h
-_BITS_POSIX2_LIM_H = 1
-_POSIX2_BC_BASE_MAX = 99
-_POSIX2_BC_DIM_MAX = 2048
-_POSIX2_BC_SCALE_MAX = 99
-_POSIX2_BC_STRING_MAX = 1000
-_POSIX2_COLL_WEIGHTS_MAX = 2
-_POSIX2_EXPR_NEST_MAX = 32
-_POSIX2_LINE_MAX = 2048
-_POSIX2_RE_DUP_MAX = 255
-_POSIX2_CHARCLASS_NAME_MAX = 14
-BC_BASE_MAX = _POSIX2_BC_BASE_MAX
-BC_DIM_MAX = _POSIX2_BC_DIM_MAX
-BC_SCALE_MAX = _POSIX2_BC_SCALE_MAX
-BC_STRING_MAX = _POSIX2_BC_STRING_MAX
-COLL_WEIGHTS_MAX = 255
-EXPR_NEST_MAX = _POSIX2_EXPR_NEST_MAX
-LINE_MAX = _POSIX2_LINE_MAX
-CHARCLASS_NAME_MAX = 2048
-RE_DUP_MAX = (0x7fff)
-
-# Included from bits/xopen_lim.h
-_XOPEN_LIM_H = 1
-
-# Included from bits/stdio_lim.h
-L_tmpnam = 20
-TMP_MAX = 238328
-FILENAME_MAX = 4096
-L_ctermid = 9
-L_cuserid = 9
-FOPEN_MAX = 16
-IOV_MAX = 1024
-_XOPEN_IOV_MAX = _POSIX_UIO_MAXIOV
-NL_ARGMAX = _POSIX_ARG_MAX
-NL_LANGMAX = _POSIX2_LINE_MAX
-NL_MSGMAX = INT_MAX
-NL_NMAX = INT_MAX
-NL_SETMAX = INT_MAX
-NL_TEXTMAX = INT_MAX
-NZERO = 20
-WORD_BIT = 16
-WORD_BIT = 32
-WORD_BIT = 64
-WORD_BIT = 16
-WORD_BIT = 32
-WORD_BIT = 64
-WORD_BIT = 32
-LONG_BIT = 32
-LONG_BIT = 64
-LONG_BIT = 32
-LONG_BIT = 64
-LONG_BIT = 64
-LONG_BIT = 32
-from TYPES import *
-PF_UNSPEC = 0
-PF_LOCAL = 1
-PF_UNIX = PF_LOCAL
-PF_FILE = PF_LOCAL
-PF_INET = 2
-PF_AX25 = 3
-PF_IPX = 4
-PF_APPLETALK = 5
-PF_NETROM = 6
-PF_BRIDGE = 7
-PF_ATMPVC = 8
-PF_X25 = 9
-PF_INET6 = 10
-PF_ROSE = 11
-PF_DECnet = 12
-PF_NETBEUI = 13
-PF_SECURITY = 14
-PF_KEY = 15
-PF_NETLINK = 16
-PF_ROUTE = PF_NETLINK
-PF_PACKET = 17
-PF_ASH = 18
-PF_ECONET = 19
-PF_ATMSVC = 20
-PF_SNA = 22
-PF_IRDA = 23
-PF_PPPOX = 24
-PF_WANPIPE = 25
-PF_BLUETOOTH = 31
-PF_MAX = 32
-AF_UNSPEC = PF_UNSPEC
-AF_LOCAL = PF_LOCAL
-AF_UNIX = PF_UNIX
-AF_FILE = PF_FILE
-AF_INET = PF_INET
-AF_AX25 = PF_AX25
-AF_IPX = PF_IPX
-AF_APPLETALK = PF_APPLETALK
-AF_NETROM = PF_NETROM
-AF_BRIDGE = PF_BRIDGE
-AF_ATMPVC = PF_ATMPVC
-AF_X25 = PF_X25
-AF_INET6 = PF_INET6
-AF_ROSE = PF_ROSE
-AF_DECnet = PF_DECnet
-AF_NETBEUI = PF_NETBEUI
-AF_SECURITY = PF_SECURITY
-AF_KEY = PF_KEY
-AF_NETLINK = PF_NETLINK
-AF_ROUTE = PF_ROUTE
-AF_PACKET = PF_PACKET
-AF_ASH = PF_ASH
-AF_ECONET = PF_ECONET
-AF_ATMSVC = PF_ATMSVC
-AF_SNA = PF_SNA
-AF_IRDA = PF_IRDA
-AF_PPPOX = PF_PPPOX
-AF_WANPIPE = PF_WANPIPE
-AF_BLUETOOTH = PF_BLUETOOTH
-AF_MAX = PF_MAX
-SOL_RAW = 255
-SOL_DECNET = 261
-SOL_X25 = 262
-SOL_PACKET = 263
-SOL_ATM = 264
-SOL_AAL = 265
-SOL_IRDA = 266
-SOMAXCONN = 128
-
-# Included from bits/sockaddr.h
-_BITS_SOCKADDR_H = 1
-def __SOCKADDR_COMMON(sa_prefix): return \
-
-_SS_SIZE = 128
-def CMSG_FIRSTHDR(mhdr): return \
-
-
-# Included from asm/socket.h
-
-# Included from asm/sockios.h
-FIOSETOWN = 0x8901
-SIOCSPGRP = 0x8902
-FIOGETOWN = 0x8903
-SIOCGPGRP = 0x8904
-SIOCATMARK = 0x8905
-SIOCGSTAMP = 0x8906
-SOL_SOCKET = 1
-SO_DEBUG = 1
-SO_REUSEADDR = 2
-SO_TYPE = 3
-SO_ERROR = 4
-SO_DONTROUTE = 5
-SO_BROADCAST = 6
-SO_SNDBUF = 7
-SO_RCVBUF = 8
-SO_KEEPALIVE = 9
-SO_OOBINLINE = 10
-SO_NO_CHECK = 11
-SO_PRIORITY = 12
-SO_LINGER = 13
-SO_BSDCOMPAT = 14
-SO_PASSCRED = 16
-SO_PEERCRED = 17
-SO_RCVLOWAT = 18
-SO_SNDLOWAT = 19
-SO_RCVTIMEO = 20
-SO_SNDTIMEO = 21
-SO_SECURITY_AUTHENTICATION = 22
-SO_SECURITY_ENCRYPTION_TRANSPORT = 23
-SO_SECURITY_ENCRYPTION_NETWORK = 24
-SO_BINDTODEVICE = 25
-SO_ATTACH_FILTER = 26
-SO_DETACH_FILTER = 27
-SO_PEERNAME = 28
-SO_TIMESTAMP = 29
-SCM_TIMESTAMP = SO_TIMESTAMP
-SO_ACCEPTCONN = 30
-SOCK_STREAM = 1
-SOCK_DGRAM = 2
-SOCK_RAW = 3
-SOCK_RDM = 4
-SOCK_SEQPACKET = 5
-SOCK_PACKET = 10
-SOCK_MAX = (SOCK_PACKET+1)
-
-# Included from bits/in.h
-IP_TOS = 1
-IP_TTL = 2
-IP_HDRINCL = 3
-IP_OPTIONS = 4
-IP_ROUTER_ALERT = 5
-IP_RECVOPTS = 6
-IP_RETOPTS = 7
-IP_PKTINFO = 8
-IP_PKTOPTIONS = 9
-IP_PMTUDISC = 10
-IP_MTU_DISCOVER = 10
-IP_RECVERR = 11
-IP_RECVTTL = 12
-IP_RECVTOS = 13
-IP_MULTICAST_IF = 32
-IP_MULTICAST_TTL = 33
-IP_MULTICAST_LOOP = 34
-IP_ADD_MEMBERSHIP = 35
-IP_DROP_MEMBERSHIP = 36
-IP_RECVRETOPTS = IP_RETOPTS
-IP_PMTUDISC_DONT = 0
-IP_PMTUDISC_WANT = 1
-IP_PMTUDISC_DO = 2
-SOL_IP = 0
-IP_DEFAULT_MULTICAST_TTL = 1
-IP_DEFAULT_MULTICAST_LOOP = 1
-IP_MAX_MEMBERSHIPS = 20
-IPV6_ADDRFORM = 1
-IPV6_PKTINFO = 2
-IPV6_HOPOPTS = 3
-IPV6_DSTOPTS = 4
-IPV6_RTHDR = 5
-IPV6_PKTOPTIONS = 6
-IPV6_CHECKSUM = 7
-IPV6_HOPLIMIT = 8
-IPV6_NEXTHOP = 9
-IPV6_AUTHHDR = 10
-IPV6_UNICAST_HOPS = 16
-IPV6_MULTICAST_IF = 17
-IPV6_MULTICAST_HOPS = 18
-IPV6_MULTICAST_LOOP = 19
-IPV6_JOIN_GROUP = 20
-IPV6_LEAVE_GROUP = 21
-IPV6_ROUTER_ALERT = 22
-IPV6_MTU_DISCOVER = 23
-IPV6_MTU = 24
-IPV6_RECVERR = 25
-IPV6_RXHOPOPTS = IPV6_HOPOPTS
-IPV6_RXDSTOPTS = IPV6_DSTOPTS
-IPV6_ADD_MEMBERSHIP = IPV6_JOIN_GROUP
-IPV6_DROP_MEMBERSHIP = IPV6_LEAVE_GROUP
-IPV6_PMTUDISC_DONT = 0
-IPV6_PMTUDISC_WANT = 1
-IPV6_PMTUDISC_DO = 2
-SOL_IPV6 = 41
-SOL_ICMPV6 = 58
-IPV6_RTHDR_LOOSE = 0
-IPV6_RTHDR_STRICT = 1
-IPV6_RTHDR_TYPE_0 = 0
-
-# Included from endian.h
-_ENDIAN_H = 1
-__LITTLE_ENDIAN = 1234
-__BIG_ENDIAN = 4321
-__PDP_ENDIAN = 3412
-
-# Included from bits/endian.h
-__BYTE_ORDER = __LITTLE_ENDIAN
-__FLOAT_WORD_ORDER = __BYTE_ORDER
-LITTLE_ENDIAN = __LITTLE_ENDIAN
-BIG_ENDIAN = __BIG_ENDIAN
-PDP_ENDIAN = __PDP_ENDIAN
-BYTE_ORDER = __BYTE_ORDER
-
-# Included from bits/byteswap.h
-_BITS_BYTESWAP_H = 1
-def __bswap_constant_16(x): return \
-
-def __bswap_16(x): return \
-
-def __bswap_16(x): return __bswap_constant_16 (x)
-
-def __bswap_constant_32(x): return \
-
-def __bswap_32(x): return \
-
-def __bswap_32(x): return \
-
-def __bswap_32(x): return __bswap_constant_32 (x)
-
-def __bswap_constant_64(x): return \
-
-def __bswap_64(x): return \
-
-def ntohl(x): return (x)
-
-def ntohs(x): return (x)
-
-def htonl(x): return (x)
-
-def htons(x): return (x)
-
-def ntohl(x): return __bswap_32 (x)
-
-def ntohs(x): return __bswap_16 (x)
-
-def htonl(x): return __bswap_32 (x)
-
-def htons(x): return __bswap_16 (x)
-
-def IN6_IS_ADDR_UNSPECIFIED(a): return \
-
-def IN6_IS_ADDR_LOOPBACK(a): return \
-
-def IN6_IS_ADDR_LINKLOCAL(a): return \
-
-def IN6_IS_ADDR_SITELOCAL(a): return \
-
-def IN6_IS_ADDR_V4MAPPED(a): return \
-
-def IN6_IS_ADDR_V4COMPAT(a): return \
-
-def IN6_IS_ADDR_MC_NODELOCAL(a): return \
-
-def IN6_IS_ADDR_MC_LINKLOCAL(a): return \
-
-def IN6_IS_ADDR_MC_SITELOCAL(a): return \
-
-def IN6_IS_ADDR_MC_ORGLOCAL(a): return \
-
-def IN6_IS_ADDR_MC_GLOBAL(a): return
diff --git a/Lib/plat-linux3/TYPES.py b/Lib/plat-linux3/TYPES.py
deleted file mode 100644
--- a/Lib/plat-linux3/TYPES.py
+++ /dev/null
@@ -1,170 +0,0 @@
-# Generated by h2py from /usr/include/sys/types.h
-_SYS_TYPES_H = 1
-
-# Included from features.h
-_FEATURES_H = 1
-__USE_ANSI = 1
-__FAVOR_BSD = 1
-_ISOC99_SOURCE = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 199506
-_XOPEN_SOURCE = 600
-_XOPEN_SOURCE_EXTENDED = 1
-_LARGEFILE64_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-__USE_ISOC99 = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 2
-_POSIX_C_SOURCE = 199506
-__USE_POSIX = 1
-__USE_POSIX2 = 1
-__USE_POSIX199309 = 1
-__USE_POSIX199506 = 1
-__USE_XOPEN = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_UNIX98 = 1
-_LARGEFILE_SOURCE = 1
-__USE_XOPEN2K = 1
-__USE_ISOC99 = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_FILE_OFFSET64 = 1
-__USE_MISC = 1
-__USE_BSD = 1
-__USE_SVID = 1
-__USE_GNU = 1
-__USE_REENTRANT = 1
-__STDC_IEC_559__ = 1
-__STDC_IEC_559_COMPLEX__ = 1
-__STDC_ISO_10646__ = 200009
-__GNU_LIBRARY__ = 6
-__GLIBC__ = 2
-__GLIBC_MINOR__ = 2
-
-# Included from sys/cdefs.h
-_SYS_CDEFS_H = 1
-def __PMT(args): return args
-
-def __P(args): return args
-
-def __PMT(args): return args
-
-def __STRING(x): return #x
-
-__flexarr = []
-__flexarr = [0]
-__flexarr = []
-__flexarr = [1]
-def __ASMNAME(cname): return __ASMNAME2 (__USER_LABEL_PREFIX__, cname)
-
-def __attribute__(xyz): return
-
-def __attribute_format_arg__(x): return __attribute__ ((__format_arg__ (x)))
-
-def __attribute_format_arg__(x): return
-
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_EXTERN_INLINES = 1
-
-# Included from gnu/stubs.h
-
-# Included from bits/types.h
-_BITS_TYPES_H = 1
-__FD_SETSIZE = 1024
-
-# Included from bits/pthreadtypes.h
-_BITS_PTHREADTYPES_H = 1
-
-# Included from bits/sched.h
-SCHED_OTHER = 0
-SCHED_FIFO = 1
-SCHED_RR = 2
-CSIGNAL = 0x000000ff
-CLONE_VM = 0x00000100
-CLONE_FS = 0x00000200
-CLONE_FILES = 0x00000400
-CLONE_SIGHAND = 0x00000800
-CLONE_PID = 0x00001000
-CLONE_PTRACE = 0x00002000
-CLONE_VFORK = 0x00004000
-__defined_schedparam = 1
-
-# Included from time.h
-_TIME_H = 1
-
-# Included from bits/time.h
-_BITS_TIME_H = 1
-CLOCKS_PER_SEC = 1000000
-CLOCK_REALTIME = 0
-CLOCK_PROCESS_CPUTIME_ID = 2
-CLOCK_THREAD_CPUTIME_ID = 3
-TIMER_ABSTIME = 1
-_STRUCT_TIMEVAL = 1
-CLK_TCK = CLOCKS_PER_SEC
-__clock_t_defined = 1
-__time_t_defined = 1
-__clockid_t_defined = 1
-__timer_t_defined = 1
-__timespec_defined = 1
-def __isleap(year): return \
-
-__BIT_TYPES_DEFINED__ = 1
-
-# Included from endian.h
-_ENDIAN_H = 1
-__LITTLE_ENDIAN = 1234
-__BIG_ENDIAN = 4321
-__PDP_ENDIAN = 3412
-
-# Included from bits/endian.h
-__BYTE_ORDER = __LITTLE_ENDIAN
-__FLOAT_WORD_ORDER = __BYTE_ORDER
-LITTLE_ENDIAN = __LITTLE_ENDIAN
-BIG_ENDIAN = __BIG_ENDIAN
-PDP_ENDIAN = __PDP_ENDIAN
-BYTE_ORDER = __BYTE_ORDER
-
-# Included from sys/select.h
-_SYS_SELECT_H = 1
-
-# Included from bits/select.h
-def __FD_ZERO(fdsp): return \
-
-def __FD_ZERO(set): return \
-
-
-# Included from bits/sigset.h
-_SIGSET_H_types = 1
-_SIGSET_H_fns = 1
-def __sigmask(sig): return \
-
-def __sigemptyset(set): return \
-
-def __sigfillset(set): return \
-
-def __sigisemptyset(set): return \
-
-def __FDELT(d): return ((d) / __NFDBITS)
-
-FD_SETSIZE = __FD_SETSIZE
-def FD_ZERO(fdsetp): return __FD_ZERO (fdsetp)
-
-
-# Included from sys/sysmacros.h
-_SYS_SYSMACROS_H = 1
-def major(dev): return ((int)(((dev) >> 8) & 0xff))
-
-def minor(dev): return ((int)((dev) & 0xff))
-
-def major(dev): return (((dev).__val[1] >> 8) & 0xff)
-
-def minor(dev): return ((dev).__val[1] & 0xff)
-
-def major(dev): return (((dev).__val[0] >> 8) & 0xff)
-
-def minor(dev): return ((dev).__val[0] & 0xff)
diff --git a/Lib/plat-linux3/regen b/Lib/plat-linux3/regen
deleted file mode 100755
--- a/Lib/plat-linux3/regen
+++ /dev/null
@@ -1,8 +0,0 @@
-#! /bin/sh
-case `uname` in
-Linux*)	;;
-*)	echo Probably not on a Linux system 1>&2
-	exit 1;;
-esac
-set -v
-h2py -i '(u_long)' /usr/include/sys/types.h /usr/include/netinet/in.h /usr/include/dlfcn.h

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 21:40:13 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 05 Sep 2011 21:40:13 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_null_merge_3=2E2?=
Message-ID: <E1R0f1h-0003Eh-1c@dinsdale.python.org>

http://hg.python.org/cpython/rev/33c671ac2e53
changeset:   72292:33c671ac2e53
parent:      72288:8f1187288fac
parent:      72291:cb47cf5138a4
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Sep 05 21:39:25 2011 +0200
summary:
  null merge 3.2

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 22:36:39 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 05 Sep 2011 22:36:39 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Update_sys=2Epl?=
	=?utf8?q?atform_doc_for_=2312326=2E?=
Message-ID: <E1R0fuJ-0007R5-Ho@dinsdale.python.org>

http://hg.python.org/cpython/rev/0fe571d43317
changeset:   72293:0fe571d43317
branch:      2.7
parent:      72290:d95c4b030eac
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Sep 05 22:33:55 2011 +0200
summary:
  Update sys.platform doc for #12326.

Backport from Python 3.2 (e11b4c945f7e).

files:
  Doc/library/sys.rst |  44 ++++++++++++++++++++------------
  1 files changed, 27 insertions(+), 17 deletions(-)


diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -709,28 +709,38 @@
    This string contains a platform identifier that can be used to append
    platform-specific components to :data:`sys.path`, for instance.
 
-   For Unix systems, this is the lowercased OS name as returned by ``uname -s``
-   with the first part of the version as returned by ``uname -r`` appended,
-   e.g. ``'sunos5'`` or ``'linux2'``, *at the time when Python was built*.
-   Unless you want to test for a specific system version, it is therefore
-   recommended to use the following idiom::
+   For most Unix systems, this is the lowercased OS name as returned by ``uname
+   -s`` with the first part of the version as returned by ``uname -r`` appended,
+   e.g. ``'sunos5'``, *at the time when Python was built*.  Unless you want to
+   test for a specific system version, it is therefore recommended to use the
+   following idiom::
 
-      if sys.platform.startswith('linux'):
+      if sys.platform.startswith('freebsd'):
+          # FreeBSD-specific code here...
+      elif sys.platform.startswith('linux'):
           # Linux-specific code here...
 
+   .. versionchanged:: 2.7.3
+      Since lots of code check for ``sys.platform == 'linux2'``, and there is
+      no essential change between Linux 2.x and 3.x, ``sys.platform`` is always
+      set to ``'linux2'``, even on Linux 3.x.  In Python 3.3 and later, the
+      value will always be set to ``'linux'``, so it is recommended to always
+      use the ``startswith`` idiom presented above.
+
    For other systems, the values are:
 
-   ================ ===========================
-   System           :data:`platform` value
-   ================ ===========================
-   Windows          ``'win32'``
-   Windows/Cygwin   ``'cygwin'``
-   Mac OS X         ``'darwin'``
-   OS/2             ``'os2'``
-   OS/2 EMX         ``'os2emx'``
-   RiscOS           ``'riscos'``
-   AtheOS           ``'atheos'``
-   ================ ===========================
+   ===================== ===========================
+   System                :data:`platform` value
+   ===================== ===========================
+   Linux (2.x *and* 3.x) ``'linux2'``
+   Windows               ``'win32'``
+   Windows/Cygwin        ``'cygwin'``
+   Mac OS X              ``'darwin'``
+   OS/2                  ``'os2'``
+   OS/2 EMX              ``'os2emx'``
+   RiscOS                ``'riscos'``
+   AtheOS                ``'atheos'``
+   ===================== ===========================
 
    .. seealso::
       :attr:`os.name` has a coarser granularity.  :func:`os.uname` gives

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 23:50:27 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 05 Sep 2011 23:50:27 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzk1NjE6?=
 =?utf8?q?_distutils_now_reads_and_writes_egg-info_files_using_UTF-8?=
Message-ID: <E1R0h3j-00047q-3J@dinsdale.python.org>

http://hg.python.org/cpython/rev/fb4d2e6d393e
changeset:   72294:fb4d2e6d393e
branch:      3.2
parent:      72291:cb47cf5138a4
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Sep 05 23:44:56 2011 +0200
summary:
  Issue #9561: distutils now reads and writes egg-info files using UTF-8

instead of the locale encoding.

files:
  Lib/distutils/command/install_egg_info.py |  5 ++---
  Lib/distutils/dist.py                     |  6 ++----
  Misc/NEWS                                 |  5 ++++-
  3 files changed, 8 insertions(+), 8 deletions(-)


diff --git a/Lib/distutils/command/install_egg_info.py b/Lib/distutils/command/install_egg_info.py
--- a/Lib/distutils/command/install_egg_info.py
+++ b/Lib/distutils/command/install_egg_info.py
@@ -40,9 +40,8 @@
                          "Creating "+self.install_dir)
         log.info("Writing %s", target)
         if not self.dry_run:
-            f = open(target, 'w')
-            self.distribution.metadata.write_pkg_file(f)
-            f.close()
+            with open(target, 'w', encoding='UTF-8') as f:
+                self.distribution.metadata.write_pkg_file(f)
 
     def get_outputs(self):
         return self.outputs
diff --git a/Lib/distutils/dist.py b/Lib/distutils/dist.py
--- a/Lib/distutils/dist.py
+++ b/Lib/distutils/dist.py
@@ -1010,11 +1010,9 @@
     def write_pkg_info(self, base_dir):
         """Write the PKG-INFO file into the release tree.
         """
-        pkg_info = open(os.path.join(base_dir, 'PKG-INFO'), 'w')
-        try:
+        with open(os.path.join(base_dir, 'PKG-INFO'), 'w',
+                  encoding='UTF-8') as pkg_info:
             self.write_pkg_file(pkg_info)
-        finally:
-            pkg_info.close()
 
     def write_pkg_file(self, file):
         """Write the PKG-INFO format data to a file object.
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -25,6 +25,9 @@
 Library
 -------
 
+- Issue #9561: distutils now reads and writes egg-info files using UTF-8,
+  instead of the locale encoding.
+
 - Issue #12888: Fix a bug in HTMLParser.unescape that prevented it to escape
   more than 128 entities.  Patch by Peter Otten.
 
@@ -72,7 +75,7 @@
 
 Library
 -------
- 
+
 - Issue #8286: The distutils command sdist will print a warning message instead
   of crashing when an invalid path is given in the manifest template.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep  5 23:50:27 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 05 Sep 2011 23:50:27 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_3=2E2=3A_Issue_=239561=3A_distutils_now_reads_and_writ?=
 =?utf8?q?es_egg-info_files_using?=
Message-ID: <E1R0h3j-00047u-V2@dinsdale.python.org>

http://hg.python.org/cpython/rev/3c080bf75342
changeset:   72295:3c080bf75342
parent:      72292:33c671ac2e53
parent:      72294:fb4d2e6d393e
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Sep 05 23:46:05 2011 +0200
summary:
  Merge 3.2: Issue #9561: distutils now reads and writes egg-info files using UTF-8

instead of the locale encoding.

files:
  Lib/distutils/command/install_egg_info.py |  5 ++---
  Lib/distutils/dist.py                     |  6 ++----
  Misc/NEWS                                 |  3 +++
  3 files changed, 7 insertions(+), 7 deletions(-)


diff --git a/Lib/distutils/command/install_egg_info.py b/Lib/distutils/command/install_egg_info.py
--- a/Lib/distutils/command/install_egg_info.py
+++ b/Lib/distutils/command/install_egg_info.py
@@ -40,9 +40,8 @@
                          "Creating "+self.install_dir)
         log.info("Writing %s", target)
         if not self.dry_run:
-            f = open(target, 'w')
-            self.distribution.metadata.write_pkg_file(f)
-            f.close()
+            with open(target, 'w', encoding='UTF-8') as f:
+                self.distribution.metadata.write_pkg_file(f)
 
     def get_outputs(self):
         return self.outputs
diff --git a/Lib/distutils/dist.py b/Lib/distutils/dist.py
--- a/Lib/distutils/dist.py
+++ b/Lib/distutils/dist.py
@@ -1010,11 +1010,9 @@
     def write_pkg_info(self, base_dir):
         """Write the PKG-INFO file into the release tree.
         """
-        pkg_info = open(os.path.join(base_dir, 'PKG-INFO'), 'w')
-        try:
+        with open(os.path.join(base_dir, 'PKG-INFO'), 'w',
+                  encoding='UTF-8') as pkg_info:
             self.write_pkg_file(pkg_info)
-        finally:
-            pkg_info.close()
 
     def write_pkg_file(self, file):
         """Write the PKG-INFO format data to a file object.
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -271,6 +271,9 @@
 Library
 -------
 
+- Issue #9561: distutils now reads and writes egg-info files using UTF-8,
+  instead of the locale encoding.
+
 - Issue #8286: The distutils command sdist will print a warning message instead
   of crashing when an invalid path is given in the manifest template.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep  6 00:11:39 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 06 Sep 2011 00:11:39 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=239561=3A_packaging_?=
 =?utf8?q?now_writes_egg-info_files_using_UTF-8?=
Message-ID: <E1R0hOF-0005Uh-07@dinsdale.python.org>

http://hg.python.org/cpython/rev/56ab3257ca13
changeset:   72296:56ab3257ca13
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Sep 06 00:11:13 2011 +0200
summary:
  Issue #9561: packaging now writes egg-info files using UTF-8

instead of the locale encoding

files:
  Lib/packaging/tests/test_util.py |  4 ++--
  Misc/NEWS                        |  2 +-
  2 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Lib/packaging/tests/test_util.py b/Lib/packaging/tests/test_util.py
--- a/Lib/packaging/tests/test_util.py
+++ b/Lib/packaging/tests/test_util.py
@@ -946,7 +946,7 @@
 
     def _distutils_pkg_info(self):
         tmp = self._distutils_setup_py_pkg()
-        self.write_file([tmp, 'PKG-INFO'], '')
+        self.write_file([tmp, 'PKG-INFO'], '', encoding='UTF-8')
         return tmp
 
     def _setup_cfg_with_no_metadata_pkg(self):
@@ -971,7 +971,7 @@
 
     def _pkg_info_with_no_distutils(self):
         tmp = self._random_setup_py_pkg()
-        self.write_file([tmp, 'PKG-INFO'], '')
+        self.write_file([tmp, 'PKG-INFO'], '', encoding='UTF-8')
         return tmp
 
     def _random_setup_py_pkg(self):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -271,7 +271,7 @@
 Library
 -------
 
-- Issue #9561: distutils now reads and writes egg-info files using UTF-8,
+- Issue #9561: distutils and packaging now writes egg-info files using UTF-8,
   instead of the locale encoding.
 
 - Issue #8286: The distutils command sdist will print a warning message instead

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep  6 01:53:32 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 06 Sep 2011 01:53:32 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312567=3A_Add_curse?=
 =?utf8?q?s=2Eunget=5Fwch=28=29_function?=
Message-ID: <E1R0iyq-0003HY-5e@dinsdale.python.org>

http://hg.python.org/cpython/rev/b1e03d10391e
changeset:   72297:b1e03d10391e
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Sep 06 01:53:03 2011 +0200
summary:
  Issue #12567: Add curses.unget_wch() function

Push a character so the next get_wch() will return it.

files:
  Doc/library/curses.rst  |  11 ++++
  Lib/test/test_curses.py |  15 ++++++
  Misc/NEWS               |   3 +
  Modules/_cursesmodule.c |  68 +++++++++++++++++++++++++++++
  4 files changed, 97 insertions(+), 0 deletions(-)


diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst
--- a/Doc/library/curses.rst
+++ b/Doc/library/curses.rst
@@ -598,6 +598,17 @@
       Only one *ch* can be pushed before :meth:`getch` is called.
 
 
+.. function:: unget_wch(ch)
+
+   Push *ch* so the next :meth:`get_wch` will return it.
+
+   .. note::
+
+      Only one *ch* can be pushed before :meth:`get_wch` is called.
+
+   .. versionadded:: 3.3
+
+
 .. function:: ungetmouse(id, x, y, z, bstate)
 
    Push a :const:`KEY_MOUSE` event onto the input queue, associating the given
diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py
--- a/Lib/test/test_curses.py
+++ b/Lib/test/test_curses.py
@@ -264,6 +264,20 @@
     curses.ungetch(1025)
     stdscr.getkey()
 
+def test_unget_wch(stdscr):
+    ch = '\xe9'
+    curses.unget_wch(ch)
+    read = stdscr.get_wch()
+    read = chr(read)
+    if read != ch:
+        raise AssertionError("%r != %r" % (read, ch))
+
+    ch = ord('\xe9')
+    curses.unget_wch(ch)
+    read = stdscr.get_wch()
+    if read != ch:
+        raise AssertionError("%r != %r" % (read, ch))
+
 def main(stdscr):
     curses.savetty()
     try:
@@ -272,6 +286,7 @@
         test_userptr_without_set(stdscr)
         test_resize_term(stdscr)
         test_issue6243(stdscr)
+        test_unget_wch(stdscr)
     finally:
         curses.resetty()
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -271,6 +271,9 @@
 Library
 -------
 
+- Issue #12567: Add curses.unget_wch() function. Push a character so the next
+  get_wch() will return it.
+
 - Issue #9561: distutils and packaging now writes egg-info files using UTF-8,
   instead of the locale encoding.
 
diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c
--- a/Modules/_cursesmodule.c
+++ b/Modules/_cursesmodule.c
@@ -2696,6 +2696,71 @@
     return PyCursesCheckERR(ungetch(ch), "ungetch");
 }
 
+#ifdef HAVE_NCURSESW
+/* Convert an object to a character (wchar_t):
+
+    - int
+    - str of length 1
+
+   Return 1 on success, 0 on error. */
+static int
+PyCurses_ConvertToWchar_t(PyObject *obj,
+                          wchar_t *wch)
+{
+    if (PyUnicode_Check(obj)) {
+        wchar_t buffer[2];
+        if (PyUnicode_AsWideChar(obj, buffer, 2) != 1) {
+            PyErr_Format(PyExc_TypeError,
+                         "expect bytes or str of length 1, or int, "
+                         "got a str of length %zi",
+                         PyUnicode_GET_SIZE(obj));
+            return 0;
+        }
+        *wch = buffer[0];
+        return 2;
+    }
+    else if (PyLong_CheckExact(obj)) {
+        long value;
+        int overflow;
+        value = PyLong_AsLongAndOverflow(obj, &overflow);
+        if (overflow) {
+            PyErr_SetString(PyExc_OverflowError,
+                            "int doesn't fit in long");
+            return 0;
+        }
+        *wch = (wchar_t)value;
+        if ((long)*wch != value) {
+            PyErr_Format(PyExc_OverflowError,
+                         "character doesn't fit in wchar_t");
+            return 0;
+        }
+        return 1;
+    }
+    else {
+        PyErr_Format(PyExc_TypeError,
+                     "expect bytes or str of length 1, or int, got %s",
+                     Py_TYPE(obj)->tp_name);
+        return 0;
+    }
+}
+
+static PyObject *
+PyCurses_Unget_Wch(PyObject *self, PyObject *args)
+{
+    PyObject *obj;
+    wchar_t wch;
+
+    PyCursesInitialised;
+
+    if (!PyArg_ParseTuple(args,"O", &obj))
+        return NULL;
+
+    if (!PyCurses_ConvertToWchar_t(obj, &wch))
+        return NULL;
+    return PyCursesCheckERR(unget_wch(wch), "unget_wch");
+}
+#endif
+
 static PyObject *
 PyCurses_Use_Env(PyObject *self, PyObject *args)
 {
@@ -2823,6 +2888,9 @@
     {"typeahead",           (PyCFunction)PyCurses_TypeAhead, METH_VARARGS},
     {"unctrl",              (PyCFunction)PyCurses_UnCtrl, METH_VARARGS},
     {"ungetch",             (PyCFunction)PyCurses_UngetCh, METH_VARARGS},
+#ifdef HAVE_NCURSESW
+    {"unget_wch",           (PyCFunction)PyCurses_Unget_Wch, METH_VARARGS},
+#endif
     {"use_env",             (PyCFunction)PyCurses_Use_Env, METH_VARARGS},
 #ifndef STRICT_SYSV_CURSES
     {"use_default_colors",  (PyCFunction)PyCurses_Use_Default_Colors, METH_NOARGS},

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep  6 02:01:58 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 06 Sep 2011 02:01:58 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_PyUnicode?=
 =?utf8?q?=5FAsWideCharString=28=29_doc=3A_size_doesn=27t_contain_the_null?=
 =?utf8?q?_character?=
Message-ID: <E1R0j70-0003rc-8g@dinsdale.python.org>

http://hg.python.org/cpython/rev/028423c89f8d
changeset:   72298:028423c89f8d
branch:      3.2
parent:      72294:fb4d2e6d393e
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Sep 06 02:00:05 2011 +0200
summary:
  Fix PyUnicode_AsWideCharString() doc: size doesn't contain the null character

Fix also spelling of the null character.

files:
  Include/unicodeobject.h |   2 +-
  Objects/unicodeobject.c |  10 +++++-----
  2 files changed, 6 insertions(+), 6 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -595,7 +595,7 @@
 
 /* Convert the Unicode object to a wide character string. The output string
    always ends with a nul character. If size is not NULL, write the number of
-   wide characters (including the nul character) into *size.
+   wide characters (excluding the null character) into *size.
 
    Returns a buffer allocated by PyMem_Alloc() (use PyMem_Free() to free it)
    on success. On error, returns NULL, *size is undefined and raises a
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -1187,12 +1187,12 @@
 /* Helper function for PyUnicode_AsWideChar() and PyUnicode_AsWideCharString():
    convert a Unicode object to a wide character string.
 
-   - If w is NULL: return the number of wide characters (including the nul
+   - If w is NULL: return the number of wide characters (including the null
      character) required to convert the unicode object. Ignore size argument.
 
-   - Otherwise: return the number of wide characters (excluding the nul
+   - Otherwise: return the number of wide characters (excluding the null
      character) written into w. Write at most size wide characters (including
-     the nul character). */
+     the null character). */
 static Py_ssize_t
 unicode_aswidechar(PyUnicodeObject *unicode,
                    wchar_t *w,
@@ -1240,7 +1240,7 @@
         return w - worig;
     }
     else {
-        nchar = 1; /* nul character at the end */
+        nchar = 1; /* null character at the end */
         while (u != uend) {
             if (0xD800 <= u[0] && u[0] <= 0xDBFF
                 && 0xDC00 <= u[1] && u[1] <= 0xDFFF)
@@ -1278,7 +1278,7 @@
         return w - worig;
     }
     else {
-        nchar = 1; /* nul character */
+        nchar = 1; /* null character */
         while (u != uend) {
             if (*u > 0xffff)
                 nchar += 2;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep  6 02:01:59 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 06 Sep 2011 02:01:59 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_3=2E2=3A_Fix_PyUnicode=5FAsWideCharString=28=29_doc?=
Message-ID: <E1R0j71-0003rm-0o@dinsdale.python.org>

http://hg.python.org/cpython/rev/218e297f72e1
changeset:   72299:218e297f72e1
parent:      72297:b1e03d10391e
parent:      72298:028423c89f8d
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Sep 06 02:01:29 2011 +0200
summary:
  Merge 3.2: Fix PyUnicode_AsWideCharString() doc

 - Fix PyUnicode_AsWideCharString() doc: size doesn't contain the null
   character
 - Fix spelling of the null character

files:
  Include/unicodeobject.h |   2 +-
  Objects/unicodeobject.c |  10 +++++-----
  2 files changed, 6 insertions(+), 6 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -608,7 +608,7 @@
 
 /* Convert the Unicode object to a wide character string. The output string
    always ends with a nul character. If size is not NULL, write the number of
-   wide characters (including the nul character) into *size.
+   wide characters (excluding the null character) into *size.
 
    Returns a buffer allocated by PyMem_Alloc() (use PyMem_Free() to free it)
    on success. On error, returns NULL, *size is undefined and raises a
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -1204,12 +1204,12 @@
 /* Helper function for PyUnicode_AsWideChar() and PyUnicode_AsWideCharString():
    convert a Unicode object to a wide character string.
 
-   - If w is NULL: return the number of wide characters (including the nul
+   - If w is NULL: return the number of wide characters (including the null
      character) required to convert the unicode object. Ignore size argument.
 
-   - Otherwise: return the number of wide characters (excluding the nul
+   - Otherwise: return the number of wide characters (excluding the null
      character) written into w. Write at most size wide characters (including
-     the nul character). */
+     the null character). */
 static Py_ssize_t
 unicode_aswidechar(PyUnicodeObject *unicode,
                    wchar_t *w,
@@ -1257,7 +1257,7 @@
         return w - worig;
     }
     else {
-        nchar = 1; /* nul character at the end */
+        nchar = 1; /* null character at the end */
         while (u != uend) {
             if (0xD800 <= u[0] && u[0] <= 0xDBFF
                 && 0xDC00 <= u[1] && u[1] <= 0xDFFF)
@@ -1295,7 +1295,7 @@
         return w - worig;
     }
     else {
-        nchar = 1; /* nul character */
+        nchar = 1; /* null character */
         while (u != uend) {
             if (*u > 0xffff)
                 nchar += 2;

-- 
Repository URL: http://hg.python.org/cpython

From ncoghlan at gmail.com  Tue Sep  6 02:25:58 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 6 Sep 2011 10:25:58 +1000
Subject: [Python-checkins] cpython (3.2): Fix
 PyUnicode_AsWideCharString() doc: size doesn't contain the null character
In-Reply-To: <E1R0j70-0003rc-8g@dinsdale.python.org>
References: <E1R0j70-0003rc-8g@dinsdale.python.org>
Message-ID: <CADiSq7eqm2CXKVeZpwo5+X3vm07+a+xGuQE-65nvL2+8t8gaDA@mail.gmail.com>

On Tue, Sep 6, 2011 at 10:01 AM, victor.stinner
<python-checkins at python.org> wrote:
> Fix also spelling of the null character.

While these cases are legitimately changed to 'null' (since they're
lowercase descriptions of the character), I figure it's worth
mentioning again that the ASCII name for '\0' actually *is* NUL (i.e.
only one 'L'). Strange, but true [1].

Cheers,
Nick.

[1] https://secure.wikimedia.org/wikipedia/en/wiki/ASCII

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia

From solipsis at pitrou.net  Tue Sep  6 05:16:28 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Tue, 06 Sep 2011 05:16:28 +0200
Subject: [Python-checkins] Daily reference leaks (218e297f72e1): sum=24
Message-ID: <E1R0m9E-0003q6-DL@ap.vmr.nerim.net>

results for 218e297f72e1 on branch "default"
--------------------------------------------

test_multiprocessing leaked [24, 0, 0] references, sum=24


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogFph8t3', '-x']

From python-checkins at python.org  Tue Sep  6 06:09:10 2011
From: python-checkins at python.org (brett.cannon)
Date: Tue, 06 Sep 2011 06:09:10 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Minor_grammar_fix=2E?=
Message-ID: <E1R0myE-0007iG-EH@dinsdale.python.org>

http://hg.python.org/cpython/rev/84595a914e5e
changeset:   72300:84595a914e5e
user:        Brett Cannon <brett at python.org>
date:        Mon Sep 05 21:08:14 2011 -0700
summary:
  Minor grammar fix.

files:
  Doc/packaging/setupcfg.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/packaging/setupcfg.rst b/Doc/packaging/setupcfg.rst
--- a/Doc/packaging/setupcfg.rst
+++ b/Doc/packaging/setupcfg.rst
@@ -72,7 +72,7 @@
 ---------------
 
 A configuration file can be extended (i.e. included) by other files.  For this,
-a ``DEFAULT`` section must contain an ``extends`` key which value points to one
+a ``DEFAULT`` section must contain an ``extends`` key whose value points to one
 or more files which will be merged into the current files by adding new sections
 and fields.  If a file loaded by ``extends`` contains sections or keys that
 already exist in the original file, they will not override the previous values.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep  6 10:08:40 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 06 Sep 2011 10:08:40 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312567=3A_Fix_curse?=
 =?utf8?q?s=2Eunget=5Fwch=28=29_tests?=
Message-ID: <E1R0qi0-0002Zh-I3@dinsdale.python.org>

http://hg.python.org/cpython/rev/786668a4fb6b
changeset:   72301:786668a4fb6b
user:        Victor Stinner <vstinner at wyplay.com>
date:        Tue Sep 06 10:08:28 2011 +0200
summary:
  Issue #12567: Fix curses.unget_wch() tests

Skip the test if the function is missing. Use U+0061 (a) instead of U+00E9 (?)
because U+00E9 raises a _curses.error('unget_wch() returned ERR') on some
buildbots. It's maybe because of the locale encoding.

files:
  Lib/test/test_curses.py |  6 ++++--
  1 files changed, 4 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py
--- a/Lib/test/test_curses.py
+++ b/Lib/test/test_curses.py
@@ -265,14 +265,16 @@
     stdscr.getkey()
 
 def test_unget_wch(stdscr):
-    ch = '\xe9'
+    if not hasattr(curses, 'unget_wch'):
+        return
+    ch = 'a'
     curses.unget_wch(ch)
     read = stdscr.get_wch()
     read = chr(read)
     if read != ch:
         raise AssertionError("%r != %r" % (read, ch))
 
-    ch = ord('\xe9')
+    ch = ord('a')
     curses.unget_wch(ch)
     read = stdscr.get_wch()
     if read != ch:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep  6 13:57:48 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Tue, 06 Sep 2011 13:57:48 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_cast_to_getter?=
Message-ID: <E1R0uHk-0001SV-Ac@dinsdale.python.org>

http://hg.python.org/cpython/rev/4cf619af7dc8
changeset:   72302:4cf619af7dc8
branch:      3.2
parent:      72266:792606c351cf
user:        Benjamin Peterson <benjamin at python.org>
date:        Tue Sep 06 07:55:34 2011 -0400
summary:
  cast to getter

files:
  Modules/_io/iobase.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Modules/_io/iobase.c b/Modules/_io/iobase.c
--- a/Modules/_io/iobase.c
+++ b/Modules/_io/iobase.c
@@ -704,7 +704,7 @@
 };
 
 static PyGetSetDef iobase_getset[] = {
-    {"__dict__", iobase_get_dict, NULL, NULL},
+    {"__dict__", (getter)iobase_get_dict, NULL, NULL},
     {"closed", (getter)iobase_closed_get, NULL, NULL},
     {NULL}
 };

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep  6 13:57:49 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Tue, 06 Sep 2011 13:57:49 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?b?OiBtZXJnZSAzLjIgKCMxNjE2KQ==?=
Message-ID: <E1R0uHl-0001Se-2q@dinsdale.python.org>

http://hg.python.org/cpython/rev/f24352b0df86
changeset:   72303:f24352b0df86
parent:      72267:df673c6d3e0b
parent:      72302:4cf619af7dc8
user:        Benjamin Peterson <benjamin at python.org>
date:        Tue Sep 06 07:56:47 2011 -0400
summary:
  merge 3.2 (#1616)

files:
  Modules/_io/iobase.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Modules/_io/iobase.c b/Modules/_io/iobase.c
--- a/Modules/_io/iobase.c
+++ b/Modules/_io/iobase.c
@@ -704,7 +704,7 @@
 };
 
 static PyGetSetDef iobase_getset[] = {
-    {"__dict__", iobase_get_dict, NULL, NULL},
+    {"__dict__", (getter)iobase_get_dict, NULL, NULL},
     {"closed", (getter)iobase_closed_get, NULL, NULL},
     {NULL}
 };

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep  6 13:57:49 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Tue, 06 Sep 2011 13:57:49 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMik6?=
	=?utf8?q?_merge_heads?=
Message-ID: <E1R0uHl-0001Sl-TD@dinsdale.python.org>

http://hg.python.org/cpython/rev/a0628933a2f9
changeset:   72304:a0628933a2f9
branch:      3.2
parent:      72302:4cf619af7dc8
parent:      72298:028423c89f8d
user:        Benjamin Peterson <benjamin at python.org>
date:        Tue Sep 06 07:57:26 2011 -0400
summary:
  merge heads

files:
  Doc/library/multiprocessing.rst           |   12 +-
  Include/unicodeobject.h                   |    2 +-
  Lib/distutils/command/install_egg_info.py |    5 +-
  Lib/distutils/command/sdist.py            |    5 +-
  Lib/distutils/dist.py                     |    6 +-
  Lib/distutils/tests/test_sdist.py         |   37 +-
  Lib/html/parser.py                        |    2 +-
  Lib/plat-linux3/CDROM.py                  |  207 ---
  Lib/plat-linux3/DLFCN.py                  |   83 -
  Lib/plat-linux3/IN.py                     |  615 ----------
  Lib/plat-linux3/TYPES.py                  |  170 --
  Lib/plat-linux3/regen                     |    8 -
  Lib/tarfile.py                            |   10 +-
  Lib/test/test_htmlparser.py               |    3 +-
  Misc/ACKS                                 |    1 +
  Misc/NEWS                                 |   14 +
  Objects/unicodeobject.c                   |   10 +-
  Tools/msi/uuids.py                        |    5 +
  18 files changed, 76 insertions(+), 1119 deletions(-)


diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst
--- a/Doc/library/multiprocessing.rst
+++ b/Doc/library/multiprocessing.rst
@@ -552,9 +552,9 @@
       Return ``True`` if the queue is full, ``False`` otherwise.  Because of
       multithreading/multiprocessing semantics, this is not reliable.
 
-   .. method:: put(item[, block[, timeout]])
-
-      Put item into the queue.  If the optional argument *block* is ``True``
+   .. method:: put(obj[, block[, timeout]])
+
+      Put obj into the queue.  If the optional argument *block* is ``True``
       (the default) and *timeout* is ``None`` (the default), block if necessary until
       a free slot is available.  If *timeout* is a positive number, it blocks at
       most *timeout* seconds and raises the :exc:`queue.Full` exception if no
@@ -563,9 +563,9 @@
       available, else raise the :exc:`queue.Full` exception (*timeout* is
       ignored in that case).
 
-   .. method:: put_nowait(item)
-
-      Equivalent to ``put(item, False)``.
+   .. method:: put_nowait(obj)
+
+      Equivalent to ``put(obj, False)``.
 
    .. method:: get([block[, timeout]])
 
diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -595,7 +595,7 @@
 
 /* Convert the Unicode object to a wide character string. The output string
    always ends with a nul character. If size is not NULL, write the number of
-   wide characters (including the nul character) into *size.
+   wide characters (excluding the null character) into *size.
 
    Returns a buffer allocated by PyMem_Alloc() (use PyMem_Free() to free it)
    on success. On error, returns NULL, *size is undefined and raises a
diff --git a/Lib/distutils/command/install_egg_info.py b/Lib/distutils/command/install_egg_info.py
--- a/Lib/distutils/command/install_egg_info.py
+++ b/Lib/distutils/command/install_egg_info.py
@@ -40,9 +40,8 @@
                          "Creating "+self.install_dir)
         log.info("Writing %s", target)
         if not self.dry_run:
-            f = open(target, 'w')
-            self.distribution.metadata.write_pkg_file(f)
-            f.close()
+            with open(target, 'w', encoding='UTF-8') as f:
+                self.distribution.metadata.write_pkg_file(f)
 
     def get_outputs(self):
         return self.outputs
diff --git a/Lib/distutils/command/sdist.py b/Lib/distutils/command/sdist.py
--- a/Lib/distutils/command/sdist.py
+++ b/Lib/distutils/command/sdist.py
@@ -306,7 +306,10 @@
 
                 try:
                     self.filelist.process_template_line(line)
-                except DistutilsTemplateError as msg:
+                # the call above can raise a DistutilsTemplateError for
+                # malformed lines, or a ValueError from the lower-level
+                # convert_path function
+                except (DistutilsTemplateError, ValueError) as msg:
                     self.warn("%s, line %d: %s" % (template.filename,
                                                    template.current_line,
                                                    msg))
diff --git a/Lib/distutils/dist.py b/Lib/distutils/dist.py
--- a/Lib/distutils/dist.py
+++ b/Lib/distutils/dist.py
@@ -1010,11 +1010,9 @@
     def write_pkg_info(self, base_dir):
         """Write the PKG-INFO file into the release tree.
         """
-        pkg_info = open(os.path.join(base_dir, 'PKG-INFO'), 'w')
-        try:
+        with open(os.path.join(base_dir, 'PKG-INFO'), 'w',
+                  encoding='UTF-8') as pkg_info:
             self.write_pkg_file(pkg_info)
-        finally:
-            pkg_info.close()
 
     def write_pkg_file(self, file):
         """Write the PKG-INFO format data to a file object.
diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -15,6 +15,7 @@
 from distutils.errors import DistutilsOptionError
 from distutils.spawn import find_executable
 from distutils.log import WARN
+from distutils.filelist import FileList
 from distutils.archive_util import ARCHIVE_FORMATS
 
 SETUP_PY = """
@@ -78,9 +79,6 @@
         dist.include_package_data = True
         cmd = sdist(dist)
         cmd.dist_dir = 'dist'
-        def _warn(*args):
-            pass
-        cmd.warn = _warn
         return dist, cmd
 
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
@@ -235,7 +233,8 @@
         # with the `check` subcommand
         cmd.ensure_finalized()
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 2)
 
         # trying with a complete set of metadata
@@ -244,7 +243,8 @@
         cmd.ensure_finalized()
         cmd.metadata_check = 0
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 0)
 
     def test_check_metadata_deprecated(self):
@@ -266,7 +266,6 @@
         self.assertEqual(len(output), num_formats)
 
     def test_finalize_options(self):
-
         dist, cmd = self.get_cmd()
         cmd.finalize_options()
 
@@ -286,6 +285,32 @@
         cmd.formats = 'supazipa'
         self.assertRaises(DistutilsOptionError, cmd.finalize_options)
 
+    # the following tests make sure there is a nice error message instead
+    # of a traceback when parsing an invalid manifest template
+
+    def _test_template(self, content):
+        dist, cmd = self.get_cmd()
+        os.chdir(self.tmp_dir)
+        self.write_file('MANIFEST.in', content)
+        cmd.ensure_finalized()
+        cmd.filelist = FileList()
+        cmd.read_template()
+        warnings = self.get_logs(WARN)
+        self.assertEqual(len(warnings), 1)
+
+    def test_invalid_template_unknown_command(self):
+        self._test_template('taunt knights *')
+
+    def test_invalid_template_wrong_arguments(self):
+        # this manifest command takes one argument
+        self._test_template('prune')
+
+    @unittest.skipIf(os.name != 'nt', 'test relevant for Windows only')
+    def test_invalid_template_wrong_path(self):
+        # on Windows, trailing slashes are not allowed
+        # this used to crash instead of raising a warning: #8286
+        self._test_template('include examples/')
+
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
     def test_get_file_list(self):
         # make sure MANIFEST is recalculated
diff --git a/Lib/html/parser.py b/Lib/html/parser.py
--- a/Lib/html/parser.py
+++ b/Lib/html/parser.py
@@ -458,4 +458,4 @@
                     return '&'+s+';'
 
         return re.sub(r"&(#?[xX]?(?:[0-9a-fA-F]+|\w{1,8}));",
-                      replaceEntities, s, re.ASCII)
+                      replaceEntities, s, flags=re.ASCII)
diff --git a/Lib/plat-linux3/CDROM.py b/Lib/plat-linux3/CDROM.py
deleted file mode 100644
--- a/Lib/plat-linux3/CDROM.py
+++ /dev/null
@@ -1,207 +0,0 @@
-# Generated by h2py from /usr/include/linux/cdrom.h
-
-CDROMPAUSE = 0x5301
-CDROMRESUME = 0x5302
-CDROMPLAYMSF = 0x5303
-CDROMPLAYTRKIND = 0x5304
-CDROMREADTOCHDR = 0x5305
-CDROMREADTOCENTRY = 0x5306
-CDROMSTOP = 0x5307
-CDROMSTART = 0x5308
-CDROMEJECT = 0x5309
-CDROMVOLCTRL = 0x530a
-CDROMSUBCHNL = 0x530b
-CDROMREADMODE2 = 0x530c
-CDROMREADMODE1 = 0x530d
-CDROMREADAUDIO = 0x530e
-CDROMEJECT_SW = 0x530f
-CDROMMULTISESSION = 0x5310
-CDROM_GET_MCN = 0x5311
-CDROM_GET_UPC = CDROM_GET_MCN
-CDROMRESET = 0x5312
-CDROMVOLREAD = 0x5313
-CDROMREADRAW = 0x5314
-CDROMREADCOOKED = 0x5315
-CDROMSEEK = 0x5316
-CDROMPLAYBLK = 0x5317
-CDROMREADALL = 0x5318
-CDROMGETSPINDOWN = 0x531d
-CDROMSETSPINDOWN = 0x531e
-CDROMCLOSETRAY = 0x5319
-CDROM_SET_OPTIONS = 0x5320
-CDROM_CLEAR_OPTIONS = 0x5321
-CDROM_SELECT_SPEED = 0x5322
-CDROM_SELECT_DISC = 0x5323
-CDROM_MEDIA_CHANGED = 0x5325
-CDROM_DRIVE_STATUS = 0x5326
-CDROM_DISC_STATUS = 0x5327
-CDROM_CHANGER_NSLOTS = 0x5328
-CDROM_LOCKDOOR = 0x5329
-CDROM_DEBUG = 0x5330
-CDROM_GET_CAPABILITY = 0x5331
-CDROMAUDIOBUFSIZ = 0x5382
-DVD_READ_STRUCT = 0x5390
-DVD_WRITE_STRUCT = 0x5391
-DVD_AUTH = 0x5392
-CDROM_SEND_PACKET = 0x5393
-CDROM_NEXT_WRITABLE = 0x5394
-CDROM_LAST_WRITTEN = 0x5395
-CDROM_PACKET_SIZE = 12
-CGC_DATA_UNKNOWN = 0
-CGC_DATA_WRITE = 1
-CGC_DATA_READ = 2
-CGC_DATA_NONE = 3
-CD_MINS = 74
-CD_SECS = 60
-CD_FRAMES = 75
-CD_SYNC_SIZE = 12
-CD_MSF_OFFSET = 150
-CD_CHUNK_SIZE = 24
-CD_NUM_OF_CHUNKS = 98
-CD_FRAMESIZE_SUB = 96
-CD_HEAD_SIZE = 4
-CD_SUBHEAD_SIZE = 8
-CD_EDC_SIZE = 4
-CD_ZERO_SIZE = 8
-CD_ECC_SIZE = 276
-CD_FRAMESIZE = 2048
-CD_FRAMESIZE_RAW = 2352
-CD_FRAMESIZE_RAWER = 2646
-CD_FRAMESIZE_RAW1 = (CD_FRAMESIZE_RAW-CD_SYNC_SIZE)
-CD_FRAMESIZE_RAW0 = (CD_FRAMESIZE_RAW-CD_SYNC_SIZE-CD_HEAD_SIZE)
-CD_XA_HEAD = (CD_HEAD_SIZE+CD_SUBHEAD_SIZE)
-CD_XA_TAIL = (CD_EDC_SIZE+CD_ECC_SIZE)
-CD_XA_SYNC_HEAD = (CD_SYNC_SIZE+CD_XA_HEAD)
-CDROM_LBA = 0x01
-CDROM_MSF = 0x02
-CDROM_DATA_TRACK = 0x04
-CDROM_LEADOUT = 0xAA
-CDROM_AUDIO_INVALID = 0x00
-CDROM_AUDIO_PLAY = 0x11
-CDROM_AUDIO_PAUSED = 0x12
-CDROM_AUDIO_COMPLETED = 0x13
-CDROM_AUDIO_ERROR = 0x14
-CDROM_AUDIO_NO_STATUS = 0x15
-CDC_CLOSE_TRAY = 0x1
-CDC_OPEN_TRAY = 0x2
-CDC_LOCK = 0x4
-CDC_SELECT_SPEED = 0x8
-CDC_SELECT_DISC = 0x10
-CDC_MULTI_SESSION = 0x20
-CDC_MCN = 0x40
-CDC_MEDIA_CHANGED = 0x80
-CDC_PLAY_AUDIO = 0x100
-CDC_RESET = 0x200
-CDC_IOCTLS = 0x400
-CDC_DRIVE_STATUS = 0x800
-CDC_GENERIC_PACKET = 0x1000
-CDC_CD_R = 0x2000
-CDC_CD_RW = 0x4000
-CDC_DVD = 0x8000
-CDC_DVD_R = 0x10000
-CDC_DVD_RAM = 0x20000
-CDS_NO_INFO = 0
-CDS_NO_DISC = 1
-CDS_TRAY_OPEN = 2
-CDS_DRIVE_NOT_READY = 3
-CDS_DISC_OK = 4
-CDS_AUDIO = 100
-CDS_DATA_1 = 101
-CDS_DATA_2 = 102
-CDS_XA_2_1 = 103
-CDS_XA_2_2 = 104
-CDS_MIXED = 105
-CDO_AUTO_CLOSE = 0x1
-CDO_AUTO_EJECT = 0x2
-CDO_USE_FFLAGS = 0x4
-CDO_LOCK = 0x8
-CDO_CHECK_TYPE = 0x10
-CD_PART_MAX = 64
-CD_PART_MASK = (CD_PART_MAX - 1)
-GPCMD_BLANK = 0xa1
-GPCMD_CLOSE_TRACK = 0x5b
-GPCMD_FLUSH_CACHE = 0x35
-GPCMD_FORMAT_UNIT = 0x04
-GPCMD_GET_CONFIGURATION = 0x46
-GPCMD_GET_EVENT_STATUS_NOTIFICATION = 0x4a
-GPCMD_GET_PERFORMANCE = 0xac
-GPCMD_INQUIRY = 0x12
-GPCMD_LOAD_UNLOAD = 0xa6
-GPCMD_MECHANISM_STATUS = 0xbd
-GPCMD_MODE_SELECT_10 = 0x55
-GPCMD_MODE_SENSE_10 = 0x5a
-GPCMD_PAUSE_RESUME = 0x4b
-GPCMD_PLAY_AUDIO_10 = 0x45
-GPCMD_PLAY_AUDIO_MSF = 0x47
-GPCMD_PLAY_AUDIO_TI = 0x48
-GPCMD_PLAY_CD = 0xbc
-GPCMD_PREVENT_ALLOW_MEDIUM_REMOVAL = 0x1e
-GPCMD_READ_10 = 0x28
-GPCMD_READ_12 = 0xa8
-GPCMD_READ_CDVD_CAPACITY = 0x25
-GPCMD_READ_CD = 0xbe
-GPCMD_READ_CD_MSF = 0xb9
-GPCMD_READ_DISC_INFO = 0x51
-GPCMD_READ_DVD_STRUCTURE = 0xad
-GPCMD_READ_FORMAT_CAPACITIES = 0x23
-GPCMD_READ_HEADER = 0x44
-GPCMD_READ_TRACK_RZONE_INFO = 0x52
-GPCMD_READ_SUBCHANNEL = 0x42
-GPCMD_READ_TOC_PMA_ATIP = 0x43
-GPCMD_REPAIR_RZONE_TRACK = 0x58
-GPCMD_REPORT_KEY = 0xa4
-GPCMD_REQUEST_SENSE = 0x03
-GPCMD_RESERVE_RZONE_TRACK = 0x53
-GPCMD_SCAN = 0xba
-GPCMD_SEEK = 0x2b
-GPCMD_SEND_DVD_STRUCTURE = 0xad
-GPCMD_SEND_EVENT = 0xa2
-GPCMD_SEND_KEY = 0xa3
-GPCMD_SEND_OPC = 0x54
-GPCMD_SET_READ_AHEAD = 0xa7
-GPCMD_SET_STREAMING = 0xb6
-GPCMD_START_STOP_UNIT = 0x1b
-GPCMD_STOP_PLAY_SCAN = 0x4e
-GPCMD_TEST_UNIT_READY = 0x00
-GPCMD_VERIFY_10 = 0x2f
-GPCMD_WRITE_10 = 0x2a
-GPCMD_WRITE_AND_VERIFY_10 = 0x2e
-GPCMD_SET_SPEED = 0xbb
-GPCMD_PLAYAUDIO_TI = 0x48
-GPCMD_GET_MEDIA_STATUS = 0xda
-GPMODE_R_W_ERROR_PAGE = 0x01
-GPMODE_WRITE_PARMS_PAGE = 0x05
-GPMODE_AUDIO_CTL_PAGE = 0x0e
-GPMODE_POWER_PAGE = 0x1a
-GPMODE_FAULT_FAIL_PAGE = 0x1c
-GPMODE_TO_PROTECT_PAGE = 0x1d
-GPMODE_CAPABILITIES_PAGE = 0x2a
-GPMODE_ALL_PAGES = 0x3f
-GPMODE_CDROM_PAGE = 0x0d
-DVD_STRUCT_PHYSICAL = 0x00
-DVD_STRUCT_COPYRIGHT = 0x01
-DVD_STRUCT_DISCKEY = 0x02
-DVD_STRUCT_BCA = 0x03
-DVD_STRUCT_MANUFACT = 0x04
-DVD_LAYERS = 4
-DVD_LU_SEND_AGID = 0
-DVD_HOST_SEND_CHALLENGE = 1
-DVD_LU_SEND_KEY1 = 2
-DVD_LU_SEND_CHALLENGE = 3
-DVD_HOST_SEND_KEY2 = 4
-DVD_AUTH_ESTABLISHED = 5
-DVD_AUTH_FAILURE = 6
-DVD_LU_SEND_TITLE_KEY = 7
-DVD_LU_SEND_ASF = 8
-DVD_INVALIDATE_AGID = 9
-DVD_LU_SEND_RPC_STATE = 10
-DVD_HOST_SEND_RPC_STATE = 11
-DVD_CPM_NO_COPYRIGHT = 0
-DVD_CPM_COPYRIGHTED = 1
-DVD_CP_SEC_NONE = 0
-DVD_CP_SEC_EXIST = 1
-DVD_CGMS_UNRESTRICTED = 0
-DVD_CGMS_SINGLE = 2
-DVD_CGMS_RESTRICTED = 3
-
-CDROM_MAX_SLOTS = 256
diff --git a/Lib/plat-linux3/DLFCN.py b/Lib/plat-linux3/DLFCN.py
deleted file mode 100644
--- a/Lib/plat-linux3/DLFCN.py
+++ /dev/null
@@ -1,83 +0,0 @@
-# Generated by h2py from /usr/include/dlfcn.h
-_DLFCN_H = 1
-
-# Included from features.h
-_FEATURES_H = 1
-__USE_ANSI = 1
-__FAVOR_BSD = 1
-_ISOC99_SOURCE = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 199506
-_XOPEN_SOURCE = 600
-_XOPEN_SOURCE_EXTENDED = 1
-_LARGEFILE64_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-__USE_ISOC99 = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 2
-_POSIX_C_SOURCE = 199506
-__USE_POSIX = 1
-__USE_POSIX2 = 1
-__USE_POSIX199309 = 1
-__USE_POSIX199506 = 1
-__USE_XOPEN = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_UNIX98 = 1
-_LARGEFILE_SOURCE = 1
-__USE_XOPEN2K = 1
-__USE_ISOC99 = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_FILE_OFFSET64 = 1
-__USE_MISC = 1
-__USE_BSD = 1
-__USE_SVID = 1
-__USE_GNU = 1
-__USE_REENTRANT = 1
-__STDC_IEC_559__ = 1
-__STDC_IEC_559_COMPLEX__ = 1
-__STDC_ISO_10646__ = 200009
-__GNU_LIBRARY__ = 6
-__GLIBC__ = 2
-__GLIBC_MINOR__ = 2
-
-# Included from sys/cdefs.h
-_SYS_CDEFS_H = 1
-def __PMT(args): return args
-
-def __P(args): return args
-
-def __PMT(args): return args
-
-def __STRING(x): return #x
-
-__flexarr = []
-__flexarr = [0]
-__flexarr = []
-__flexarr = [1]
-def __ASMNAME(cname): return __ASMNAME2 (__USER_LABEL_PREFIX__, cname)
-
-def __attribute__(xyz): return
-
-def __attribute_format_arg__(x): return __attribute__ ((__format_arg__ (x)))
-
-def __attribute_format_arg__(x): return
-
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_EXTERN_INLINES = 1
-
-# Included from gnu/stubs.h
-
-# Included from bits/dlfcn.h
-RTLD_LAZY = 0x00001
-RTLD_NOW = 0x00002
-RTLD_BINDING_MASK = 0x3
-RTLD_NOLOAD = 0x00004
-RTLD_GLOBAL = 0x00100
-RTLD_LOCAL = 0
-RTLD_NODELETE = 0x01000
diff --git a/Lib/plat-linux3/IN.py b/Lib/plat-linux3/IN.py
deleted file mode 100644
--- a/Lib/plat-linux3/IN.py
+++ /dev/null
@@ -1,615 +0,0 @@
-# Generated by h2py from /usr/include/netinet/in.h
-_NETINET_IN_H = 1
-
-# Included from features.h
-_FEATURES_H = 1
-__USE_ANSI = 1
-__FAVOR_BSD = 1
-_ISOC99_SOURCE = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 199506
-_XOPEN_SOURCE = 600
-_XOPEN_SOURCE_EXTENDED = 1
-_LARGEFILE64_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-__USE_ISOC99 = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 2
-_POSIX_C_SOURCE = 199506
-__USE_POSIX = 1
-__USE_POSIX2 = 1
-__USE_POSIX199309 = 1
-__USE_POSIX199506 = 1
-__USE_XOPEN = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_UNIX98 = 1
-_LARGEFILE_SOURCE = 1
-__USE_XOPEN2K = 1
-__USE_ISOC99 = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_FILE_OFFSET64 = 1
-__USE_MISC = 1
-__USE_BSD = 1
-__USE_SVID = 1
-__USE_GNU = 1
-__USE_REENTRANT = 1
-__STDC_IEC_559__ = 1
-__STDC_IEC_559_COMPLEX__ = 1
-__STDC_ISO_10646__ = 200009
-__GNU_LIBRARY__ = 6
-__GLIBC__ = 2
-__GLIBC_MINOR__ = 2
-
-# Included from sys/cdefs.h
-_SYS_CDEFS_H = 1
-def __PMT(args): return args
-
-def __P(args): return args
-
-def __PMT(args): return args
-
-def __STRING(x): return #x
-
-__flexarr = []
-__flexarr = [0]
-__flexarr = []
-__flexarr = [1]
-def __ASMNAME(cname): return __ASMNAME2 (__USER_LABEL_PREFIX__, cname)
-
-def __attribute__(xyz): return
-
-def __attribute_format_arg__(x): return __attribute__ ((__format_arg__ (x)))
-
-def __attribute_format_arg__(x): return
-
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_EXTERN_INLINES = 1
-
-# Included from gnu/stubs.h
-
-# Included from stdint.h
-_STDINT_H = 1
-
-# Included from bits/wchar.h
-_BITS_WCHAR_H = 1
-__WCHAR_MIN = (-2147483647 - 1)
-__WCHAR_MAX = (2147483647)
-
-# Included from bits/wordsize.h
-__WORDSIZE = 32
-def __INT64_C(c): return c ## L
-
-def __UINT64_C(c): return c ## UL
-
-def __INT64_C(c): return c ## LL
-
-def __UINT64_C(c): return c ## ULL
-
-INT8_MIN = (-128)
-INT16_MIN = (-32767-1)
-INT32_MIN = (-2147483647-1)
-INT64_MIN = (-__INT64_C(9223372036854775807)-1)
-INT8_MAX = (127)
-INT16_MAX = (32767)
-INT32_MAX = (2147483647)
-INT64_MAX = (__INT64_C(9223372036854775807))
-UINT8_MAX = (255)
-UINT16_MAX = (65535)
-UINT64_MAX = (__UINT64_C(18446744073709551615))
-INT_LEAST8_MIN = (-128)
-INT_LEAST16_MIN = (-32767-1)
-INT_LEAST32_MIN = (-2147483647-1)
-INT_LEAST64_MIN = (-__INT64_C(9223372036854775807)-1)
-INT_LEAST8_MAX = (127)
-INT_LEAST16_MAX = (32767)
-INT_LEAST32_MAX = (2147483647)
-INT_LEAST64_MAX = (__INT64_C(9223372036854775807))
-UINT_LEAST8_MAX = (255)
-UINT_LEAST16_MAX = (65535)
-UINT_LEAST64_MAX = (__UINT64_C(18446744073709551615))
-INT_FAST8_MIN = (-128)
-INT_FAST16_MIN = (-9223372036854775807-1)
-INT_FAST32_MIN = (-9223372036854775807-1)
-INT_FAST16_MIN = (-2147483647-1)
-INT_FAST32_MIN = (-2147483647-1)
-INT_FAST64_MIN = (-__INT64_C(9223372036854775807)-1)
-INT_FAST8_MAX = (127)
-INT_FAST16_MAX = (9223372036854775807)
-INT_FAST32_MAX = (9223372036854775807)
-INT_FAST16_MAX = (2147483647)
-INT_FAST32_MAX = (2147483647)
-INT_FAST64_MAX = (__INT64_C(9223372036854775807))
-UINT_FAST8_MAX = (255)
-UINT_FAST64_MAX = (__UINT64_C(18446744073709551615))
-INTPTR_MIN = (-9223372036854775807-1)
-INTPTR_MAX = (9223372036854775807)
-INTPTR_MIN = (-2147483647-1)
-INTPTR_MAX = (2147483647)
-INTMAX_MIN = (-__INT64_C(9223372036854775807)-1)
-INTMAX_MAX = (__INT64_C(9223372036854775807))
-UINTMAX_MAX = (__UINT64_C(18446744073709551615))
-PTRDIFF_MIN = (-9223372036854775807-1)
-PTRDIFF_MAX = (9223372036854775807)
-PTRDIFF_MIN = (-2147483647-1)
-PTRDIFF_MAX = (2147483647)
-SIG_ATOMIC_MIN = (-2147483647-1)
-SIG_ATOMIC_MAX = (2147483647)
-WCHAR_MIN = __WCHAR_MIN
-WCHAR_MAX = __WCHAR_MAX
-def INT8_C(c): return c
-
-def INT16_C(c): return c
-
-def INT32_C(c): return c
-
-def INT64_C(c): return c ## L
-
-def INT64_C(c): return c ## LL
-
-def UINT8_C(c): return c ## U
-
-def UINT16_C(c): return c ## U
-
-def UINT32_C(c): return c ## U
-
-def UINT64_C(c): return c ## UL
-
-def UINT64_C(c): return c ## ULL
-
-def INTMAX_C(c): return c ## L
-
-def UINTMAX_C(c): return c ## UL
-
-def INTMAX_C(c): return c ## LL
-
-def UINTMAX_C(c): return c ## ULL
-
-
-# Included from bits/types.h
-_BITS_TYPES_H = 1
-__FD_SETSIZE = 1024
-
-# Included from bits/pthreadtypes.h
-_BITS_PTHREADTYPES_H = 1
-
-# Included from bits/sched.h
-SCHED_OTHER = 0
-SCHED_FIFO = 1
-SCHED_RR = 2
-CSIGNAL = 0x000000ff
-CLONE_VM = 0x00000100
-CLONE_FS = 0x00000200
-CLONE_FILES = 0x00000400
-CLONE_SIGHAND = 0x00000800
-CLONE_PID = 0x00001000
-CLONE_PTRACE = 0x00002000
-CLONE_VFORK = 0x00004000
-__defined_schedparam = 1
-def IN_CLASSA(a): return ((((in_addr_t)(a)) & (-2147483648)) == 0)
-
-IN_CLASSA_NET = (-16777216)
-IN_CLASSA_NSHIFT = 24
-IN_CLASSA_HOST = ((-1) & ~IN_CLASSA_NET)
-IN_CLASSA_MAX = 128
-def IN_CLASSB(a): return ((((in_addr_t)(a)) & (-1073741824)) == (-2147483648))
-
-IN_CLASSB_NET = (-65536)
-IN_CLASSB_NSHIFT = 16
-IN_CLASSB_HOST = ((-1) & ~IN_CLASSB_NET)
-IN_CLASSB_MAX = 65536
-def IN_CLASSC(a): return ((((in_addr_t)(a)) & (-536870912)) == (-1073741824))
-
-IN_CLASSC_NET = (-256)
-IN_CLASSC_NSHIFT = 8
-IN_CLASSC_HOST = ((-1) & ~IN_CLASSC_NET)
-def IN_CLASSD(a): return ((((in_addr_t)(a)) & (-268435456)) == (-536870912))
-
-def IN_MULTICAST(a): return IN_CLASSD(a)
-
-def IN_EXPERIMENTAL(a): return ((((in_addr_t)(a)) & (-536870912)) == (-536870912))
-
-def IN_BADCLASS(a): return ((((in_addr_t)(a)) & (-268435456)) == (-268435456))
-
-IN_LOOPBACKNET = 127
-INET_ADDRSTRLEN = 16
-INET6_ADDRSTRLEN = 46
-
-# Included from bits/socket.h
-
-# Included from limits.h
-_LIBC_LIMITS_H_ = 1
-MB_LEN_MAX = 16
-_LIMITS_H = 1
-CHAR_BIT = 8
-SCHAR_MIN = (-128)
-SCHAR_MAX = 127
-UCHAR_MAX = 255
-CHAR_MIN = 0
-CHAR_MAX = UCHAR_MAX
-CHAR_MIN = SCHAR_MIN
-CHAR_MAX = SCHAR_MAX
-SHRT_MIN = (-32768)
-SHRT_MAX = 32767
-USHRT_MAX = 65535
-INT_MAX = 2147483647
-LONG_MAX = 9223372036854775807
-LONG_MAX = 2147483647
-LONG_MIN = (-LONG_MAX - 1)
-
-# Included from bits/posix1_lim.h
-_BITS_POSIX1_LIM_H = 1
-_POSIX_AIO_LISTIO_MAX = 2
-_POSIX_AIO_MAX = 1
-_POSIX_ARG_MAX = 4096
-_POSIX_CHILD_MAX = 6
-_POSIX_DELAYTIMER_MAX = 32
-_POSIX_LINK_MAX = 8
-_POSIX_MAX_CANON = 255
-_POSIX_MAX_INPUT = 255
-_POSIX_MQ_OPEN_MAX = 8
-_POSIX_MQ_PRIO_MAX = 32
-_POSIX_NGROUPS_MAX = 0
-_POSIX_OPEN_MAX = 16
-_POSIX_FD_SETSIZE = _POSIX_OPEN_MAX
-_POSIX_NAME_MAX = 14
-_POSIX_PATH_MAX = 256
-_POSIX_PIPE_BUF = 512
-_POSIX_RTSIG_MAX = 8
-_POSIX_SEM_NSEMS_MAX = 256
-_POSIX_SEM_VALUE_MAX = 32767
-_POSIX_SIGQUEUE_MAX = 32
-_POSIX_SSIZE_MAX = 32767
-_POSIX_STREAM_MAX = 8
-_POSIX_TZNAME_MAX = 6
-_POSIX_QLIMIT = 1
-_POSIX_HIWAT = _POSIX_PIPE_BUF
-_POSIX_UIO_MAXIOV = 16
-_POSIX_TTY_NAME_MAX = 9
-_POSIX_TIMER_MAX = 32
-_POSIX_LOGIN_NAME_MAX = 9
-_POSIX_CLOCKRES_MIN = 20000000
-
-# Included from bits/local_lim.h
-
-# Included from linux/limits.h
-NR_OPEN = 1024
-NGROUPS_MAX = 32
-ARG_MAX = 131072
-CHILD_MAX = 999
-OPEN_MAX = 256
-LINK_MAX = 127
-MAX_CANON = 255
-MAX_INPUT = 255
-NAME_MAX = 255
-PATH_MAX = 4096
-PIPE_BUF = 4096
-RTSIG_MAX = 32
-_POSIX_THREAD_KEYS_MAX = 128
-PTHREAD_KEYS_MAX = 1024
-_POSIX_THREAD_DESTRUCTOR_ITERATIONS = 4
-PTHREAD_DESTRUCTOR_ITERATIONS = _POSIX_THREAD_DESTRUCTOR_ITERATIONS
-_POSIX_THREAD_THREADS_MAX = 64
-PTHREAD_THREADS_MAX = 1024
-AIO_PRIO_DELTA_MAX = 20
-PTHREAD_STACK_MIN = 16384
-TIMER_MAX = 256
-SSIZE_MAX = LONG_MAX
-NGROUPS_MAX = _POSIX_NGROUPS_MAX
-
-# Included from bits/posix2_lim.h
-_BITS_POSIX2_LIM_H = 1
-_POSIX2_BC_BASE_MAX = 99
-_POSIX2_BC_DIM_MAX = 2048
-_POSIX2_BC_SCALE_MAX = 99
-_POSIX2_BC_STRING_MAX = 1000
-_POSIX2_COLL_WEIGHTS_MAX = 2
-_POSIX2_EXPR_NEST_MAX = 32
-_POSIX2_LINE_MAX = 2048
-_POSIX2_RE_DUP_MAX = 255
-_POSIX2_CHARCLASS_NAME_MAX = 14
-BC_BASE_MAX = _POSIX2_BC_BASE_MAX
-BC_DIM_MAX = _POSIX2_BC_DIM_MAX
-BC_SCALE_MAX = _POSIX2_BC_SCALE_MAX
-BC_STRING_MAX = _POSIX2_BC_STRING_MAX
-COLL_WEIGHTS_MAX = 255
-EXPR_NEST_MAX = _POSIX2_EXPR_NEST_MAX
-LINE_MAX = _POSIX2_LINE_MAX
-CHARCLASS_NAME_MAX = 2048
-RE_DUP_MAX = (0x7fff)
-
-# Included from bits/xopen_lim.h
-_XOPEN_LIM_H = 1
-
-# Included from bits/stdio_lim.h
-L_tmpnam = 20
-TMP_MAX = 238328
-FILENAME_MAX = 4096
-L_ctermid = 9
-L_cuserid = 9
-FOPEN_MAX = 16
-IOV_MAX = 1024
-_XOPEN_IOV_MAX = _POSIX_UIO_MAXIOV
-NL_ARGMAX = _POSIX_ARG_MAX
-NL_LANGMAX = _POSIX2_LINE_MAX
-NL_MSGMAX = INT_MAX
-NL_NMAX = INT_MAX
-NL_SETMAX = INT_MAX
-NL_TEXTMAX = INT_MAX
-NZERO = 20
-WORD_BIT = 16
-WORD_BIT = 32
-WORD_BIT = 64
-WORD_BIT = 16
-WORD_BIT = 32
-WORD_BIT = 64
-WORD_BIT = 32
-LONG_BIT = 32
-LONG_BIT = 64
-LONG_BIT = 32
-LONG_BIT = 64
-LONG_BIT = 64
-LONG_BIT = 32
-from TYPES import *
-PF_UNSPEC = 0
-PF_LOCAL = 1
-PF_UNIX = PF_LOCAL
-PF_FILE = PF_LOCAL
-PF_INET = 2
-PF_AX25 = 3
-PF_IPX = 4
-PF_APPLETALK = 5
-PF_NETROM = 6
-PF_BRIDGE = 7
-PF_ATMPVC = 8
-PF_X25 = 9
-PF_INET6 = 10
-PF_ROSE = 11
-PF_DECnet = 12
-PF_NETBEUI = 13
-PF_SECURITY = 14
-PF_KEY = 15
-PF_NETLINK = 16
-PF_ROUTE = PF_NETLINK
-PF_PACKET = 17
-PF_ASH = 18
-PF_ECONET = 19
-PF_ATMSVC = 20
-PF_SNA = 22
-PF_IRDA = 23
-PF_PPPOX = 24
-PF_WANPIPE = 25
-PF_BLUETOOTH = 31
-PF_MAX = 32
-AF_UNSPEC = PF_UNSPEC
-AF_LOCAL = PF_LOCAL
-AF_UNIX = PF_UNIX
-AF_FILE = PF_FILE
-AF_INET = PF_INET
-AF_AX25 = PF_AX25
-AF_IPX = PF_IPX
-AF_APPLETALK = PF_APPLETALK
-AF_NETROM = PF_NETROM
-AF_BRIDGE = PF_BRIDGE
-AF_ATMPVC = PF_ATMPVC
-AF_X25 = PF_X25
-AF_INET6 = PF_INET6
-AF_ROSE = PF_ROSE
-AF_DECnet = PF_DECnet
-AF_NETBEUI = PF_NETBEUI
-AF_SECURITY = PF_SECURITY
-AF_KEY = PF_KEY
-AF_NETLINK = PF_NETLINK
-AF_ROUTE = PF_ROUTE
-AF_PACKET = PF_PACKET
-AF_ASH = PF_ASH
-AF_ECONET = PF_ECONET
-AF_ATMSVC = PF_ATMSVC
-AF_SNA = PF_SNA
-AF_IRDA = PF_IRDA
-AF_PPPOX = PF_PPPOX
-AF_WANPIPE = PF_WANPIPE
-AF_BLUETOOTH = PF_BLUETOOTH
-AF_MAX = PF_MAX
-SOL_RAW = 255
-SOL_DECNET = 261
-SOL_X25 = 262
-SOL_PACKET = 263
-SOL_ATM = 264
-SOL_AAL = 265
-SOL_IRDA = 266
-SOMAXCONN = 128
-
-# Included from bits/sockaddr.h
-_BITS_SOCKADDR_H = 1
-def __SOCKADDR_COMMON(sa_prefix): return \
-
-_SS_SIZE = 128
-def CMSG_FIRSTHDR(mhdr): return \
-
-
-# Included from asm/socket.h
-
-# Included from asm/sockios.h
-FIOSETOWN = 0x8901
-SIOCSPGRP = 0x8902
-FIOGETOWN = 0x8903
-SIOCGPGRP = 0x8904
-SIOCATMARK = 0x8905
-SIOCGSTAMP = 0x8906
-SOL_SOCKET = 1
-SO_DEBUG = 1
-SO_REUSEADDR = 2
-SO_TYPE = 3
-SO_ERROR = 4
-SO_DONTROUTE = 5
-SO_BROADCAST = 6
-SO_SNDBUF = 7
-SO_RCVBUF = 8
-SO_KEEPALIVE = 9
-SO_OOBINLINE = 10
-SO_NO_CHECK = 11
-SO_PRIORITY = 12
-SO_LINGER = 13
-SO_BSDCOMPAT = 14
-SO_PASSCRED = 16
-SO_PEERCRED = 17
-SO_RCVLOWAT = 18
-SO_SNDLOWAT = 19
-SO_RCVTIMEO = 20
-SO_SNDTIMEO = 21
-SO_SECURITY_AUTHENTICATION = 22
-SO_SECURITY_ENCRYPTION_TRANSPORT = 23
-SO_SECURITY_ENCRYPTION_NETWORK = 24
-SO_BINDTODEVICE = 25
-SO_ATTACH_FILTER = 26
-SO_DETACH_FILTER = 27
-SO_PEERNAME = 28
-SO_TIMESTAMP = 29
-SCM_TIMESTAMP = SO_TIMESTAMP
-SO_ACCEPTCONN = 30
-SOCK_STREAM = 1
-SOCK_DGRAM = 2
-SOCK_RAW = 3
-SOCK_RDM = 4
-SOCK_SEQPACKET = 5
-SOCK_PACKET = 10
-SOCK_MAX = (SOCK_PACKET+1)
-
-# Included from bits/in.h
-IP_TOS = 1
-IP_TTL = 2
-IP_HDRINCL = 3
-IP_OPTIONS = 4
-IP_ROUTER_ALERT = 5
-IP_RECVOPTS = 6
-IP_RETOPTS = 7
-IP_PKTINFO = 8
-IP_PKTOPTIONS = 9
-IP_PMTUDISC = 10
-IP_MTU_DISCOVER = 10
-IP_RECVERR = 11
-IP_RECVTTL = 12
-IP_RECVTOS = 13
-IP_MULTICAST_IF = 32
-IP_MULTICAST_TTL = 33
-IP_MULTICAST_LOOP = 34
-IP_ADD_MEMBERSHIP = 35
-IP_DROP_MEMBERSHIP = 36
-IP_RECVRETOPTS = IP_RETOPTS
-IP_PMTUDISC_DONT = 0
-IP_PMTUDISC_WANT = 1
-IP_PMTUDISC_DO = 2
-SOL_IP = 0
-IP_DEFAULT_MULTICAST_TTL = 1
-IP_DEFAULT_MULTICAST_LOOP = 1
-IP_MAX_MEMBERSHIPS = 20
-IPV6_ADDRFORM = 1
-IPV6_PKTINFO = 2
-IPV6_HOPOPTS = 3
-IPV6_DSTOPTS = 4
-IPV6_RTHDR = 5
-IPV6_PKTOPTIONS = 6
-IPV6_CHECKSUM = 7
-IPV6_HOPLIMIT = 8
-IPV6_NEXTHOP = 9
-IPV6_AUTHHDR = 10
-IPV6_UNICAST_HOPS = 16
-IPV6_MULTICAST_IF = 17
-IPV6_MULTICAST_HOPS = 18
-IPV6_MULTICAST_LOOP = 19
-IPV6_JOIN_GROUP = 20
-IPV6_LEAVE_GROUP = 21
-IPV6_ROUTER_ALERT = 22
-IPV6_MTU_DISCOVER = 23
-IPV6_MTU = 24
-IPV6_RECVERR = 25
-IPV6_RXHOPOPTS = IPV6_HOPOPTS
-IPV6_RXDSTOPTS = IPV6_DSTOPTS
-IPV6_ADD_MEMBERSHIP = IPV6_JOIN_GROUP
-IPV6_DROP_MEMBERSHIP = IPV6_LEAVE_GROUP
-IPV6_PMTUDISC_DONT = 0
-IPV6_PMTUDISC_WANT = 1
-IPV6_PMTUDISC_DO = 2
-SOL_IPV6 = 41
-SOL_ICMPV6 = 58
-IPV6_RTHDR_LOOSE = 0
-IPV6_RTHDR_STRICT = 1
-IPV6_RTHDR_TYPE_0 = 0
-
-# Included from endian.h
-_ENDIAN_H = 1
-__LITTLE_ENDIAN = 1234
-__BIG_ENDIAN = 4321
-__PDP_ENDIAN = 3412
-
-# Included from bits/endian.h
-__BYTE_ORDER = __LITTLE_ENDIAN
-__FLOAT_WORD_ORDER = __BYTE_ORDER
-LITTLE_ENDIAN = __LITTLE_ENDIAN
-BIG_ENDIAN = __BIG_ENDIAN
-PDP_ENDIAN = __PDP_ENDIAN
-BYTE_ORDER = __BYTE_ORDER
-
-# Included from bits/byteswap.h
-_BITS_BYTESWAP_H = 1
-def __bswap_constant_16(x): return \
-
-def __bswap_16(x): return \
-
-def __bswap_16(x): return __bswap_constant_16 (x)
-
-def __bswap_constant_32(x): return \
-
-def __bswap_32(x): return \
-
-def __bswap_32(x): return \
-
-def __bswap_32(x): return __bswap_constant_32 (x)
-
-def __bswap_constant_64(x): return \
-
-def __bswap_64(x): return \
-
-def ntohl(x): return (x)
-
-def ntohs(x): return (x)
-
-def htonl(x): return (x)
-
-def htons(x): return (x)
-
-def ntohl(x): return __bswap_32 (x)
-
-def ntohs(x): return __bswap_16 (x)
-
-def htonl(x): return __bswap_32 (x)
-
-def htons(x): return __bswap_16 (x)
-
-def IN6_IS_ADDR_UNSPECIFIED(a): return \
-
-def IN6_IS_ADDR_LOOPBACK(a): return \
-
-def IN6_IS_ADDR_LINKLOCAL(a): return \
-
-def IN6_IS_ADDR_SITELOCAL(a): return \
-
-def IN6_IS_ADDR_V4MAPPED(a): return \
-
-def IN6_IS_ADDR_V4COMPAT(a): return \
-
-def IN6_IS_ADDR_MC_NODELOCAL(a): return \
-
-def IN6_IS_ADDR_MC_LINKLOCAL(a): return \
-
-def IN6_IS_ADDR_MC_SITELOCAL(a): return \
-
-def IN6_IS_ADDR_MC_ORGLOCAL(a): return \
-
-def IN6_IS_ADDR_MC_GLOBAL(a): return
diff --git a/Lib/plat-linux3/TYPES.py b/Lib/plat-linux3/TYPES.py
deleted file mode 100644
--- a/Lib/plat-linux3/TYPES.py
+++ /dev/null
@@ -1,170 +0,0 @@
-# Generated by h2py from /usr/include/sys/types.h
-_SYS_TYPES_H = 1
-
-# Included from features.h
-_FEATURES_H = 1
-__USE_ANSI = 1
-__FAVOR_BSD = 1
-_ISOC99_SOURCE = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 199506
-_XOPEN_SOURCE = 600
-_XOPEN_SOURCE_EXTENDED = 1
-_LARGEFILE64_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-_BSD_SOURCE = 1
-_SVID_SOURCE = 1
-__USE_ISOC99 = 1
-_POSIX_SOURCE = 1
-_POSIX_C_SOURCE = 2
-_POSIX_C_SOURCE = 199506
-__USE_POSIX = 1
-__USE_POSIX2 = 1
-__USE_POSIX199309 = 1
-__USE_POSIX199506 = 1
-__USE_XOPEN = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_UNIX98 = 1
-_LARGEFILE_SOURCE = 1
-__USE_XOPEN2K = 1
-__USE_ISOC99 = 1
-__USE_XOPEN_EXTENDED = 1
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_FILE_OFFSET64 = 1
-__USE_MISC = 1
-__USE_BSD = 1
-__USE_SVID = 1
-__USE_GNU = 1
-__USE_REENTRANT = 1
-__STDC_IEC_559__ = 1
-__STDC_IEC_559_COMPLEX__ = 1
-__STDC_ISO_10646__ = 200009
-__GNU_LIBRARY__ = 6
-__GLIBC__ = 2
-__GLIBC_MINOR__ = 2
-
-# Included from sys/cdefs.h
-_SYS_CDEFS_H = 1
-def __PMT(args): return args
-
-def __P(args): return args
-
-def __PMT(args): return args
-
-def __STRING(x): return #x
-
-__flexarr = []
-__flexarr = [0]
-__flexarr = []
-__flexarr = [1]
-def __ASMNAME(cname): return __ASMNAME2 (__USER_LABEL_PREFIX__, cname)
-
-def __attribute__(xyz): return
-
-def __attribute_format_arg__(x): return __attribute__ ((__format_arg__ (x)))
-
-def __attribute_format_arg__(x): return
-
-__USE_LARGEFILE = 1
-__USE_LARGEFILE64 = 1
-__USE_EXTERN_INLINES = 1
-
-# Included from gnu/stubs.h
-
-# Included from bits/types.h
-_BITS_TYPES_H = 1
-__FD_SETSIZE = 1024
-
-# Included from bits/pthreadtypes.h
-_BITS_PTHREADTYPES_H = 1
-
-# Included from bits/sched.h
-SCHED_OTHER = 0
-SCHED_FIFO = 1
-SCHED_RR = 2
-CSIGNAL = 0x000000ff
-CLONE_VM = 0x00000100
-CLONE_FS = 0x00000200
-CLONE_FILES = 0x00000400
-CLONE_SIGHAND = 0x00000800
-CLONE_PID = 0x00001000
-CLONE_PTRACE = 0x00002000
-CLONE_VFORK = 0x00004000
-__defined_schedparam = 1
-
-# Included from time.h
-_TIME_H = 1
-
-# Included from bits/time.h
-_BITS_TIME_H = 1
-CLOCKS_PER_SEC = 1000000
-CLOCK_REALTIME = 0
-CLOCK_PROCESS_CPUTIME_ID = 2
-CLOCK_THREAD_CPUTIME_ID = 3
-TIMER_ABSTIME = 1
-_STRUCT_TIMEVAL = 1
-CLK_TCK = CLOCKS_PER_SEC
-__clock_t_defined = 1
-__time_t_defined = 1
-__clockid_t_defined = 1
-__timer_t_defined = 1
-__timespec_defined = 1
-def __isleap(year): return \
-
-__BIT_TYPES_DEFINED__ = 1
-
-# Included from endian.h
-_ENDIAN_H = 1
-__LITTLE_ENDIAN = 1234
-__BIG_ENDIAN = 4321
-__PDP_ENDIAN = 3412
-
-# Included from bits/endian.h
-__BYTE_ORDER = __LITTLE_ENDIAN
-__FLOAT_WORD_ORDER = __BYTE_ORDER
-LITTLE_ENDIAN = __LITTLE_ENDIAN
-BIG_ENDIAN = __BIG_ENDIAN
-PDP_ENDIAN = __PDP_ENDIAN
-BYTE_ORDER = __BYTE_ORDER
-
-# Included from sys/select.h
-_SYS_SELECT_H = 1
-
-# Included from bits/select.h
-def __FD_ZERO(fdsp): return \
-
-def __FD_ZERO(set): return \
-
-
-# Included from bits/sigset.h
-_SIGSET_H_types = 1
-_SIGSET_H_fns = 1
-def __sigmask(sig): return \
-
-def __sigemptyset(set): return \
-
-def __sigfillset(set): return \
-
-def __sigisemptyset(set): return \
-
-def __FDELT(d): return ((d) / __NFDBITS)
-
-FD_SETSIZE = __FD_SETSIZE
-def FD_ZERO(fdsetp): return __FD_ZERO (fdsetp)
-
-
-# Included from sys/sysmacros.h
-_SYS_SYSMACROS_H = 1
-def major(dev): return ((int)(((dev) >> 8) & 0xff))
-
-def minor(dev): return ((int)((dev) & 0xff))
-
-def major(dev): return (((dev).__val[1] >> 8) & 0xff)
-
-def minor(dev): return ((dev).__val[1] & 0xff)
-
-def major(dev): return (((dev).__val[0] >> 8) & 0xff)
-
-def minor(dev): return ((dev).__val[0] & 0xff)
diff --git a/Lib/plat-linux3/regen b/Lib/plat-linux3/regen
deleted file mode 100755
--- a/Lib/plat-linux3/regen
+++ /dev/null
@@ -1,8 +0,0 @@
-#! /bin/sh
-case `uname` in
-Linux*)	;;
-*)	echo Probably not on a Linux system 1>&2
-	exit 1;;
-esac
-set -v
-h2py -i '(u_long)' /usr/include/sys/types.h /usr/include/netinet/in.h /usr/include/dlfcn.h
diff --git a/Lib/tarfile.py b/Lib/tarfile.py
--- a/Lib/tarfile.py
+++ b/Lib/tarfile.py
@@ -2368,17 +2368,11 @@
             try:
                 g = grp.getgrnam(tarinfo.gname)[2]
             except KeyError:
-                try:
-                    g = grp.getgrgid(tarinfo.gid)[2]
-                except KeyError:
-                    g = os.getgid()
+                g = tarinfo.gid
             try:
                 u = pwd.getpwnam(tarinfo.uname)[2]
             except KeyError:
-                try:
-                    u = pwd.getpwuid(tarinfo.uid)[2]
-                except KeyError:
-                    u = os.getuid()
+                u = tarinfo.uid
             try:
                 if tarinfo.issym() and hasattr(os, "lchown"):
                     os.lchown(targetpath, u, g)
diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -377,7 +377,8 @@
         p = html.parser.HTMLParser()
         self.assertEqual(p.unescape('&#bad;'),'&#bad;')
         self.assertEqual(p.unescape('&#0038;'),'&')
-
+        # see #12888
+        self.assertEqual(p.unescape('&#123; ' * 1050), '{ ' * 1050)
 
 def test_main():
     support.run_unittest(HTMLParserTestCase, HTMLParserTolerantTestCase)
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -661,6 +661,7 @@
 Michele Orr?
 Oleg Oshmyan
 Denis S. Otkidach
+Peter Otten
 Michael Otteneder
 R. M. Oudkerk
 Russel Owen
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -25,6 +25,12 @@
 Library
 -------
 
+- Issue #9561: distutils now reads and writes egg-info files using UTF-8,
+  instead of the locale encoding.
+
+- Issue #12888: Fix a bug in HTMLParser.unescape that prevented it to escape
+  more than 128 entities.  Patch by Peter Otten.
+
 - Issue #12878: Expose a __dict__ attribute on io.IOBase and its subclasses.
 
 - Issue #12636: IDLE reads the coding cookie when executing a Python script.
@@ -70,6 +76,14 @@
 Library
 -------
 
+- Issue #8286: The distutils command sdist will print a warning message instead
+  of crashing when an invalid path is given in the manifest template.
+
+- Issue #12841: tarfile unnecessarily checked the existence of numerical user
+  and group ids on extraction. If one of them did not exist the respective id
+  of the current user (i.e. root) was used for the file and ownership
+  information was lost.
+
 - Issue #10946: The distutils commands bdist_dumb, bdist_wininst and bdist_msi
   now respect a --skip-build option given to bdist.
 
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -1187,12 +1187,12 @@
 /* Helper function for PyUnicode_AsWideChar() and PyUnicode_AsWideCharString():
    convert a Unicode object to a wide character string.
 
-   - If w is NULL: return the number of wide characters (including the nul
+   - If w is NULL: return the number of wide characters (including the null
      character) required to convert the unicode object. Ignore size argument.
 
-   - Otherwise: return the number of wide characters (excluding the nul
+   - Otherwise: return the number of wide characters (excluding the null
      character) written into w. Write at most size wide characters (including
-     the nul character). */
+     the null character). */
 static Py_ssize_t
 unicode_aswidechar(PyUnicodeObject *unicode,
                    wchar_t *w,
@@ -1240,7 +1240,7 @@
         return w - worig;
     }
     else {
-        nchar = 1; /* nul character at the end */
+        nchar = 1; /* null character at the end */
         while (u != uend) {
             if (0xD800 <= u[0] && u[0] <= 0xDBFF
                 && 0xDC00 <= u[1] && u[1] <= 0xDFFF)
@@ -1278,7 +1278,7 @@
         return w - worig;
     }
     else {
-        nchar = 1; /* nul character */
+        nchar = 1; /* null character */
         while (u != uend) {
             if (*u > 0xffff)
                 nchar += 2;
diff --git a/Tools/msi/uuids.py b/Tools/msi/uuids.py
--- a/Tools/msi/uuids.py
+++ b/Tools/msi/uuids.py
@@ -90,4 +90,9 @@
     '3.2.1121':'{4f90de4a-83dd-4443-b625-ca130ff361dd}', # 3.2.1rc1
     '3.2.1122':'{dc5eb04d-ff8a-4bed-8f96-23942fd59e5f}', # 3.2.1rc2
     '3.2.1150':'{34b2530c-6349-4292-9dc3-60bda4aed93c}', # 3.2.1
+    '3.2.2121':'{DFB29A53-ACC4-44e6-85A6-D0DA26FE8E4E}', # 3.2.2rc1
+    '3.2.2150':'{4CDE3168-D060-4b7c-BC74-4D8F9BB01AFD}', # 3.2.2
+    '3.2.3121':'{B8E8CFF7-E4C6-4a7c-9F06-BB3A8B75DDA8}', # 3.2.3rc1
+    '3.2.3150':'{789C9644-9F82-44d3-B4CA-AC31F46F5882}', # 3.2.3
+
 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep  6 13:57:50 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Tue, 06 Sep 2011 13:57:50 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_merge_heads?=
Message-ID: <E1R0uHm-0001Sv-P6@dinsdale.python.org>

http://hg.python.org/cpython/rev/48808bbf662d
changeset:   72305:48808bbf662d
parent:      72303:f24352b0df86
parent:      72301:786668a4fb6b
user:        Benjamin Peterson <benjamin at python.org>
date:        Tue Sep 06 07:57:38 2011 -0400
summary:
  merge heads

files:
  Doc/library/curses.rst                    |  11 +
  Doc/library/multiprocessing.rst           |  12 +-
  Doc/packaging/setupcfg.rst                |   2 +-
  Include/unicodeobject.h                   |   2 +-
  Lib/distutils/command/install_egg_info.py |   5 +-
  Lib/distutils/command/sdist.py            |   5 +-
  Lib/distutils/dist.py                     |   6 +-
  Lib/distutils/tests/test_sdist.py         |  37 +++++-
  Lib/html/parser.py                        |   2 +-
  Lib/packaging/tests/test_util.py          |   4 +-
  Lib/tarfile.py                            |  10 +-
  Lib/test/test_curses.py                   |  17 ++
  Lib/test/test_htmlparser.py               |   3 +-
  Misc/ACKS                                 |   1 +
  Misc/NEWS                                 |  17 ++
  Modules/_cursesmodule.c                   |  68 +++++++++++
  Objects/unicodeobject.c                   |  10 +-
  Tools/msi/uuids.py                        |   5 +
  18 files changed, 178 insertions(+), 39 deletions(-)


diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst
--- a/Doc/library/curses.rst
+++ b/Doc/library/curses.rst
@@ -598,6 +598,17 @@
       Only one *ch* can be pushed before :meth:`getch` is called.
 
 
+.. function:: unget_wch(ch)
+
+   Push *ch* so the next :meth:`get_wch` will return it.
+
+   .. note::
+
+      Only one *ch* can be pushed before :meth:`get_wch` is called.
+
+   .. versionadded:: 3.3
+
+
 .. function:: ungetmouse(id, x, y, z, bstate)
 
    Push a :const:`KEY_MOUSE` event onto the input queue, associating the given
diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst
--- a/Doc/library/multiprocessing.rst
+++ b/Doc/library/multiprocessing.rst
@@ -571,9 +571,9 @@
       Return ``True`` if the queue is full, ``False`` otherwise.  Because of
       multithreading/multiprocessing semantics, this is not reliable.
 
-   .. method:: put(item[, block[, timeout]])
-
-      Put item into the queue.  If the optional argument *block* is ``True``
+   .. method:: put(obj[, block[, timeout]])
+
+      Put obj into the queue.  If the optional argument *block* is ``True``
       (the default) and *timeout* is ``None`` (the default), block if necessary until
       a free slot is available.  If *timeout* is a positive number, it blocks at
       most *timeout* seconds and raises the :exc:`queue.Full` exception if no
@@ -582,9 +582,9 @@
       available, else raise the :exc:`queue.Full` exception (*timeout* is
       ignored in that case).
 
-   .. method:: put_nowait(item)
-
-      Equivalent to ``put(item, False)``.
+   .. method:: put_nowait(obj)
+
+      Equivalent to ``put(obj, False)``.
 
    .. method:: get([block[, timeout]])
 
diff --git a/Doc/packaging/setupcfg.rst b/Doc/packaging/setupcfg.rst
--- a/Doc/packaging/setupcfg.rst
+++ b/Doc/packaging/setupcfg.rst
@@ -72,7 +72,7 @@
 ---------------
 
 A configuration file can be extended (i.e. included) by other files.  For this,
-a ``DEFAULT`` section must contain an ``extends`` key which value points to one
+a ``DEFAULT`` section must contain an ``extends`` key whose value points to one
 or more files which will be merged into the current files by adding new sections
 and fields.  If a file loaded by ``extends`` contains sections or keys that
 already exist in the original file, they will not override the previous values.
diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -608,7 +608,7 @@
 
 /* Convert the Unicode object to a wide character string. The output string
    always ends with a nul character. If size is not NULL, write the number of
-   wide characters (including the nul character) into *size.
+   wide characters (excluding the null character) into *size.
 
    Returns a buffer allocated by PyMem_Alloc() (use PyMem_Free() to free it)
    on success. On error, returns NULL, *size is undefined and raises a
diff --git a/Lib/distutils/command/install_egg_info.py b/Lib/distutils/command/install_egg_info.py
--- a/Lib/distutils/command/install_egg_info.py
+++ b/Lib/distutils/command/install_egg_info.py
@@ -40,9 +40,8 @@
                          "Creating "+self.install_dir)
         log.info("Writing %s", target)
         if not self.dry_run:
-            f = open(target, 'w')
-            self.distribution.metadata.write_pkg_file(f)
-            f.close()
+            with open(target, 'w', encoding='UTF-8') as f:
+                self.distribution.metadata.write_pkg_file(f)
 
     def get_outputs(self):
         return self.outputs
diff --git a/Lib/distutils/command/sdist.py b/Lib/distutils/command/sdist.py
--- a/Lib/distutils/command/sdist.py
+++ b/Lib/distutils/command/sdist.py
@@ -306,7 +306,10 @@
 
                 try:
                     self.filelist.process_template_line(line)
-                except DistutilsTemplateError as msg:
+                # the call above can raise a DistutilsTemplateError for
+                # malformed lines, or a ValueError from the lower-level
+                # convert_path function
+                except (DistutilsTemplateError, ValueError) as msg:
                     self.warn("%s, line %d: %s" % (template.filename,
                                                    template.current_line,
                                                    msg))
diff --git a/Lib/distutils/dist.py b/Lib/distutils/dist.py
--- a/Lib/distutils/dist.py
+++ b/Lib/distutils/dist.py
@@ -1010,11 +1010,9 @@
     def write_pkg_info(self, base_dir):
         """Write the PKG-INFO file into the release tree.
         """
-        pkg_info = open(os.path.join(base_dir, 'PKG-INFO'), 'w')
-        try:
+        with open(os.path.join(base_dir, 'PKG-INFO'), 'w',
+                  encoding='UTF-8') as pkg_info:
             self.write_pkg_file(pkg_info)
-        finally:
-            pkg_info.close()
 
     def write_pkg_file(self, file):
         """Write the PKG-INFO format data to a file object.
diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -15,6 +15,7 @@
 from distutils.errors import DistutilsOptionError
 from distutils.spawn import find_executable
 from distutils.log import WARN
+from distutils.filelist import FileList
 from distutils.archive_util import ARCHIVE_FORMATS
 
 SETUP_PY = """
@@ -78,9 +79,6 @@
         dist.include_package_data = True
         cmd = sdist(dist)
         cmd.dist_dir = 'dist'
-        def _warn(*args):
-            pass
-        cmd.warn = _warn
         return dist, cmd
 
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
@@ -235,7 +233,8 @@
         # with the `check` subcommand
         cmd.ensure_finalized()
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 2)
 
         # trying with a complete set of metadata
@@ -244,7 +243,8 @@
         cmd.ensure_finalized()
         cmd.metadata_check = 0
         cmd.run()
-        warnings = self.get_logs(WARN)
+        warnings = [msg for msg in self.get_logs(WARN) if
+                    msg.startswith('warning: check:')]
         self.assertEqual(len(warnings), 0)
 
     def test_check_metadata_deprecated(self):
@@ -266,7 +266,6 @@
         self.assertEqual(len(output), num_formats)
 
     def test_finalize_options(self):
-
         dist, cmd = self.get_cmd()
         cmd.finalize_options()
 
@@ -286,6 +285,32 @@
         cmd.formats = 'supazipa'
         self.assertRaises(DistutilsOptionError, cmd.finalize_options)
 
+    # the following tests make sure there is a nice error message instead
+    # of a traceback when parsing an invalid manifest template
+
+    def _test_template(self, content):
+        dist, cmd = self.get_cmd()
+        os.chdir(self.tmp_dir)
+        self.write_file('MANIFEST.in', content)
+        cmd.ensure_finalized()
+        cmd.filelist = FileList()
+        cmd.read_template()
+        warnings = self.get_logs(WARN)
+        self.assertEqual(len(warnings), 1)
+
+    def test_invalid_template_unknown_command(self):
+        self._test_template('taunt knights *')
+
+    def test_invalid_template_wrong_arguments(self):
+        # this manifest command takes one argument
+        self._test_template('prune')
+
+    @unittest.skipIf(os.name != 'nt', 'test relevant for Windows only')
+    def test_invalid_template_wrong_path(self):
+        # on Windows, trailing slashes are not allowed
+        # this used to crash instead of raising a warning: #8286
+        self._test_template('include examples/')
+
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
     def test_get_file_list(self):
         # make sure MANIFEST is recalculated
diff --git a/Lib/html/parser.py b/Lib/html/parser.py
--- a/Lib/html/parser.py
+++ b/Lib/html/parser.py
@@ -458,4 +458,4 @@
                     return '&'+s+';'
 
         return re.sub(r"&(#?[xX]?(?:[0-9a-fA-F]+|\w{1,8}));",
-                      replaceEntities, s, re.ASCII)
+                      replaceEntities, s, flags=re.ASCII)
diff --git a/Lib/packaging/tests/test_util.py b/Lib/packaging/tests/test_util.py
--- a/Lib/packaging/tests/test_util.py
+++ b/Lib/packaging/tests/test_util.py
@@ -946,7 +946,7 @@
 
     def _distutils_pkg_info(self):
         tmp = self._distutils_setup_py_pkg()
-        self.write_file([tmp, 'PKG-INFO'], '')
+        self.write_file([tmp, 'PKG-INFO'], '', encoding='UTF-8')
         return tmp
 
     def _setup_cfg_with_no_metadata_pkg(self):
@@ -971,7 +971,7 @@
 
     def _pkg_info_with_no_distutils(self):
         tmp = self._random_setup_py_pkg()
-        self.write_file([tmp, 'PKG-INFO'], '')
+        self.write_file([tmp, 'PKG-INFO'], '', encoding='UTF-8')
         return tmp
 
     def _random_setup_py_pkg(self):
diff --git a/Lib/tarfile.py b/Lib/tarfile.py
--- a/Lib/tarfile.py
+++ b/Lib/tarfile.py
@@ -2366,17 +2366,11 @@
             try:
                 g = grp.getgrnam(tarinfo.gname)[2]
             except KeyError:
-                try:
-                    g = grp.getgrgid(tarinfo.gid)[2]
-                except KeyError:
-                    g = os.getgid()
+                g = tarinfo.gid
             try:
                 u = pwd.getpwnam(tarinfo.uname)[2]
             except KeyError:
-                try:
-                    u = pwd.getpwuid(tarinfo.uid)[2]
-                except KeyError:
-                    u = os.getuid()
+                u = tarinfo.uid
             try:
                 if tarinfo.issym() and hasattr(os, "lchown"):
                     os.lchown(targetpath, u, g)
diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py
--- a/Lib/test/test_curses.py
+++ b/Lib/test/test_curses.py
@@ -264,6 +264,22 @@
     curses.ungetch(1025)
     stdscr.getkey()
 
+def test_unget_wch(stdscr):
+    if not hasattr(curses, 'unget_wch'):
+        return
+    ch = 'a'
+    curses.unget_wch(ch)
+    read = stdscr.get_wch()
+    read = chr(read)
+    if read != ch:
+        raise AssertionError("%r != %r" % (read, ch))
+
+    ch = ord('a')
+    curses.unget_wch(ch)
+    read = stdscr.get_wch()
+    if read != ch:
+        raise AssertionError("%r != %r" % (read, ch))
+
 def main(stdscr):
     curses.savetty()
     try:
@@ -272,6 +288,7 @@
         test_userptr_without_set(stdscr)
         test_resize_term(stdscr)
         test_issue6243(stdscr)
+        test_unget_wch(stdscr)
     finally:
         curses.resetty()
 
diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -377,7 +377,8 @@
         p = html.parser.HTMLParser()
         self.assertEqual(p.unescape('&#bad;'),'&#bad;')
         self.assertEqual(p.unescape('&#0038;'),'&')
-
+        # see #12888
+        self.assertEqual(p.unescape('&#123; ' * 1050), '{ ' * 1050)
 
 def test_main():
     support.run_unittest(HTMLParserTestCase, HTMLParserTolerantTestCase)
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -704,6 +704,7 @@
 Michele Orr?
 Oleg Oshmyan
 Denis S. Otkidach
+Peter Otten
 Michael Otteneder
 R. M. Oudkerk
 Russel Owen
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -271,6 +271,23 @@
 Library
 -------
 
+- Issue #12567: Add curses.unget_wch() function. Push a character so the next
+  get_wch() will return it.
+
+- Issue #9561: distutils and packaging now writes egg-info files using UTF-8,
+  instead of the locale encoding.
+
+- Issue #8286: The distutils command sdist will print a warning message instead
+  of crashing when an invalid path is given in the manifest template.
+
+- Issue #12841: tarfile unnecessarily checked the existence of numerical user
+  and group ids on extraction. If one of them did not exist the respective id
+  of the current user (i.e. root) was used for the file and ownership
+  information was lost.
+
+- Issue #12888: Fix a bug in HTMLParser.unescape that prevented it to escape
+  more than 128 entities.  Patch by Peter Otten.
+
 - Issue #12878: Expose a __dict__ attribute on io.IOBase and its subclasses.
 
 - Issue #12636: IDLE reads the coding cookie when executing a Python script.
diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c
--- a/Modules/_cursesmodule.c
+++ b/Modules/_cursesmodule.c
@@ -2696,6 +2696,71 @@
     return PyCursesCheckERR(ungetch(ch), "ungetch");
 }
 
+#ifdef HAVE_NCURSESW
+/* Convert an object to a character (wchar_t):
+
+    - int
+    - str of length 1
+
+   Return 1 on success, 0 on error. */
+static int
+PyCurses_ConvertToWchar_t(PyObject *obj,
+                          wchar_t *wch)
+{
+    if (PyUnicode_Check(obj)) {
+        wchar_t buffer[2];
+        if (PyUnicode_AsWideChar(obj, buffer, 2) != 1) {
+            PyErr_Format(PyExc_TypeError,
+                         "expect bytes or str of length 1, or int, "
+                         "got a str of length %zi",
+                         PyUnicode_GET_SIZE(obj));
+            return 0;
+        }
+        *wch = buffer[0];
+        return 2;
+    }
+    else if (PyLong_CheckExact(obj)) {
+        long value;
+        int overflow;
+        value = PyLong_AsLongAndOverflow(obj, &overflow);
+        if (overflow) {
+            PyErr_SetString(PyExc_OverflowError,
+                            "int doesn't fit in long");
+            return 0;
+        }
+        *wch = (wchar_t)value;
+        if ((long)*wch != value) {
+            PyErr_Format(PyExc_OverflowError,
+                         "character doesn't fit in wchar_t");
+            return 0;
+        }
+        return 1;
+    }
+    else {
+        PyErr_Format(PyExc_TypeError,
+                     "expect bytes or str of length 1, or int, got %s",
+                     Py_TYPE(obj)->tp_name);
+        return 0;
+    }
+}
+
+static PyObject *
+PyCurses_Unget_Wch(PyObject *self, PyObject *args)
+{
+    PyObject *obj;
+    wchar_t wch;
+
+    PyCursesInitialised;
+
+    if (!PyArg_ParseTuple(args,"O", &obj))
+        return NULL;
+
+    if (!PyCurses_ConvertToWchar_t(obj, &wch))
+        return NULL;
+    return PyCursesCheckERR(unget_wch(wch), "unget_wch");
+}
+#endif
+
 static PyObject *
 PyCurses_Use_Env(PyObject *self, PyObject *args)
 {
@@ -2823,6 +2888,9 @@
     {"typeahead",           (PyCFunction)PyCurses_TypeAhead, METH_VARARGS},
     {"unctrl",              (PyCFunction)PyCurses_UnCtrl, METH_VARARGS},
     {"ungetch",             (PyCFunction)PyCurses_UngetCh, METH_VARARGS},
+#ifdef HAVE_NCURSESW
+    {"unget_wch",           (PyCFunction)PyCurses_Unget_Wch, METH_VARARGS},
+#endif
     {"use_env",             (PyCFunction)PyCurses_Use_Env, METH_VARARGS},
 #ifndef STRICT_SYSV_CURSES
     {"use_default_colors",  (PyCFunction)PyCurses_Use_Default_Colors, METH_NOARGS},
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -1204,12 +1204,12 @@
 /* Helper function for PyUnicode_AsWideChar() and PyUnicode_AsWideCharString():
    convert a Unicode object to a wide character string.
 
-   - If w is NULL: return the number of wide characters (including the nul
+   - If w is NULL: return the number of wide characters (including the null
      character) required to convert the unicode object. Ignore size argument.
 
-   - Otherwise: return the number of wide characters (excluding the nul
+   - Otherwise: return the number of wide characters (excluding the null
      character) written into w. Write at most size wide characters (including
-     the nul character). */
+     the null character). */
 static Py_ssize_t
 unicode_aswidechar(PyUnicodeObject *unicode,
                    wchar_t *w,
@@ -1257,7 +1257,7 @@
         return w - worig;
     }
     else {
-        nchar = 1; /* nul character at the end */
+        nchar = 1; /* null character at the end */
         while (u != uend) {
             if (0xD800 <= u[0] && u[0] <= 0xDBFF
                 && 0xDC00 <= u[1] && u[1] <= 0xDFFF)
@@ -1295,7 +1295,7 @@
         return w - worig;
     }
     else {
-        nchar = 1; /* nul character */
+        nchar = 1; /* null character */
         while (u != uend) {
             if (*u > 0xffff)
                 nchar += 2;
diff --git a/Tools/msi/uuids.py b/Tools/msi/uuids.py
--- a/Tools/msi/uuids.py
+++ b/Tools/msi/uuids.py
@@ -30,4 +30,9 @@
     '3.2.1121':'{4f90de4a-83dd-4443-b625-ca130ff361dd}', # 3.2.1rc1
     '3.2.1122':'{dc5eb04d-ff8a-4bed-8f96-23942fd59e5f}', # 3.2.1rc2
     '3.2.1150':'{34b2530c-6349-4292-9dc3-60bda4aed93c}', # 3.2.1
+    '3.2.2121':'{DFB29A53-ACC4-44e6-85A6-D0DA26FE8E4E}', # 3.2.2rc1
+    '3.2.2150':'{4CDE3168-D060-4b7c-BC74-4D8F9BB01AFD}', # 3.2.2
+    '3.2.3121':'{B8E8CFF7-E4C6-4a7c-9F06-BB3A8B75DDA8}', # 3.2.3rc1
+    '3.2.3150':'{789C9644-9F82-44d3-B4CA-AC31F46F5882}', # 3.2.3
+
 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep  6 13:57:51 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Tue, 06 Sep 2011 13:57:51 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_merge_3=2E2?=
Message-ID: <E1R0uHn-0001T3-Il@dinsdale.python.org>

http://hg.python.org/cpython/rev/6cd2086c39c1
changeset:   72306:6cd2086c39c1
parent:      72305:48808bbf662d
parent:      72304:a0628933a2f9
user:        Benjamin Peterson <benjamin at python.org>
date:        Tue Sep 06 07:57:43 2011 -0400
summary:
  merge 3.2

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep  6 15:08:37 2011
From: python-checkins at python.org (vinay.sajip)
Date: Tue, 06 Sep 2011 15:08:37 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Closes_=2312906?=
 =?utf8?q?=3A_Fixed_bug_in_YAML_configuration=2E?=
Message-ID: <E1R0vOH-0005cA-V6@dinsdale.python.org>

http://hg.python.org/cpython/rev/299ea19c3197
changeset:   72307:299ea19c3197
branch:      2.7
parent:      72293:0fe571d43317
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Tue Sep 06 14:06:24 2011 +0100
summary:
  Closes #12906: Fixed bug in YAML configuration.

files:
  Doc/howto/logging.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst
--- a/Doc/howto/logging.rst
+++ b/Doc/howto/logging.rst
@@ -670,7 +670,7 @@
     version: 1
     formatters:
       simple:
-        format: format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
+        format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
     handlers:
       console:
         class: logging.StreamHandler

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep  6 15:08:38 2011
From: python-checkins at python.org (vinay.sajip)
Date: Tue, 06 Sep 2011 15:08:38 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Closes_=2312906?=
 =?utf8?q?=3A_Fixed_bug_in_YAML_configuration=2E?=
Message-ID: <E1R0vOI-0005cE-OQ@dinsdale.python.org>

http://hg.python.org/cpython/rev/cf811943046b
changeset:   72308:cf811943046b
branch:      3.2
parent:      72304:a0628933a2f9
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Tue Sep 06 14:07:23 2011 +0100
summary:
  Closes #12906: Fixed bug in YAML configuration.

files:
  Doc/howto/logging.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst
--- a/Doc/howto/logging.rst
+++ b/Doc/howto/logging.rst
@@ -679,7 +679,7 @@
     version: 1
     formatters:
       simple:
-        format: format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
+        format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
     handlers:
       console:
         class: logging.StreamHandler

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep  6 15:08:39 2011
From: python-checkins at python.org (vinay.sajip)
Date: Tue, 06 Sep 2011 15:08:39 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Closes_=2312906=3A_Merged_fix_from_3=2E2=2E?=
Message-ID: <E1R0vOJ-0005cM-HS@dinsdale.python.org>

http://hg.python.org/cpython/rev/e9497423de7a
changeset:   72309:e9497423de7a
parent:      72306:6cd2086c39c1
parent:      72308:cf811943046b
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Tue Sep 06 14:08:24 2011 +0100
summary:
  Closes #12906: Merged fix from 3.2.

files:
  Doc/howto/logging.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst
--- a/Doc/howto/logging.rst
+++ b/Doc/howto/logging.rst
@@ -679,7 +679,7 @@
     version: 1
     formatters:
       simple:
-        format: format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
+        format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
     handlers:
       console:
         class: logging.StreamHandler

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep  6 17:06:33 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Tue, 06 Sep 2011 17:06:33 +0200
Subject: [Python-checkins] =?utf8?q?devguide=3A_Add_documentation_about_cu?=
	=?utf8?q?stom_builders?=
Message-ID: <E1R0xEP-0003Ej-E8@dinsdale.python.org>

http://hg.python.org/devguide/rev/92424edd4270
changeset:   449:92424edd4270
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue Sep 06 17:03:51 2011 +0200
summary:
  Add documentation about custom builders

files:
  buildbots.rst |  31 +++++++++++++++++++++++++++++++
  1 files changed, 31 insertions(+), 0 deletions(-)


diff --git a/buildbots.rst b/buildbots.rst
--- a/buildbots.rst
+++ b/buildbots.rst
@@ -175,3 +175,34 @@
 better if it can be diagnosed and suppressed by fixing the test's implementation,
 or by making its parameters - such as a timeout - more robust.
 
+
+Custom builders
+---------------
+
+When working on a long-lived feature branch, or on a bugfix branch which
+enables issue-specific debugging, you will probably want to test your
+work on one or several buildbots.  Since your work is hosted in a distinct
+repository, you can't trigger builds on the regular builders.  Instead,
+you have to use one of the `custom builders
+<http://www.python.org/dev/buildbot/all/waterfall?category=custom.stable&category=custom.unstable>`_.
+
+When creating ("forcing") a build on a custom builder, you have to provide
+at least two parameters:
+
+* The repository path, relative to http://hg.python.org. For example,
+  ``sandbox/myfixes`` if ``http://hg.python.org/sandbox/myfixes`` is the
+  full path to the repository.
+
+* The Mercurial id of the changeset you want to build.  To make things less
+  tedious, we suggest you do your changes in a separate named branch: you can
+  then supply the branch name instead of a specific changeset id.
+
+If you are interested in the results of a specific test file only, we
+recommend you change (temporarily, of course) the contents of the
+``buildbottest`` clause in ``Makefile.pre.in``; or, for Windows builders,
+the ``Tools/buildbot/test.bat`` and ``Tools/buildbot/test-amd64.bat``
+scripts.
+
+.. note::
+   For security reasons, it is impossible to build repositories from outside
+   the http://hg.python.org realm.

-- 
Repository URL: http://hg.python.org/devguide

From python-checkins at python.org  Tue Sep  6 19:02:08 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Tue, 06 Sep 2011 19:02:08 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312871=3A_sched=5Fg?=
 =?utf8?q?et=5Fpriority=5F=28min=7Cmax=29_might_not_be_defined_even_though?=
Message-ID: <E1R0z2G-0006hq-G0@dinsdale.python.org>

http://hg.python.org/cpython/rev/5c8b6e03ebfe
changeset:   72310:5c8b6e03ebfe
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Tue Sep 06 19:03:35 2011 +0200
summary:
  Issue #12871: sched_get_priority_(min|max) might not be defined even though
<sched.h> is available (most notably on OpenBSD when built without pthread):
add an explicit configure check.

files:
  Lib/test/test_posix.py |    2 +
  Modules/posixmodule.c  |    8 +-
  configure              |  592 +++++++++++++---------------
  configure.in           |    3 +-
  pyconfig.h.in          |    3 +
  5 files changed, 299 insertions(+), 309 deletions(-)


diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py
--- a/Lib/test/test_posix.py
+++ b/Lib/test/test_posix.py
@@ -840,6 +840,8 @@
         posix.sched_yield()
 
     @requires_sched_h
+    @unittest.skipUnless(hasattr(posix, 'sched_get_priority_max'),
+                         "requires sched_get_priority_max()")
     def test_sched_priority(self):
         # Round-robin usually has interesting priorities.
         pol = posix.SCHED_RR
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -4555,6 +4555,8 @@
 
 #ifdef HAVE_SCHED_H
 
+#ifdef HAVE_SCHED_GET_PRIORITY_MAX
+
 PyDoc_STRVAR(posix_sched_get_priority_max__doc__,
 "sched_get_priority_max(policy)\n\n\
 Get the maximum scheduling priority for *policy*.");
@@ -4589,6 +4591,8 @@
     return PyLong_FromLong(min);
 }
 
+#endif /* HAVE_SCHED_GET_PRIORITY_MAX */
+
 #ifdef HAVE_SCHED_SETSCHEDULER
 
 PyDoc_STRVAR(posix_sched_getscheduler__doc__,
@@ -10452,8 +10456,10 @@
     {"fork",            posix_fork, METH_NOARGS, posix_fork__doc__},
 #endif /* HAVE_FORK */
 #ifdef HAVE_SCHED_H
+#ifdef HAVE_SCHED_GET_PRIORITY_MAX
     {"sched_get_priority_max", posix_sched_get_priority_max, METH_VARARGS, posix_sched_get_priority_max__doc__},
     {"sched_get_priority_min", posix_sched_get_priority_min, METH_VARARGS, posix_sched_get_priority_min__doc__},
+#endif
 #ifdef HAVE_SCHED_SETPARAM
     {"sched_getparam", posix_sched_getparam, METH_VARARGS, posix_sched_getparam__doc__},
 #endif
@@ -10474,7 +10480,7 @@
     {"sched_setaffinity", posix_sched_setaffinity, METH_VARARGS, posix_sched_setaffinity__doc__},
     {"sched_getaffinity", posix_sched_getaffinity, METH_VARARGS, posix_sched_getaffinity__doc__},
 #endif
-#endif
+#endif /* HAVE_SCHED_H */
 #if defined(HAVE_OPENPTY) || defined(HAVE__GETPTY) || defined(HAVE_DEV_PTMX)
     {"openpty",         posix_openpty, METH_NOARGS, posix_openpty__doc__},
 #endif /* HAVE_OPENPTY || HAVE__GETPTY || HAVE_DEV_PTMX */
diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -1,6 +1,6 @@
 #! /bin/sh
 # Guess values for system-dependent variables and create Makefiles.
-# Generated by GNU Autoconf 2.68 for python 3.3.
+# Generated by GNU Autoconf 2.67 for python 3.3.
 #
 # Report bugs to <http://bugs.python.org/>.
 #
@@ -91,7 +91,6 @@
 IFS=" ""	$as_nl"
 
 # Find who we are.  Look in the path if we contain no directory separator.
-as_myself=
 case $0 in #((
   *[\\/]* ) as_myself=$0 ;;
   *) as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
@@ -217,18 +216,11 @@
   # We cannot yet assume a decent shell, so we have to provide a
 	# neutralization value for shells without unset; and this also
 	# works around shells that cannot unset nonexistent variables.
-	# Preserve -v and -x to the replacement shell.
 	BASH_ENV=/dev/null
 	ENV=/dev/null
 	(unset BASH_ENV) >/dev/null 2>&1 && unset BASH_ENV ENV
 	export CONFIG_SHELL
-	case $- in # ((((
-	  *v*x* | *x*v* ) as_opts=-vx ;;
-	  *v* ) as_opts=-v ;;
-	  *x* ) as_opts=-x ;;
-	  * ) as_opts= ;;
-	esac
-	exec "$CONFIG_SHELL" $as_opts "$as_myself" ${1+"$@"}
+	exec "$CONFIG_SHELL" "$as_myself" ${1+"$@"}
 fi
 
     if test x$as_have_required = xno; then :
@@ -1183,7 +1175,7 @@
     $as_echo "$as_me: WARNING: you should use --build, --host, --target" >&2
     expr "x$ac_option" : ".*[^-._$as_cr_alnum]" >/dev/null &&
       $as_echo "$as_me: WARNING: invalid host type: $ac_option" >&2
-    : "${build_alias=$ac_option} ${host_alias=$ac_option} ${target_alias=$ac_option}"
+    : ${build_alias=$ac_option} ${host_alias=$ac_option} ${target_alias=$ac_option}
     ;;
 
   esac
@@ -1520,7 +1512,7 @@
 if $ac_init_version; then
   cat <<\_ACEOF
 python configure 3.3
-generated by GNU Autoconf 2.68
+generated by GNU Autoconf 2.67
 
 Copyright (C) 2010 Free Software Foundation, Inc.
 This configure script is free software; the Free Software Foundation
@@ -1566,7 +1558,7 @@
 
 	ac_retval=1
 fi
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
   as_fn_set_status $ac_retval
 
 } # ac_fn_c_try_compile
@@ -1612,7 +1604,7 @@
   # interfere with the next link command; also delete a directory that is
   # left behind by Apple's compiler.  We do this before executing the actions.
   rm -rf conftest.dSYM conftest_ipa8_conftest.oo
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
   as_fn_set_status $ac_retval
 
 } # ac_fn_c_try_link
@@ -1649,7 +1641,7 @@
 
     ac_retval=1
 fi
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
   as_fn_set_status $ac_retval
 
 } # ac_fn_c_try_cpp
@@ -1662,10 +1654,10 @@
 ac_fn_c_check_header_mongrel ()
 {
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
-  if eval \${$3+:} false; then :
+  if eval "test \"\${$3+set}\"" = set; then :
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2" >&5
 $as_echo_n "checking for $2... " >&6; }
-if eval \${$3+:} false; then :
+if eval "test \"\${$3+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 fi
 eval ac_res=\$$3
@@ -1732,7 +1724,7 @@
 esac
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2" >&5
 $as_echo_n "checking for $2... " >&6; }
-if eval \${$3+:} false; then :
+if eval "test \"\${$3+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   eval "$3=\$ac_header_compiler"
@@ -1741,7 +1733,7 @@
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
 fi
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
 
 } # ac_fn_c_check_header_mongrel
 
@@ -1782,7 +1774,7 @@
        ac_retval=$ac_status
 fi
   rm -rf conftest.dSYM conftest_ipa8_conftest.oo
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
   as_fn_set_status $ac_retval
 
 } # ac_fn_c_try_run
@@ -1796,7 +1788,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2" >&5
 $as_echo_n "checking for $2... " >&6; }
-if eval \${$3+:} false; then :
+if eval "test \"\${$3+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -1814,7 +1806,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
 
 } # ac_fn_c_check_header_compile
 
@@ -1827,7 +1819,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2" >&5
 $as_echo_n "checking for $2... " >&6; }
-if eval \${$3+:} false; then :
+if eval "test \"\${$3+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   eval "$3=no"
@@ -1868,7 +1860,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
 
 } # ac_fn_c_check_type
 
@@ -1881,7 +1873,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for uint$2_t" >&5
 $as_echo_n "checking for uint$2_t... " >&6; }
-if eval \${$3+:} false; then :
+if eval "test \"\${$3+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   eval "$3=no"
@@ -1921,7 +1913,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
 
 } # ac_fn_c_find_uintX_t
 
@@ -1934,7 +1926,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for int$2_t" >&5
 $as_echo_n "checking for int$2_t... " >&6; }
-if eval \${$3+:} false; then :
+if eval "test \"\${$3+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   eval "$3=no"
@@ -1995,7 +1987,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
 
 } # ac_fn_c_find_intX_t
 
@@ -2172,7 +2164,7 @@
 rm -f conftest.val
 
   fi
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
   as_fn_set_status $ac_retval
 
 } # ac_fn_c_compute_int
@@ -2185,7 +2177,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2" >&5
 $as_echo_n "checking for $2... " >&6; }
-if eval \${$3+:} false; then :
+if eval "test \"\${$3+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -2240,7 +2232,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
 
 } # ac_fn_c_check_func
 
@@ -2253,7 +2245,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2.$3" >&5
 $as_echo_n "checking for $2.$3... " >&6; }
-if eval \${$4+:} false; then :
+if eval "test \"\${$4+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -2297,7 +2289,7 @@
 eval ac_res=\$$4
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
 
 } # ac_fn_c_check_member
 
@@ -2312,7 +2304,7 @@
   as_decl_use=`echo $2|sed -e 's/(/((/' -e 's/)/) 0&/' -e 's/,/) 0& (/g'`
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $as_decl_name is declared" >&5
 $as_echo_n "checking whether $as_decl_name is declared... " >&6; }
-if eval \${$3+:} false; then :
+if eval "test \"\${$3+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -2343,7 +2335,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
 
 } # ac_fn_c_check_decl
 cat >config.log <<_ACEOF
@@ -2351,7 +2343,7 @@
 running configure, to aid debugging if configure makes a mistake.
 
 It was created by python $as_me 3.3, which was
-generated by GNU Autoconf 2.68.  Invocation command line was
+generated by GNU Autoconf 2.67.  Invocation command line was
 
   $ $0 $@
 
@@ -2609,7 +2601,7 @@
       || { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "failed to load site script $ac_site_file
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
   fi
 done
 
@@ -2709,7 +2701,7 @@
 set dummy hg; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_HAS_HG+:} false; then :
+if test "${ac_cv_prog_HAS_HG+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$HAS_HG"; then
@@ -3258,7 +3250,7 @@
 set dummy ${ac_tool_prefix}gcc; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_CC+:} false; then :
+if test "${ac_cv_prog_CC+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$CC"; then
@@ -3298,7 +3290,7 @@
 set dummy gcc; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_ac_ct_CC+:} false; then :
+if test "${ac_cv_prog_ac_ct_CC+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$ac_ct_CC"; then
@@ -3351,7 +3343,7 @@
 set dummy ${ac_tool_prefix}cc; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_CC+:} false; then :
+if test "${ac_cv_prog_CC+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$CC"; then
@@ -3391,7 +3383,7 @@
 set dummy cc; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_CC+:} false; then :
+if test "${ac_cv_prog_CC+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$CC"; then
@@ -3450,7 +3442,7 @@
 set dummy $ac_tool_prefix$ac_prog; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_CC+:} false; then :
+if test "${ac_cv_prog_CC+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$CC"; then
@@ -3494,7 +3486,7 @@
 set dummy $ac_prog; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_ac_ct_CC+:} false; then :
+if test "${ac_cv_prog_ac_ct_CC+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$ac_ct_CC"; then
@@ -3549,7 +3541,7 @@
 test -z "$CC" && { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "no acceptable C compiler found in \$PATH
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
 
 # Provide some information about the compiler.
 $as_echo "$as_me:${as_lineno-$LINENO}: checking for C compiler version" >&5
@@ -3664,7 +3656,7 @@
 { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "C compiler cannot create executables
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
 $as_echo "yes" >&6; }
@@ -3707,7 +3699,7 @@
   { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "cannot compute suffix of executables: cannot compile and link
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
 fi
 rm -f conftest conftest$ac_cv_exeext
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_exeext" >&5
@@ -3766,7 +3758,7 @@
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "cannot run C compiled programs.
 If you meant to cross compile, use \`--host'.
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
     fi
   fi
 fi
@@ -3777,7 +3769,7 @@
 ac_clean_files=$ac_clean_files_save
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for suffix of object files" >&5
 $as_echo_n "checking for suffix of object files... " >&6; }
-if ${ac_cv_objext+:} false; then :
+if test "${ac_cv_objext+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -3818,7 +3810,7 @@
 { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "cannot compute suffix of object files: cannot compile
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
 fi
 rm -f conftest.$ac_cv_objext conftest.$ac_ext
 fi
@@ -3828,7 +3820,7 @@
 ac_objext=$OBJEXT
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether we are using the GNU C compiler" >&5
 $as_echo_n "checking whether we are using the GNU C compiler... " >&6; }
-if ${ac_cv_c_compiler_gnu+:} false; then :
+if test "${ac_cv_c_compiler_gnu+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -3865,7 +3857,7 @@
 ac_save_CFLAGS=$CFLAGS
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC accepts -g" >&5
 $as_echo_n "checking whether $CC accepts -g... " >&6; }
-if ${ac_cv_prog_cc_g+:} false; then :
+if test "${ac_cv_prog_cc_g+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_save_c_werror_flag=$ac_c_werror_flag
@@ -3943,7 +3935,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $CC option to accept ISO C89" >&5
 $as_echo_n "checking for $CC option to accept ISO C89... " >&6; }
-if ${ac_cv_prog_cc_c89+:} false; then :
+if test "${ac_cv_prog_cc_c89+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_cv_prog_cc_c89=no
@@ -4078,7 +4070,7 @@
 set dummy g++; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_path_CXX+:} false; then :
+if test "${ac_cv_path_CXX+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   case $CXX in
@@ -4119,7 +4111,7 @@
 set dummy c++; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_path_CXX+:} false; then :
+if test "${ac_cv_path_CXX+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   case $CXX in
@@ -4170,7 +4162,7 @@
 set dummy $ac_prog; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_CXX+:} false; then :
+if test "${ac_cv_prog_CXX+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$CXX"; then
@@ -4271,7 +4263,7 @@
   CPP=
 fi
 if test -z "$CPP"; then
-  if ${ac_cv_prog_CPP+:} false; then :
+  if test "${ac_cv_prog_CPP+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
       # Double quotes because CPP needs to be expanded
@@ -4387,7 +4379,7 @@
   { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "C preprocessor \"$CPP\" fails sanity check
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
 fi
 
 ac_ext=c
@@ -4399,7 +4391,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for grep that handles long lines and -e" >&5
 $as_echo_n "checking for grep that handles long lines and -e... " >&6; }
-if ${ac_cv_path_GREP+:} false; then :
+if test "${ac_cv_path_GREP+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -z "$GREP"; then
@@ -4462,7 +4454,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for egrep" >&5
 $as_echo_n "checking for egrep... " >&6; }
-if ${ac_cv_path_EGREP+:} false; then :
+if test "${ac_cv_path_EGREP+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if echo a | $GREP -E '(a|b)' >/dev/null 2>&1
@@ -4529,7 +4521,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ANSI C header files" >&5
 $as_echo_n "checking for ANSI C header files... " >&6; }
-if ${ac_cv_header_stdc+:} false; then :
+if test "${ac_cv_header_stdc+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -4658,7 +4650,7 @@
 
 
   ac_fn_c_check_header_mongrel "$LINENO" "minix/config.h" "ac_cv_header_minix_config_h" "$ac_includes_default"
-if test "x$ac_cv_header_minix_config_h" = xyes; then :
+if test "x$ac_cv_header_minix_config_h" = x""yes; then :
   MINIX=yes
 else
   MINIX=
@@ -4680,7 +4672,7 @@
 
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether it is safe to define __EXTENSIONS__" >&5
 $as_echo_n "checking whether it is safe to define __EXTENSIONS__... " >&6; }
-if ${ac_cv_safe_to_define___extensions__+:} false; then :
+if test "${ac_cv_safe_to_define___extensions__+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -4873,7 +4865,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for inline" >&5
 $as_echo_n "checking for inline... " >&6; }
-if ${ac_cv_c_inline+:} false; then :
+if test "${ac_cv_c_inline+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_cv_c_inline=no
@@ -5069,7 +5061,7 @@
 set dummy ${ac_tool_prefix}ranlib; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_RANLIB+:} false; then :
+if test "${ac_cv_prog_RANLIB+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$RANLIB"; then
@@ -5109,7 +5101,7 @@
 set dummy ranlib; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_ac_ct_RANLIB+:} false; then :
+if test "${ac_cv_prog_ac_ct_RANLIB+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$ac_ct_RANLIB"; then
@@ -5163,7 +5155,7 @@
 set dummy $ac_prog; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_AR+:} false; then :
+if test "${ac_cv_prog_AR+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$AR"; then
@@ -5214,7 +5206,7 @@
 set dummy python; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_HAS_PYTHON+:} false; then :
+if test "${ac_cv_prog_HAS_PYTHON+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$HAS_PYTHON"; then
@@ -5308,7 +5300,7 @@
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for a BSD-compatible install" >&5
 $as_echo_n "checking for a BSD-compatible install... " >&6; }
 if test -z "$INSTALL"; then
-if ${ac_cv_path_install+:} false; then :
+if test "${ac_cv_path_install+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
@@ -5494,7 +5486,7 @@
      ac_save_cc="$CC"
      CC="$CC -fno-strict-aliasing"
      save_CFLAGS="$CFLAGS"
-     if ${ac_cv_no_strict_aliasing+:} false; then :
+     if test "${ac_cv_no_strict_aliasing+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -5744,7 +5736,7 @@
 # options before we can check whether -Kpthread improves anything.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether pthreads are available without options" >&5
 $as_echo_n "checking whether pthreads are available without options... " >&6; }
-if ${ac_cv_pthread_is_default+:} false; then :
+if test "${ac_cv_pthread_is_default+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -5797,7 +5789,7 @@
 # function available.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC accepts -Kpthread" >&5
 $as_echo_n "checking whether $CC accepts -Kpthread... " >&6; }
-if ${ac_cv_kpthread+:} false; then :
+if test "${ac_cv_kpthread+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_save_cc="$CC"
@@ -5846,7 +5838,7 @@
 # function available.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC accepts -Kthread" >&5
 $as_echo_n "checking whether $CC accepts -Kthread... " >&6; }
-if ${ac_cv_kthread+:} false; then :
+if test "${ac_cv_kthread+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_save_cc="$CC"
@@ -5895,7 +5887,7 @@
 # function available.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC accepts -pthread" >&5
 $as_echo_n "checking whether $CC accepts -pthread... " >&6; }
-if ${ac_cv_thread+:} false; then :
+if test "${ac_cv_thread+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_save_cc="$CC"
@@ -5980,7 +5972,7 @@
 # checks for header files
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ANSI C header files" >&5
 $as_echo_n "checking for ANSI C header files... " >&6; }
-if ${ac_cv_header_stdc+:} false; then :
+if test "${ac_cv_header_stdc+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -6119,7 +6111,7 @@
   as_ac_Header=`$as_echo "ac_cv_header_dirent_$ac_hdr" | $as_tr_sh`
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_hdr that defines DIR" >&5
 $as_echo_n "checking for $ac_hdr that defines DIR... " >&6; }
-if eval \${$as_ac_Header+:} false; then :
+if eval "test \"\${$as_ac_Header+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -6159,7 +6151,7 @@
 if test $ac_header_dirent = dirent.h; then
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing opendir" >&5
 $as_echo_n "checking for library containing opendir... " >&6; }
-if ${ac_cv_search_opendir+:} false; then :
+if test "${ac_cv_search_opendir+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_func_search_save_LIBS=$LIBS
@@ -6193,11 +6185,11 @@
 fi
 rm -f core conftest.err conftest.$ac_objext \
     conftest$ac_exeext
-  if ${ac_cv_search_opendir+:} false; then :
+  if test "${ac_cv_search_opendir+set}" = set; then :
   break
 fi
 done
-if ${ac_cv_search_opendir+:} false; then :
+if test "${ac_cv_search_opendir+set}" = set; then :
 
 else
   ac_cv_search_opendir=no
@@ -6216,7 +6208,7 @@
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing opendir" >&5
 $as_echo_n "checking for library containing opendir... " >&6; }
-if ${ac_cv_search_opendir+:} false; then :
+if test "${ac_cv_search_opendir+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_func_search_save_LIBS=$LIBS
@@ -6250,11 +6242,11 @@
 fi
 rm -f core conftest.err conftest.$ac_objext \
     conftest$ac_exeext
-  if ${ac_cv_search_opendir+:} false; then :
+  if test "${ac_cv_search_opendir+set}" = set; then :
   break
 fi
 done
-if ${ac_cv_search_opendir+:} false; then :
+if test "${ac_cv_search_opendir+set}" = set; then :
 
 else
   ac_cv_search_opendir=no
@@ -6274,7 +6266,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether sys/types.h defines makedev" >&5
 $as_echo_n "checking whether sys/types.h defines makedev... " >&6; }
-if ${ac_cv_header_sys_types_h_makedev+:} false; then :
+if test "${ac_cv_header_sys_types_h_makedev+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -6302,7 +6294,7 @@
 
 if test $ac_cv_header_sys_types_h_makedev = no; then
 ac_fn_c_check_header_mongrel "$LINENO" "sys/mkdev.h" "ac_cv_header_sys_mkdev_h" "$ac_includes_default"
-if test "x$ac_cv_header_sys_mkdev_h" = xyes; then :
+if test "x$ac_cv_header_sys_mkdev_h" = x""yes; then :
 
 $as_echo "#define MAJOR_IN_MKDEV 1" >>confdefs.h
 
@@ -6312,7 +6304,7 @@
 
   if test $ac_cv_header_sys_mkdev_h = no; then
     ac_fn_c_check_header_mongrel "$LINENO" "sys/sysmacros.h" "ac_cv_header_sys_sysmacros_h" "$ac_includes_default"
-if test "x$ac_cv_header_sys_sysmacros_h" = xyes; then :
+if test "x$ac_cv_header_sys_sysmacros_h" = x""yes; then :
 
 $as_echo "#define MAJOR_IN_SYSMACROS 1" >>confdefs.h
 
@@ -6340,7 +6332,7 @@
 #endif
 
 "
-if test "x$ac_cv_header_net_if_h" = xyes; then :
+if test "x$ac_cv_header_net_if_h" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_NET_IF_H 1
 _ACEOF
@@ -6360,7 +6352,7 @@
 #endif
 
 "
-if test "x$ac_cv_header_term_h" = xyes; then :
+if test "x$ac_cv_header_term_h" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_TERM_H 1
 _ACEOF
@@ -6382,7 +6374,7 @@
 #endif
 
 "
-if test "x$ac_cv_header_linux_netlink_h" = xyes; then :
+if test "x$ac_cv_header_linux_netlink_h" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_LINUX_NETLINK_H 1
 _ACEOF
@@ -6518,7 +6510,7 @@
 
 # Type availability checks
 ac_fn_c_check_type "$LINENO" "mode_t" "ac_cv_type_mode_t" "$ac_includes_default"
-if test "x$ac_cv_type_mode_t" = xyes; then :
+if test "x$ac_cv_type_mode_t" = x""yes; then :
 
 else
 
@@ -6529,7 +6521,7 @@
 fi
 
 ac_fn_c_check_type "$LINENO" "off_t" "ac_cv_type_off_t" "$ac_includes_default"
-if test "x$ac_cv_type_off_t" = xyes; then :
+if test "x$ac_cv_type_off_t" = x""yes; then :
 
 else
 
@@ -6540,7 +6532,7 @@
 fi
 
 ac_fn_c_check_type "$LINENO" "pid_t" "ac_cv_type_pid_t" "$ac_includes_default"
-if test "x$ac_cv_type_pid_t" = xyes; then :
+if test "x$ac_cv_type_pid_t" = x""yes; then :
 
 else
 
@@ -6556,7 +6548,7 @@
 _ACEOF
 
 ac_fn_c_check_type "$LINENO" "size_t" "ac_cv_type_size_t" "$ac_includes_default"
-if test "x$ac_cv_type_size_t" = xyes; then :
+if test "x$ac_cv_type_size_t" = x""yes; then :
 
 else
 
@@ -6568,7 +6560,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for uid_t in sys/types.h" >&5
 $as_echo_n "checking for uid_t in sys/types.h... " >&6; }
-if ${ac_cv_type_uid_t+:} false; then :
+if test "${ac_cv_type_uid_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -6647,7 +6639,7 @@
 esac
 
 ac_fn_c_check_type "$LINENO" "ssize_t" "ac_cv_type_ssize_t" "$ac_includes_default"
-if test "x$ac_cv_type_ssize_t" = xyes; then :
+if test "x$ac_cv_type_ssize_t" = x""yes; then :
 
 $as_echo "#define HAVE_SSIZE_T 1" >>confdefs.h
 
@@ -6662,7 +6654,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of int" >&5
 $as_echo_n "checking size of int... " >&6; }
-if ${ac_cv_sizeof_int+:} false; then :
+if test "${ac_cv_sizeof_int+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (int))" "ac_cv_sizeof_int"        "$ac_includes_default"; then :
@@ -6672,7 +6664,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (int)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_int=0
    fi
@@ -6695,7 +6687,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of long" >&5
 $as_echo_n "checking size of long... " >&6; }
-if ${ac_cv_sizeof_long+:} false; then :
+if test "${ac_cv_sizeof_long+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (long))" "ac_cv_sizeof_long"        "$ac_includes_default"; then :
@@ -6705,7 +6697,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (long)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_long=0
    fi
@@ -6728,7 +6720,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of void *" >&5
 $as_echo_n "checking size of void *... " >&6; }
-if ${ac_cv_sizeof_void_p+:} false; then :
+if test "${ac_cv_sizeof_void_p+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (void *))" "ac_cv_sizeof_void_p"        "$ac_includes_default"; then :
@@ -6738,7 +6730,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (void *)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_void_p=0
    fi
@@ -6761,7 +6753,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of short" >&5
 $as_echo_n "checking size of short... " >&6; }
-if ${ac_cv_sizeof_short+:} false; then :
+if test "${ac_cv_sizeof_short+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (short))" "ac_cv_sizeof_short"        "$ac_includes_default"; then :
@@ -6771,7 +6763,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (short)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_short=0
    fi
@@ -6794,7 +6786,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of float" >&5
 $as_echo_n "checking size of float... " >&6; }
-if ${ac_cv_sizeof_float+:} false; then :
+if test "${ac_cv_sizeof_float+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (float))" "ac_cv_sizeof_float"        "$ac_includes_default"; then :
@@ -6804,7 +6796,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (float)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_float=0
    fi
@@ -6827,7 +6819,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of double" >&5
 $as_echo_n "checking size of double... " >&6; }
-if ${ac_cv_sizeof_double+:} false; then :
+if test "${ac_cv_sizeof_double+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (double))" "ac_cv_sizeof_double"        "$ac_includes_default"; then :
@@ -6837,7 +6829,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (double)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_double=0
    fi
@@ -6860,7 +6852,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of fpos_t" >&5
 $as_echo_n "checking size of fpos_t... " >&6; }
-if ${ac_cv_sizeof_fpos_t+:} false; then :
+if test "${ac_cv_sizeof_fpos_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (fpos_t))" "ac_cv_sizeof_fpos_t"        "$ac_includes_default"; then :
@@ -6870,7 +6862,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (fpos_t)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_fpos_t=0
    fi
@@ -6893,7 +6885,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of size_t" >&5
 $as_echo_n "checking size of size_t... " >&6; }
-if ${ac_cv_sizeof_size_t+:} false; then :
+if test "${ac_cv_sizeof_size_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (size_t))" "ac_cv_sizeof_size_t"        "$ac_includes_default"; then :
@@ -6903,7 +6895,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (size_t)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_size_t=0
    fi
@@ -6926,7 +6918,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of pid_t" >&5
 $as_echo_n "checking size of pid_t... " >&6; }
-if ${ac_cv_sizeof_pid_t+:} false; then :
+if test "${ac_cv_sizeof_pid_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (pid_t))" "ac_cv_sizeof_pid_t"        "$ac_includes_default"; then :
@@ -6936,7 +6928,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (pid_t)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_pid_t=0
    fi
@@ -6986,7 +6978,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of long long" >&5
 $as_echo_n "checking size of long long... " >&6; }
-if ${ac_cv_sizeof_long_long+:} false; then :
+if test "${ac_cv_sizeof_long_long+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (long long))" "ac_cv_sizeof_long_long"        "$ac_includes_default"; then :
@@ -6996,7 +6988,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (long long)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_long_long=0
    fi
@@ -7047,7 +7039,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of long double" >&5
 $as_echo_n "checking size of long double... " >&6; }
-if ${ac_cv_sizeof_long_double+:} false; then :
+if test "${ac_cv_sizeof_long_double+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (long double))" "ac_cv_sizeof_long_double"        "$ac_includes_default"; then :
@@ -7057,7 +7049,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (long double)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_long_double=0
    fi
@@ -7109,7 +7101,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of _Bool" >&5
 $as_echo_n "checking size of _Bool... " >&6; }
-if ${ac_cv_sizeof__Bool+:} false; then :
+if test "${ac_cv_sizeof__Bool+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (_Bool))" "ac_cv_sizeof__Bool"        "$ac_includes_default"; then :
@@ -7119,7 +7111,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (_Bool)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof__Bool=0
    fi
@@ -7145,7 +7137,7 @@
         #include <inttypes.h>
         #endif
 "
-if test "x$ac_cv_type_uintptr_t" = xyes; then :
+if test "x$ac_cv_type_uintptr_t" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_UINTPTR_T 1
@@ -7157,7 +7149,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of uintptr_t" >&5
 $as_echo_n "checking size of uintptr_t... " >&6; }
-if ${ac_cv_sizeof_uintptr_t+:} false; then :
+if test "${ac_cv_sizeof_uintptr_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (uintptr_t))" "ac_cv_sizeof_uintptr_t"        "$ac_includes_default"; then :
@@ -7167,7 +7159,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (uintptr_t)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_uintptr_t=0
    fi
@@ -7193,7 +7185,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of off_t" >&5
 $as_echo_n "checking size of off_t... " >&6; }
-if ${ac_cv_sizeof_off_t+:} false; then :
+if test "${ac_cv_sizeof_off_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (off_t))" "ac_cv_sizeof_off_t"        "
@@ -7208,7 +7200,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (off_t)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_off_t=0
    fi
@@ -7252,7 +7244,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of time_t" >&5
 $as_echo_n "checking size of time_t... " >&6; }
-if ${ac_cv_sizeof_time_t+:} false; then :
+if test "${ac_cv_sizeof_time_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (time_t))" "ac_cv_sizeof_time_t"        "
@@ -7270,7 +7262,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (time_t)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_time_t=0
    fi
@@ -7327,7 +7319,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of pthread_t" >&5
 $as_echo_n "checking size of pthread_t... " >&6; }
-if ${ac_cv_sizeof_pthread_t+:} false; then :
+if test "${ac_cv_sizeof_pthread_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (pthread_t))" "ac_cv_sizeof_pthread_t"        "
@@ -7342,7 +7334,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (pthread_t)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_pthread_t=0
    fi
@@ -7773,7 +7765,7 @@
 # checks for libraries
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sendfile in -lsendfile" >&5
 $as_echo_n "checking for sendfile in -lsendfile... " >&6; }
-if ${ac_cv_lib_sendfile_sendfile+:} false; then :
+if test "${ac_cv_lib_sendfile_sendfile+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -7807,7 +7799,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_sendfile_sendfile" >&5
 $as_echo "$ac_cv_lib_sendfile_sendfile" >&6; }
-if test "x$ac_cv_lib_sendfile_sendfile" = xyes; then :
+if test "x$ac_cv_lib_sendfile_sendfile" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_LIBSENDFILE 1
 _ACEOF
@@ -7818,7 +7810,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for dlopen in -ldl" >&5
 $as_echo_n "checking for dlopen in -ldl... " >&6; }
-if ${ac_cv_lib_dl_dlopen+:} false; then :
+if test "${ac_cv_lib_dl_dlopen+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -7852,7 +7844,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_dl_dlopen" >&5
 $as_echo "$ac_cv_lib_dl_dlopen" >&6; }
-if test "x$ac_cv_lib_dl_dlopen" = xyes; then :
+if test "x$ac_cv_lib_dl_dlopen" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_LIBDL 1
 _ACEOF
@@ -7863,7 +7855,7 @@
 	# Dynamic linking for SunOS/Solaris and SYSV
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for shl_load in -ldld" >&5
 $as_echo_n "checking for shl_load in -ldld... " >&6; }
-if ${ac_cv_lib_dld_shl_load+:} false; then :
+if test "${ac_cv_lib_dld_shl_load+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -7897,7 +7889,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_dld_shl_load" >&5
 $as_echo "$ac_cv_lib_dld_shl_load" >&6; }
-if test "x$ac_cv_lib_dld_shl_load" = xyes; then :
+if test "x$ac_cv_lib_dld_shl_load" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_LIBDLD 1
 _ACEOF
@@ -7911,7 +7903,7 @@
 if test "$with_threads" = "yes" -o -z "$with_threads"; then
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing sem_init" >&5
 $as_echo_n "checking for library containing sem_init... " >&6; }
-if ${ac_cv_search_sem_init+:} false; then :
+if test "${ac_cv_search_sem_init+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_func_search_save_LIBS=$LIBS
@@ -7945,11 +7937,11 @@
 fi
 rm -f core conftest.err conftest.$ac_objext \
     conftest$ac_exeext
-  if ${ac_cv_search_sem_init+:} false; then :
+  if test "${ac_cv_search_sem_init+set}" = set; then :
   break
 fi
 done
-if ${ac_cv_search_sem_init+:} false; then :
+if test "${ac_cv_search_sem_init+set}" = set; then :
 
 else
   ac_cv_search_sem_init=no
@@ -7972,7 +7964,7 @@
 # check if we need libintl for locale functions
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for textdomain in -lintl" >&5
 $as_echo_n "checking for textdomain in -lintl... " >&6; }
-if ${ac_cv_lib_intl_textdomain+:} false; then :
+if test "${ac_cv_lib_intl_textdomain+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8006,7 +7998,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_intl_textdomain" >&5
 $as_echo "$ac_cv_lib_intl_textdomain" >&6; }
-if test "x$ac_cv_lib_intl_textdomain" = xyes; then :
+if test "x$ac_cv_lib_intl_textdomain" = x""yes; then :
 
 $as_echo "#define WITH_LIBINTL 1" >>confdefs.h
 
@@ -8053,7 +8045,7 @@
 # Most SVR4 platforms (e.g. Solaris) need -lsocket and -lnsl.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for t_open in -lnsl" >&5
 $as_echo_n "checking for t_open in -lnsl... " >&6; }
-if ${ac_cv_lib_nsl_t_open+:} false; then :
+if test "${ac_cv_lib_nsl_t_open+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8087,13 +8079,13 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_nsl_t_open" >&5
 $as_echo "$ac_cv_lib_nsl_t_open" >&6; }
-if test "x$ac_cv_lib_nsl_t_open" = xyes; then :
+if test "x$ac_cv_lib_nsl_t_open" = x""yes; then :
   LIBS="-lnsl $LIBS"
 fi
  # SVR4
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for socket in -lsocket" >&5
 $as_echo_n "checking for socket in -lsocket... " >&6; }
-if ${ac_cv_lib_socket_socket+:} false; then :
+if test "${ac_cv_lib_socket_socket+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8127,7 +8119,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_socket_socket" >&5
 $as_echo "$ac_cv_lib_socket_socket" >&6; }
-if test "x$ac_cv_lib_socket_socket" = xyes; then :
+if test "x$ac_cv_lib_socket_socket" = x""yes; then :
   LIBS="-lsocket $LIBS"
 fi
  # SVR4 sockets
@@ -8153,7 +8145,7 @@
 set dummy ${ac_tool_prefix}pkg-config; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_path_PKG_CONFIG+:} false; then :
+if test "${ac_cv_path_PKG_CONFIG+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   case $PKG_CONFIG in
@@ -8196,7 +8188,7 @@
 set dummy pkg-config; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_path_ac_pt_PKG_CONFIG+:} false; then :
+if test "${ac_cv_path_ac_pt_PKG_CONFIG+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   case $ac_pt_PKG_CONFIG in
@@ -8492,7 +8484,7 @@
 
     LIBS=$_libs
     ac_fn_c_check_func "$LINENO" "pthread_detach" "ac_cv_func_pthread_detach"
-if test "x$ac_cv_func_pthread_detach" = xyes; then :
+if test "x$ac_cv_func_pthread_detach" = x""yes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     posix_threads=yes
@@ -8501,7 +8493,7 @@
 
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread_create in -lpthreads" >&5
 $as_echo_n "checking for pthread_create in -lpthreads... " >&6; }
-if ${ac_cv_lib_pthreads_pthread_create+:} false; then :
+if test "${ac_cv_lib_pthreads_pthread_create+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8535,7 +8527,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_pthreads_pthread_create" >&5
 $as_echo "$ac_cv_lib_pthreads_pthread_create" >&6; }
-if test "x$ac_cv_lib_pthreads_pthread_create" = xyes; then :
+if test "x$ac_cv_lib_pthreads_pthread_create" = x""yes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     posix_threads=yes
@@ -8545,7 +8537,7 @@
 
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread_create in -lc_r" >&5
 $as_echo_n "checking for pthread_create in -lc_r... " >&6; }
-if ${ac_cv_lib_c_r_pthread_create+:} false; then :
+if test "${ac_cv_lib_c_r_pthread_create+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8579,7 +8571,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_c_r_pthread_create" >&5
 $as_echo "$ac_cv_lib_c_r_pthread_create" >&6; }
-if test "x$ac_cv_lib_c_r_pthread_create" = xyes; then :
+if test "x$ac_cv_lib_c_r_pthread_create" = x""yes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     posix_threads=yes
@@ -8589,7 +8581,7 @@
 
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for __pthread_create_system in -lpthread" >&5
 $as_echo_n "checking for __pthread_create_system in -lpthread... " >&6; }
-if ${ac_cv_lib_pthread___pthread_create_system+:} false; then :
+if test "${ac_cv_lib_pthread___pthread_create_system+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8623,7 +8615,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_pthread___pthread_create_system" >&5
 $as_echo "$ac_cv_lib_pthread___pthread_create_system" >&6; }
-if test "x$ac_cv_lib_pthread___pthread_create_system" = xyes; then :
+if test "x$ac_cv_lib_pthread___pthread_create_system" = x""yes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     posix_threads=yes
@@ -8633,7 +8625,7 @@
 
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread_create in -lcma" >&5
 $as_echo_n "checking for pthread_create in -lcma... " >&6; }
-if ${ac_cv_lib_cma_pthread_create+:} false; then :
+if test "${ac_cv_lib_cma_pthread_create+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8667,7 +8659,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_cma_pthread_create" >&5
 $as_echo "$ac_cv_lib_cma_pthread_create" >&6; }
-if test "x$ac_cv_lib_cma_pthread_create" = xyes; then :
+if test "x$ac_cv_lib_cma_pthread_create" = x""yes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     posix_threads=yes
@@ -8693,7 +8685,7 @@
 
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for usconfig in -lmpc" >&5
 $as_echo_n "checking for usconfig in -lmpc... " >&6; }
-if ${ac_cv_lib_mpc_usconfig+:} false; then :
+if test "${ac_cv_lib_mpc_usconfig+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8727,7 +8719,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_mpc_usconfig" >&5
 $as_echo "$ac_cv_lib_mpc_usconfig" >&6; }
-if test "x$ac_cv_lib_mpc_usconfig" = xyes; then :
+if test "x$ac_cv_lib_mpc_usconfig" = x""yes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     LIBS="$LIBS -lmpc"
@@ -8739,7 +8731,7 @@
     if test "$posix_threads" != "yes"; then
       { $as_echo "$as_me:${as_lineno-$LINENO}: checking for thr_create in -lthread" >&5
 $as_echo_n "checking for thr_create in -lthread... " >&6; }
-if ${ac_cv_lib_thread_thr_create+:} false; then :
+if test "${ac_cv_lib_thread_thr_create+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8773,7 +8765,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_thread_thr_create" >&5
 $as_echo "$ac_cv_lib_thread_thr_create" >&6; }
-if test "x$ac_cv_lib_thread_thr_create" = xyes; then :
+if test "x$ac_cv_lib_thread_thr_create" = x""yes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
       LIBS="$LIBS -lthread"
@@ -8809,7 +8801,7 @@
 
       { $as_echo "$as_me:${as_lineno-$LINENO}: checking if PTHREAD_SCOPE_SYSTEM is supported" >&5
 $as_echo_n "checking if PTHREAD_SCOPE_SYSTEM is supported... " >&6; }
-      if ${ac_cv_pthread_system_supported+:} false; then :
+      if test "${ac_cv_pthread_system_supported+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -8852,7 +8844,7 @@
       for ac_func in pthread_sigmask
 do :
   ac_fn_c_check_func "$LINENO" "pthread_sigmask" "ac_cv_func_pthread_sigmask"
-if test "x$ac_cv_func_pthread_sigmask" = xyes; then :
+if test "x$ac_cv_func_pthread_sigmask" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_PTHREAD_SIGMASK 1
 _ACEOF
@@ -9244,7 +9236,7 @@
 $as_echo "$with_valgrind" >&6; }
 if test "$with_valgrind" != no; then
     ac_fn_c_check_header_mongrel "$LINENO" "valgrind/valgrind.h" "ac_cv_header_valgrind_valgrind_h" "$ac_includes_default"
-if test "x$ac_cv_header_valgrind_valgrind_h" = xyes; then :
+if test "x$ac_cv_header_valgrind_valgrind_h" = x""yes; then :
 
 $as_echo "#define WITH_VALGRIND 1" >>confdefs.h
 
@@ -9266,7 +9258,7 @@
 for ac_func in dlopen
 do :
   ac_fn_c_check_func "$LINENO" "dlopen" "ac_cv_func_dlopen"
-if test "x$ac_cv_func_dlopen" = xyes; then :
+if test "x$ac_cv_func_dlopen" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_DLOPEN 1
 _ACEOF
@@ -9340,7 +9332,8 @@
  select sem_open sem_timedwait sem_getvalue sem_unlink sendfile setegid seteuid \
  setgid sethostname \
  setlocale setregid setreuid setresuid setresgid setsid setpgid setpgrp setpriority setuid setvbuf \
- sched_setaffinity sched_setscheduler sched_setparam sched_rr_get_interval \
+ sched_get_priority_max sched_setaffinity sched_setscheduler sched_setparam \
+ sched_rr_get_interval \
  sigaction sigaltstack siginterrupt sigpending sigrelse \
  sigtimedwait sigwait sigwaitinfo snprintf strftime strlcpy symlinkat sync \
  sysconf tcgetpgrp tcsetpgrp tempnam timegm times tmpfile tmpnam tmpnam_r \
@@ -9599,7 +9592,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for flock declaration" >&5
 $as_echo_n "checking for flock declaration... " >&6; }
-if ${ac_cv_flock_decl+:} false; then :
+if test "${ac_cv_flock_decl+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -9629,7 +9622,7 @@
   for ac_func in flock
 do :
   ac_fn_c_check_func "$LINENO" "flock" "ac_cv_func_flock"
-if test "x$ac_cv_func_flock" = xyes; then :
+if test "x$ac_cv_func_flock" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_FLOCK 1
 _ACEOF
@@ -9637,7 +9630,7 @@
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for flock in -lbsd" >&5
 $as_echo_n "checking for flock in -lbsd... " >&6; }
-if ${ac_cv_lib_bsd_flock+:} false; then :
+if test "${ac_cv_lib_bsd_flock+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -9671,7 +9664,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_bsd_flock" >&5
 $as_echo "$ac_cv_lib_bsd_flock" >&6; }
-if test "x$ac_cv_lib_bsd_flock" = xyes; then :
+if test "x$ac_cv_lib_bsd_flock" = x""yes; then :
   $as_echo "#define HAVE_FLOCK 1" >>confdefs.h
 
 
@@ -9748,7 +9741,7 @@
 set dummy $ac_prog; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_TRUE+:} false; then :
+if test "${ac_cv_prog_TRUE+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$TRUE"; then
@@ -9788,7 +9781,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for inet_aton in -lc" >&5
 $as_echo_n "checking for inet_aton in -lc... " >&6; }
-if ${ac_cv_lib_c_inet_aton+:} false; then :
+if test "${ac_cv_lib_c_inet_aton+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -9822,12 +9815,12 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_c_inet_aton" >&5
 $as_echo "$ac_cv_lib_c_inet_aton" >&6; }
-if test "x$ac_cv_lib_c_inet_aton" = xyes; then :
+if test "x$ac_cv_lib_c_inet_aton" = x""yes; then :
   $ac_cv_prog_TRUE
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for inet_aton in -lresolv" >&5
 $as_echo_n "checking for inet_aton in -lresolv... " >&6; }
-if ${ac_cv_lib_resolv_inet_aton+:} false; then :
+if test "${ac_cv_lib_resolv_inet_aton+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -9861,7 +9854,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_resolv_inet_aton" >&5
 $as_echo "$ac_cv_lib_resolv_inet_aton" >&6; }
-if test "x$ac_cv_lib_resolv_inet_aton" = xyes; then :
+if test "x$ac_cv_lib_resolv_inet_aton" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_LIBRESOLV 1
 _ACEOF
@@ -9878,7 +9871,7 @@
 # exit Python
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for chflags" >&5
 $as_echo_n "checking for chflags... " >&6; }
-if ${ac_cv_have_chflags+:} false; then :
+if test "${ac_cv_have_chflags+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -9912,7 +9905,7 @@
 $as_echo "$ac_cv_have_chflags" >&6; }
 if test "$ac_cv_have_chflags" = cross ; then
   ac_fn_c_check_func "$LINENO" "chflags" "ac_cv_func_chflags"
-if test "x$ac_cv_func_chflags" = xyes; then :
+if test "x$ac_cv_func_chflags" = x""yes; then :
   ac_cv_have_chflags="yes"
 else
   ac_cv_have_chflags="no"
@@ -9927,7 +9920,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for lchflags" >&5
 $as_echo_n "checking for lchflags... " >&6; }
-if ${ac_cv_have_lchflags+:} false; then :
+if test "${ac_cv_have_lchflags+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -9961,7 +9954,7 @@
 $as_echo "$ac_cv_have_lchflags" >&6; }
 if test "$ac_cv_have_lchflags" = cross ; then
   ac_fn_c_check_func "$LINENO" "lchflags" "ac_cv_func_lchflags"
-if test "x$ac_cv_func_lchflags" = xyes; then :
+if test "x$ac_cv_func_lchflags" = x""yes; then :
   ac_cv_have_lchflags="yes"
 else
   ac_cv_have_lchflags="no"
@@ -9985,7 +9978,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for inflateCopy in -lz" >&5
 $as_echo_n "checking for inflateCopy in -lz... " >&6; }
-if ${ac_cv_lib_z_inflateCopy+:} false; then :
+if test "${ac_cv_lib_z_inflateCopy+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -10019,7 +10012,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_z_inflateCopy" >&5
 $as_echo "$ac_cv_lib_z_inflateCopy" >&6; }
-if test "x$ac_cv_lib_z_inflateCopy" = xyes; then :
+if test "x$ac_cv_lib_z_inflateCopy" = x""yes; then :
 
 $as_echo "#define HAVE_ZLIB_COPY 1" >>confdefs.h
 
@@ -10162,7 +10155,7 @@
 for ac_func in openpty
 do :
   ac_fn_c_check_func "$LINENO" "openpty" "ac_cv_func_openpty"
-if test "x$ac_cv_func_openpty" = xyes; then :
+if test "x$ac_cv_func_openpty" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_OPENPTY 1
 _ACEOF
@@ -10170,7 +10163,7 @@
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for openpty in -lutil" >&5
 $as_echo_n "checking for openpty in -lutil... " >&6; }
-if ${ac_cv_lib_util_openpty+:} false; then :
+if test "${ac_cv_lib_util_openpty+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -10204,13 +10197,13 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_util_openpty" >&5
 $as_echo "$ac_cv_lib_util_openpty" >&6; }
-if test "x$ac_cv_lib_util_openpty" = xyes; then :
+if test "x$ac_cv_lib_util_openpty" = x""yes; then :
   $as_echo "#define HAVE_OPENPTY 1" >>confdefs.h
  LIBS="$LIBS -lutil"
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for openpty in -lbsd" >&5
 $as_echo_n "checking for openpty in -lbsd... " >&6; }
-if ${ac_cv_lib_bsd_openpty+:} false; then :
+if test "${ac_cv_lib_bsd_openpty+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -10244,7 +10237,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_bsd_openpty" >&5
 $as_echo "$ac_cv_lib_bsd_openpty" >&6; }
-if test "x$ac_cv_lib_bsd_openpty" = xyes; then :
+if test "x$ac_cv_lib_bsd_openpty" = x""yes; then :
   $as_echo "#define HAVE_OPENPTY 1" >>confdefs.h
  LIBS="$LIBS -lbsd"
 fi
@@ -10259,7 +10252,7 @@
 for ac_func in forkpty
 do :
   ac_fn_c_check_func "$LINENO" "forkpty" "ac_cv_func_forkpty"
-if test "x$ac_cv_func_forkpty" = xyes; then :
+if test "x$ac_cv_func_forkpty" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_FORKPTY 1
 _ACEOF
@@ -10267,7 +10260,7 @@
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for forkpty in -lutil" >&5
 $as_echo_n "checking for forkpty in -lutil... " >&6; }
-if ${ac_cv_lib_util_forkpty+:} false; then :
+if test "${ac_cv_lib_util_forkpty+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -10301,13 +10294,13 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_util_forkpty" >&5
 $as_echo "$ac_cv_lib_util_forkpty" >&6; }
-if test "x$ac_cv_lib_util_forkpty" = xyes; then :
+if test "x$ac_cv_lib_util_forkpty" = x""yes; then :
   $as_echo "#define HAVE_FORKPTY 1" >>confdefs.h
  LIBS="$LIBS -lutil"
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for forkpty in -lbsd" >&5
 $as_echo_n "checking for forkpty in -lbsd... " >&6; }
-if ${ac_cv_lib_bsd_forkpty+:} false; then :
+if test "${ac_cv_lib_bsd_forkpty+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -10341,7 +10334,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_bsd_forkpty" >&5
 $as_echo "$ac_cv_lib_bsd_forkpty" >&6; }
-if test "x$ac_cv_lib_bsd_forkpty" = xyes; then :
+if test "x$ac_cv_lib_bsd_forkpty" = x""yes; then :
   $as_echo "#define HAVE_FORKPTY 1" >>confdefs.h
  LIBS="$LIBS -lbsd"
 fi
@@ -10358,7 +10351,7 @@
 for ac_func in memmove
 do :
   ac_fn_c_check_func "$LINENO" "memmove" "ac_cv_func_memmove"
-if test "x$ac_cv_func_memmove" = xyes; then :
+if test "x$ac_cv_func_memmove" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_MEMMOVE 1
 _ACEOF
@@ -10382,7 +10375,7 @@
 
 
 ac_fn_c_check_func "$LINENO" "dup2" "ac_cv_func_dup2"
-if test "x$ac_cv_func_dup2" = xyes; then :
+if test "x$ac_cv_func_dup2" = x""yes; then :
   $as_echo "#define HAVE_DUP2 1" >>confdefs.h
 
 else
@@ -10395,7 +10388,7 @@
 fi
 
 ac_fn_c_check_func "$LINENO" "getcwd" "ac_cv_func_getcwd"
-if test "x$ac_cv_func_getcwd" = xyes; then :
+if test "x$ac_cv_func_getcwd" = x""yes; then :
   $as_echo "#define HAVE_GETCWD 1" >>confdefs.h
 
 else
@@ -10408,7 +10401,7 @@
 fi
 
 ac_fn_c_check_func "$LINENO" "strdup" "ac_cv_func_strdup"
-if test "x$ac_cv_func_strdup" = xyes; then :
+if test "x$ac_cv_func_strdup" = x""yes; then :
   $as_echo "#define HAVE_STRDUP 1" >>confdefs.h
 
 else
@@ -10424,7 +10417,7 @@
 for ac_func in getpgrp
 do :
   ac_fn_c_check_func "$LINENO" "getpgrp" "ac_cv_func_getpgrp"
-if test "x$ac_cv_func_getpgrp" = xyes; then :
+if test "x$ac_cv_func_getpgrp" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_GETPGRP 1
 _ACEOF
@@ -10452,7 +10445,7 @@
 for ac_func in setpgrp
 do :
   ac_fn_c_check_func "$LINENO" "setpgrp" "ac_cv_func_setpgrp"
-if test "x$ac_cv_func_setpgrp" = xyes; then :
+if test "x$ac_cv_func_setpgrp" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_SETPGRP 1
 _ACEOF
@@ -10480,7 +10473,7 @@
 for ac_func in gettimeofday
 do :
   ac_fn_c_check_func "$LINENO" "gettimeofday" "ac_cv_func_gettimeofday"
-if test "x$ac_cv_func_gettimeofday" = xyes; then :
+if test "x$ac_cv_func_gettimeofday" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_GETTIMEOFDAY 1
 _ACEOF
@@ -10582,7 +10575,7 @@
 then
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking getaddrinfo bug" >&5
 $as_echo_n "checking getaddrinfo bug... " >&6; }
-  if ${ac_cv_buggy_getaddrinfo+:} false; then :
+  if test "${ac_cv_buggy_getaddrinfo+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -10711,7 +10704,7 @@
 for ac_func in getnameinfo
 do :
   ac_fn_c_check_func "$LINENO" "getnameinfo" "ac_cv_func_getnameinfo"
-if test "x$ac_cv_func_getnameinfo" = xyes; then :
+if test "x$ac_cv_func_getnameinfo" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_GETNAMEINFO 1
 _ACEOF
@@ -10723,7 +10716,7 @@
 # checks for structures
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether time.h and sys/time.h may both be included" >&5
 $as_echo_n "checking whether time.h and sys/time.h may both be included... " >&6; }
-if ${ac_cv_header_time+:} false; then :
+if test "${ac_cv_header_time+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -10758,7 +10751,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether struct tm is in sys/time.h or time.h" >&5
 $as_echo_n "checking whether struct tm is in sys/time.h or time.h... " >&6; }
-if ${ac_cv_struct_tm+:} false; then :
+if test "${ac_cv_struct_tm+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -10795,7 +10788,7 @@
 #include <$ac_cv_struct_tm>
 
 "
-if test "x$ac_cv_member_struct_tm_tm_zone" = xyes; then :
+if test "x$ac_cv_member_struct_tm_tm_zone" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_TM_TM_ZONE 1
@@ -10811,7 +10804,7 @@
 else
   ac_fn_c_check_decl "$LINENO" "tzname" "ac_cv_have_decl_tzname" "#include <time.h>
 "
-if test "x$ac_cv_have_decl_tzname" = xyes; then :
+if test "x$ac_cv_have_decl_tzname" = x""yes; then :
   ac_have_decl=1
 else
   ac_have_decl=0
@@ -10823,7 +10816,7 @@
 
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for tzname" >&5
 $as_echo_n "checking for tzname... " >&6; }
-if ${ac_cv_var_tzname+:} false; then :
+if test "${ac_cv_var_tzname+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -10859,7 +10852,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_rdev" "ac_cv_member_struct_stat_st_rdev" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_rdev" = xyes; then :
+if test "x$ac_cv_member_struct_stat_st_rdev" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_RDEV 1
@@ -10869,7 +10862,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_blksize" "ac_cv_member_struct_stat_st_blksize" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_blksize" = xyes; then :
+if test "x$ac_cv_member_struct_stat_st_blksize" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_BLKSIZE 1
@@ -10879,7 +10872,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_flags" "ac_cv_member_struct_stat_st_flags" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_flags" = xyes; then :
+if test "x$ac_cv_member_struct_stat_st_flags" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_FLAGS 1
@@ -10889,7 +10882,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_gen" "ac_cv_member_struct_stat_st_gen" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_gen" = xyes; then :
+if test "x$ac_cv_member_struct_stat_st_gen" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_GEN 1
@@ -10899,7 +10892,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_birthtime" "ac_cv_member_struct_stat_st_birthtime" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_birthtime" = xyes; then :
+if test "x$ac_cv_member_struct_stat_st_birthtime" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_BIRTHTIME 1
@@ -10909,7 +10902,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_blocks" "ac_cv_member_struct_stat_st_blocks" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_blocks" = xyes; then :
+if test "x$ac_cv_member_struct_stat_st_blocks" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_BLOCKS 1
@@ -10931,7 +10924,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for time.h that defines altzone" >&5
 $as_echo_n "checking for time.h that defines altzone... " >&6; }
-if ${ac_cv_header_time_altzone+:} false; then :
+if test "${ac_cv_header_time_altzone+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -10995,7 +10988,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for addrinfo" >&5
 $as_echo_n "checking for addrinfo... " >&6; }
-if ${ac_cv_struct_addrinfo+:} false; then :
+if test "${ac_cv_struct_addrinfo+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -11027,7 +11020,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sockaddr_storage" >&5
 $as_echo_n "checking for sockaddr_storage... " >&6; }
-if ${ac_cv_struct_sockaddr_storage+:} false; then :
+if test "${ac_cv_struct_sockaddr_storage+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -11063,7 +11056,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether char is unsigned" >&5
 $as_echo_n "checking whether char is unsigned... " >&6; }
-if ${ac_cv_c_char_unsigned+:} false; then :
+if test "${ac_cv_c_char_unsigned+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -11095,7 +11088,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for an ANSI C-conforming const" >&5
 $as_echo_n "checking for an ANSI C-conforming const... " >&6; }
-if ${ac_cv_c_const+:} false; then :
+if test "${ac_cv_c_const+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -11383,7 +11376,7 @@
 
 
 ac_fn_c_check_func "$LINENO" "gethostbyname_r" "ac_cv_func_gethostbyname_r"
-if test "x$ac_cv_func_gethostbyname_r" = xyes; then :
+if test "x$ac_cv_func_gethostbyname_r" = x""yes; then :
 
   $as_echo "#define HAVE_GETHOSTBYNAME_R 1" >>confdefs.h
 
@@ -11514,7 +11507,7 @@
   for ac_func in gethostbyname
 do :
   ac_fn_c_check_func "$LINENO" "gethostbyname" "ac_cv_func_gethostbyname"
-if test "x$ac_cv_func_gethostbyname" = xyes; then :
+if test "x$ac_cv_func_gethostbyname" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_GETHOSTBYNAME 1
 _ACEOF
@@ -11536,12 +11529,12 @@
 
 # Linux requires this for correct f.p. operations
 ac_fn_c_check_func "$LINENO" "__fpu_control" "ac_cv_func___fpu_control"
-if test "x$ac_cv_func___fpu_control" = xyes; then :
+if test "x$ac_cv_func___fpu_control" = x""yes; then :
 
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for __fpu_control in -lieee" >&5
 $as_echo_n "checking for __fpu_control in -lieee... " >&6; }
-if ${ac_cv_lib_ieee___fpu_control+:} false; then :
+if test "${ac_cv_lib_ieee___fpu_control+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -11575,7 +11568,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_ieee___fpu_control" >&5
 $as_echo "$ac_cv_lib_ieee___fpu_control" >&6; }
-if test "x$ac_cv_lib_ieee___fpu_control" = xyes; then :
+if test "x$ac_cv_lib_ieee___fpu_control" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_LIBIEEE 1
 _ACEOF
@@ -11669,7 +11662,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether C doubles are little-endian IEEE 754 binary64" >&5
 $as_echo_n "checking whether C doubles are little-endian IEEE 754 binary64... " >&6; }
-if ${ac_cv_little_endian_double+:} false; then :
+if test "${ac_cv_little_endian_double+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -11711,7 +11704,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether C doubles are big-endian IEEE 754 binary64" >&5
 $as_echo_n "checking whether C doubles are big-endian IEEE 754 binary64... " >&6; }
-if ${ac_cv_big_endian_double+:} false; then :
+if test "${ac_cv_big_endian_double+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -11757,7 +11750,7 @@
 # conversions work.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether C doubles are ARM mixed-endian IEEE 754 binary64" >&5
 $as_echo_n "checking whether C doubles are ARM mixed-endian IEEE 754 binary64... " >&6; }
-if ${ac_cv_mixed_endian_double+:} false; then :
+if test "${ac_cv_mixed_endian_double+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -11927,7 +11920,7 @@
 
 ac_fn_c_check_decl "$LINENO" "isinf" "ac_cv_have_decl_isinf" "#include <math.h>
 "
-if test "x$ac_cv_have_decl_isinf" = xyes; then :
+if test "x$ac_cv_have_decl_isinf" = x""yes; then :
   ac_have_decl=1
 else
   ac_have_decl=0
@@ -11938,7 +11931,7 @@
 _ACEOF
 ac_fn_c_check_decl "$LINENO" "isnan" "ac_cv_have_decl_isnan" "#include <math.h>
 "
-if test "x$ac_cv_have_decl_isnan" = xyes; then :
+if test "x$ac_cv_have_decl_isnan" = x""yes; then :
   ac_have_decl=1
 else
   ac_have_decl=0
@@ -11949,7 +11942,7 @@
 _ACEOF
 ac_fn_c_check_decl "$LINENO" "isfinite" "ac_cv_have_decl_isfinite" "#include <math.h>
 "
-if test "x$ac_cv_have_decl_isfinite" = xyes; then :
+if test "x$ac_cv_have_decl_isfinite" = x""yes; then :
   ac_have_decl=1
 else
   ac_have_decl=0
@@ -11964,7 +11957,7 @@
 # -0. on some architectures.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether tanh preserves the sign of zero" >&5
 $as_echo_n "checking whether tanh preserves the sign of zero... " >&6; }
-if ${ac_cv_tanh_preserves_zero_sign+:} false; then :
+if test "${ac_cv_tanh_preserves_zero_sign+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -12012,7 +12005,7 @@
     # -0.  See issue #9920.
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether log1p drops the sign of negative zero" >&5
 $as_echo_n "checking whether log1p drops the sign of negative zero... " >&6; }
-    if ${ac_cv_log1p_drops_zero_sign+:} false; then :
+    if test "${ac_cv_log1p_drops_zero_sign+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -12064,7 +12057,7 @@
 # sem_open results in a 'Signal 12' error.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether POSIX semaphores are enabled" >&5
 $as_echo_n "checking whether POSIX semaphores are enabled... " >&6; }
-if ${ac_cv_posix_semaphores_enabled+:} false; then :
+if test "${ac_cv_posix_semaphores_enabled+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -12115,7 +12108,7 @@
 # Multiprocessing check for broken sem_getvalue
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for broken sem_getvalue" >&5
 $as_echo_n "checking for broken sem_getvalue... " >&6; }
-if ${ac_cv_broken_sem_getvalue+:} false; then :
+if test "${ac_cv_broken_sem_getvalue+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -12180,7 +12173,7 @@
 15|30)
   ;;
 *)
-  as_fn_error $? "bad value $enable_big_digits for --enable-big-digits; value should be 15 or 30" "$LINENO" 5 ;;
+  as_fn_error $? "bad value $enable_big_digits for --enable-big-digits; value should be 15 or 30" "$LINENO" 5  ;;
 esac
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $enable_big_digits" >&5
 $as_echo "$enable_big_digits" >&6; }
@@ -12198,7 +12191,7 @@
 
 # check for wchar.h
 ac_fn_c_check_header_mongrel "$LINENO" "wchar.h" "ac_cv_header_wchar_h" "$ac_includes_default"
-if test "x$ac_cv_header_wchar_h" = xyes; then :
+if test "x$ac_cv_header_wchar_h" = x""yes; then :
 
 
 $as_echo "#define HAVE_WCHAR_H 1" >>confdefs.h
@@ -12221,7 +12214,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of wchar_t" >&5
 $as_echo_n "checking size of wchar_t... " >&6; }
-if ${ac_cv_sizeof_wchar_t+:} false; then :
+if test "${ac_cv_sizeof_wchar_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (wchar_t))" "ac_cv_sizeof_wchar_t"        "#include <wchar.h>
@@ -12232,7 +12225,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (wchar_t)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_wchar_t=0
    fi
@@ -12287,7 +12280,7 @@
   # check whether wchar_t is signed or not
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether wchar_t is signed" >&5
 $as_echo_n "checking whether wchar_t is signed... " >&6; }
-  if ${ac_cv_wchar_t_signed+:} false; then :
+  if test "${ac_cv_wchar_t_signed+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -12383,7 +12376,7 @@
 # check for endianness
  { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether byte ordering is bigendian" >&5
 $as_echo_n "checking whether byte ordering is bigendian... " >&6; }
-if ${ac_cv_c_bigendian+:} false; then :
+if test "${ac_cv_c_bigendian+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_cv_c_bigendian=unknown
@@ -12602,7 +12595,7 @@
      ;; #(
    *)
      as_fn_error $? "unknown endianness
- presetting ac_cv_c_bigendian=no (or yes) will help" "$LINENO" 5 ;;
+ presetting ac_cv_c_bigendian=no (or yes) will help" "$LINENO" 5  ;;
  esac
 
 
@@ -12674,7 +12667,7 @@
 # or fills with zeros (like the Cray J90, according to Tim Peters).
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether right shift extends the sign bit" >&5
 $as_echo_n "checking whether right shift extends the sign bit... " >&6; }
-if ${ac_cv_rshift_extends_sign+:} false; then :
+if test "${ac_cv_rshift_extends_sign+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -12713,7 +12706,7 @@
 # check for getc_unlocked and related locking functions
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for getc_unlocked() and friends" >&5
 $as_echo_n "checking for getc_unlocked() and friends... " >&6; }
-if ${ac_cv_have_getc_unlocked+:} false; then :
+if test "${ac_cv_have_getc_unlocked+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -12811,7 +12804,7 @@
 # check for readline 2.1
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for rl_callback_handler_install in -lreadline" >&5
 $as_echo_n "checking for rl_callback_handler_install in -lreadline... " >&6; }
-if ${ac_cv_lib_readline_rl_callback_handler_install+:} false; then :
+if test "${ac_cv_lib_readline_rl_callback_handler_install+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -12845,7 +12838,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_readline_rl_callback_handler_install" >&5
 $as_echo "$ac_cv_lib_readline_rl_callback_handler_install" >&6; }
-if test "x$ac_cv_lib_readline_rl_callback_handler_install" = xyes; then :
+if test "x$ac_cv_lib_readline_rl_callback_handler_install" = x""yes; then :
 
 $as_echo "#define HAVE_RL_CALLBACK 1" >>confdefs.h
 
@@ -12897,7 +12890,7 @@
 # check for readline 4.0
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for rl_pre_input_hook in -lreadline" >&5
 $as_echo_n "checking for rl_pre_input_hook in -lreadline... " >&6; }
-if ${ac_cv_lib_readline_rl_pre_input_hook+:} false; then :
+if test "${ac_cv_lib_readline_rl_pre_input_hook+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -12931,7 +12924,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_readline_rl_pre_input_hook" >&5
 $as_echo "$ac_cv_lib_readline_rl_pre_input_hook" >&6; }
-if test "x$ac_cv_lib_readline_rl_pre_input_hook" = xyes; then :
+if test "x$ac_cv_lib_readline_rl_pre_input_hook" = x""yes; then :
 
 $as_echo "#define HAVE_RL_PRE_INPUT_HOOK 1" >>confdefs.h
 
@@ -12941,7 +12934,7 @@
 # also in 4.0
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for rl_completion_display_matches_hook in -lreadline" >&5
 $as_echo_n "checking for rl_completion_display_matches_hook in -lreadline... " >&6; }
-if ${ac_cv_lib_readline_rl_completion_display_matches_hook+:} false; then :
+if test "${ac_cv_lib_readline_rl_completion_display_matches_hook+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -12975,7 +12968,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_readline_rl_completion_display_matches_hook" >&5
 $as_echo "$ac_cv_lib_readline_rl_completion_display_matches_hook" >&6; }
-if test "x$ac_cv_lib_readline_rl_completion_display_matches_hook" = xyes; then :
+if test "x$ac_cv_lib_readline_rl_completion_display_matches_hook" = x""yes; then :
 
 $as_echo "#define HAVE_RL_COMPLETION_DISPLAY_MATCHES_HOOK 1" >>confdefs.h
 
@@ -12985,7 +12978,7 @@
 # check for readline 4.2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for rl_completion_matches in -lreadline" >&5
 $as_echo_n "checking for rl_completion_matches in -lreadline... " >&6; }
-if ${ac_cv_lib_readline_rl_completion_matches+:} false; then :
+if test "${ac_cv_lib_readline_rl_completion_matches+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -13019,7 +13012,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_readline_rl_completion_matches" >&5
 $as_echo "$ac_cv_lib_readline_rl_completion_matches" >&6; }
-if test "x$ac_cv_lib_readline_rl_completion_matches" = xyes; then :
+if test "x$ac_cv_lib_readline_rl_completion_matches" = x""yes; then :
 
 $as_echo "#define HAVE_RL_COMPLETION_MATCHES 1" >>confdefs.h
 
@@ -13060,7 +13053,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for broken nice()" >&5
 $as_echo_n "checking for broken nice()... " >&6; }
-if ${ac_cv_broken_nice+:} false; then :
+if test "${ac_cv_broken_nice+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -13101,7 +13094,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for broken poll()" >&5
 $as_echo_n "checking for broken poll()... " >&6; }
-if ${ac_cv_broken_poll+:} false; then :
+if test "${ac_cv_broken_poll+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -13156,7 +13149,7 @@
 #include <$ac_cv_struct_tm>
 
 "
-if test "x$ac_cv_member_struct_tm_tm_zone" = xyes; then :
+if test "x$ac_cv_member_struct_tm_tm_zone" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_TM_TM_ZONE 1
@@ -13172,7 +13165,7 @@
 else
   ac_fn_c_check_decl "$LINENO" "tzname" "ac_cv_have_decl_tzname" "#include <time.h>
 "
-if test "x$ac_cv_have_decl_tzname" = xyes; then :
+if test "x$ac_cv_have_decl_tzname" = x""yes; then :
   ac_have_decl=1
 else
   ac_have_decl=0
@@ -13184,7 +13177,7 @@
 
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for tzname" >&5
 $as_echo_n "checking for tzname... " >&6; }
-if ${ac_cv_var_tzname+:} false; then :
+if test "${ac_cv_var_tzname+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -13223,7 +13216,7 @@
 # check tzset(3) exists and works like we expect it to
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for working tzset()" >&5
 $as_echo_n "checking for working tzset()... " >&6; }
-if ${ac_cv_working_tzset+:} false; then :
+if test "${ac_cv_working_tzset+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -13320,7 +13313,7 @@
 # Look for subsecond timestamps in struct stat
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for tv_nsec in struct stat" >&5
 $as_echo_n "checking for tv_nsec in struct stat... " >&6; }
-if ${ac_cv_stat_tv_nsec+:} false; then :
+if test "${ac_cv_stat_tv_nsec+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -13357,7 +13350,7 @@
 # Look for BSD style subsecond timestamps in struct stat
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for tv_nsec2 in struct stat" >&5
 $as_echo_n "checking for tv_nsec2 in struct stat... " >&6; }
-if ${ac_cv_stat_tv_nsec2+:} false; then :
+if test "${ac_cv_stat_tv_nsec2+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -13394,7 +13387,7 @@
 # On HP/UX 11.0, mvwdelch is a block with a return statement
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether mvwdelch is an expression" >&5
 $as_echo_n "checking whether mvwdelch is an expression... " >&6; }
-if ${ac_cv_mvwdelch_is_expression+:} false; then :
+if test "${ac_cv_mvwdelch_is_expression+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -13431,7 +13424,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether WINDOW has _flags" >&5
 $as_echo_n "checking whether WINDOW has _flags... " >&6; }
-if ${ac_cv_window_has_flags+:} false; then :
+if test "${ac_cv_window_has_flags+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -13579,7 +13572,7 @@
 then
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for %lld and %llu printf() format support" >&5
 $as_echo_n "checking for %lld and %llu printf() format support... " >&6; }
-  if ${ac_cv_have_long_long_format+:} false; then :
+  if test "${ac_cv_have_long_long_format+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -13649,7 +13642,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for %zd printf() format support" >&5
 $as_echo_n "checking for %zd printf() format support... " >&6; }
-if ${ac_cv_have_size_t_format+:} false; then :
+if test "${ac_cv_have_size_t_format+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -13722,7 +13715,7 @@
 #endif
 
 "
-if test "x$ac_cv_type_socklen_t" = xyes; then :
+if test "x$ac_cv_type_socklen_t" = x""yes; then :
 
 else
 
@@ -13733,7 +13726,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for broken mbstowcs" >&5
 $as_echo_n "checking for broken mbstowcs... " >&6; }
-if ${ac_cv_broken_mbstowcs+:} false; then :
+if test "${ac_cv_broken_mbstowcs+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -13773,7 +13766,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC supports computed gotos" >&5
 $as_echo_n "checking whether $CC supports computed gotos... " >&6; }
-if ${ac_cv_computed_gotos+:} false; then :
+if test "${ac_cv_computed_gotos+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -13940,21 +13933,10 @@
      :end' >>confcache
 if diff "$cache_file" confcache >/dev/null 2>&1; then :; else
   if test -w "$cache_file"; then
-    if test "x$cache_file" != "x/dev/null"; then
+    test "x$cache_file" != "x/dev/null" &&
       { $as_echo "$as_me:${as_lineno-$LINENO}: updating cache $cache_file" >&5
 $as_echo "$as_me: updating cache $cache_file" >&6;}
-      if test ! -f "$cache_file" || test -h "$cache_file"; then
-	cat confcache >"$cache_file"
-      else
-        case $cache_file in #(
-        */* | ?:*)
-	  mv -f confcache "$cache_file"$$ &&
-	  mv -f "$cache_file"$$ "$cache_file" ;; #(
-        *)
-	  mv -f confcache "$cache_file" ;;
-	esac
-      fi
-    fi
+    cat confcache >$cache_file
   else
     { $as_echo "$as_me:${as_lineno-$LINENO}: not updating unwritable cache $cache_file" >&5
 $as_echo "$as_me: not updating unwritable cache $cache_file" >&6;}
@@ -13987,7 +13969,7 @@
 
 
 
-: "${CONFIG_STATUS=./config.status}"
+: ${CONFIG_STATUS=./config.status}
 ac_write_fail=0
 ac_clean_files_save=$ac_clean_files
 ac_clean_files="$ac_clean_files $CONFIG_STATUS"
@@ -14088,7 +14070,6 @@
 IFS=" ""	$as_nl"
 
 # Find who we are.  Look in the path if we contain no directory separator.
-as_myself=
 case $0 in #((
   *[\\/]* ) as_myself=$0 ;;
   *) as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
@@ -14396,7 +14377,7 @@
 # values after options handling.
 ac_log="
 This file was extended by python $as_me 3.3, which was
-generated by GNU Autoconf 2.68.  Invocation command line was
+generated by GNU Autoconf 2.67.  Invocation command line was
 
   CONFIG_FILES    = $CONFIG_FILES
   CONFIG_HEADERS  = $CONFIG_HEADERS
@@ -14458,7 +14439,7 @@
 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
 ac_cs_version="\\
 python config.status 3.3
-configured by $0, generated by GNU Autoconf 2.68,
+configured by $0, generated by GNU Autoconf 2.67,
   with options \\"\$ac_cs_config\\"
 
 Copyright (C) 2010 Free Software Foundation, Inc.
@@ -14589,7 +14570,7 @@
     "Misc/python.pc") CONFIG_FILES="$CONFIG_FILES Misc/python.pc" ;;
     "Modules/ld_so_aix") CONFIG_FILES="$CONFIG_FILES Modules/ld_so_aix" ;;
 
-  *) as_fn_error $? "invalid argument: \`$ac_config_target'" "$LINENO" 5;;
+  *) as_fn_error $? "invalid argument: \`$ac_config_target'" "$LINENO" 5 ;;
   esac
 done
 
@@ -14611,10 +14592,9 @@
 # after its creation but before its name has been assigned to `$tmp'.
 $debug ||
 {
-  tmp= ac_tmp=
+  tmp=
   trap 'exit_status=$?
-  : "${ac_tmp:=$tmp}"
-  { test ! -d "$ac_tmp" || rm -fr "$ac_tmp"; } && exit $exit_status
+  { test -z "$tmp" || test ! -d "$tmp" || rm -fr "$tmp"; } && exit $exit_status
 ' 0
   trap 'as_fn_exit 1' 1 2 13 15
 }
@@ -14622,13 +14602,12 @@
 
 {
   tmp=`(umask 077 && mktemp -d "./confXXXXXX") 2>/dev/null` &&
-  test -d "$tmp"
+  test -n "$tmp" && test -d "$tmp"
 }  ||
 {
   tmp=./conf$$-$RANDOM
   (umask 077 && mkdir "$tmp")
 } || as_fn_error $? "cannot create a temporary directory in ." "$LINENO" 5
-ac_tmp=$tmp
 
 # Set up the scripts for CONFIG_FILES section.
 # No need to generate them if there are no CONFIG_FILES.
@@ -14650,7 +14629,7 @@
   ac_cs_awk_cr=$ac_cr
 fi
 
-echo 'BEGIN {' >"$ac_tmp/subs1.awk" &&
+echo 'BEGIN {' >"$tmp/subs1.awk" &&
 _ACEOF
 
 
@@ -14678,7 +14657,7 @@
 rm -f conf$$subs.sh
 
 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
-cat >>"\$ac_tmp/subs1.awk" <<\\_ACAWK &&
+cat >>"\$tmp/subs1.awk" <<\\_ACAWK &&
 _ACEOF
 sed -n '
 h
@@ -14726,7 +14705,7 @@
 rm -f conf$$subs.awk
 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
 _ACAWK
-cat >>"\$ac_tmp/subs1.awk" <<_ACAWK &&
+cat >>"\$tmp/subs1.awk" <<_ACAWK &&
   for (key in S) S_is_set[key] = 1
   FS = ""
 
@@ -14758,7 +14737,7 @@
   sed "s/$ac_cr\$//; s/$ac_cr/$ac_cs_awk_cr/g"
 else
   cat
-fi < "$ac_tmp/subs1.awk" > "$ac_tmp/subs.awk" \
+fi < "$tmp/subs1.awk" > "$tmp/subs.awk" \
   || as_fn_error $? "could not setup config files machinery" "$LINENO" 5
 _ACEOF
 
@@ -14792,7 +14771,7 @@
 # No need to generate them if there are no CONFIG_HEADERS.
 # This happens for instance with `./config.status Makefile'.
 if test -n "$CONFIG_HEADERS"; then
-cat >"$ac_tmp/defines.awk" <<\_ACAWK ||
+cat >"$tmp/defines.awk" <<\_ACAWK ||
 BEGIN {
 _ACEOF
 
@@ -14804,8 +14783,8 @@
 # handling of long lines.
 ac_delim='%!_!# '
 for ac_last_try in false false :; do
-  ac_tt=`sed -n "/$ac_delim/p" confdefs.h`
-  if test -z "$ac_tt"; then
+  ac_t=`sed -n "/$ac_delim/p" confdefs.h`
+  if test -z "$ac_t"; then
     break
   elif $ac_last_try; then
     as_fn_error $? "could not make $CONFIG_HEADERS" "$LINENO" 5
@@ -14906,7 +14885,7 @@
   esac
   case $ac_mode$ac_tag in
   :[FHL]*:*);;
-  :L* | :C*:*) as_fn_error $? "invalid tag \`$ac_tag'" "$LINENO" 5;;
+  :L* | :C*:*) as_fn_error $? "invalid tag \`$ac_tag'" "$LINENO" 5 ;;
   :[FH]-) ac_tag=-:-;;
   :[FH]*) ac_tag=$ac_tag:$ac_tag.in;;
   esac
@@ -14925,7 +14904,7 @@
     for ac_f
     do
       case $ac_f in
-      -) ac_f="$ac_tmp/stdin";;
+      -) ac_f="$tmp/stdin";;
       *) # Look for the file first in the build tree, then in the source tree
 	 # (if the path is not absolute).  The absolute path cannot be DOS-style,
 	 # because $ac_f cannot contain `:'.
@@ -14934,7 +14913,7 @@
 	   [\\/$]*) false;;
 	   *) test -f "$srcdir/$ac_f" && ac_f="$srcdir/$ac_f";;
 	   esac ||
-	   as_fn_error 1 "cannot find input file: \`$ac_f'" "$LINENO" 5;;
+	   as_fn_error 1 "cannot find input file: \`$ac_f'" "$LINENO" 5 ;;
       esac
       case $ac_f in *\'*) ac_f=`$as_echo "$ac_f" | sed "s/'/'\\\\\\\\''/g"`;; esac
       as_fn_append ac_file_inputs " '$ac_f'"
@@ -14960,8 +14939,8 @@
     esac
 
     case $ac_tag in
-    *:-:* | *:-) cat >"$ac_tmp/stdin" \
-      || as_fn_error $? "could not create $ac_file" "$LINENO" 5 ;;
+    *:-:* | *:-) cat >"$tmp/stdin" \
+      || as_fn_error $? "could not create $ac_file" "$LINENO" 5  ;;
     esac
     ;;
   esac
@@ -15091,22 +15070,21 @@
 s&@INSTALL@&$ac_INSTALL&;t t
 $ac_datarootdir_hack
 "
-eval sed \"\$ac_sed_extra\" "$ac_file_inputs" | $AWK -f "$ac_tmp/subs.awk" \
-  >$ac_tmp/out || as_fn_error $? "could not create $ac_file" "$LINENO" 5
+eval sed \"\$ac_sed_extra\" "$ac_file_inputs" | $AWK -f "$tmp/subs.awk" >$tmp/out \
+  || as_fn_error $? "could not create $ac_file" "$LINENO" 5
 
 test -z "$ac_datarootdir_hack$ac_datarootdir_seen" &&
-  { ac_out=`sed -n '/\${datarootdir}/p' "$ac_tmp/out"`; test -n "$ac_out"; } &&
-  { ac_out=`sed -n '/^[	 ]*datarootdir[	 ]*:*=/p' \
-      "$ac_tmp/out"`; test -z "$ac_out"; } &&
+  { ac_out=`sed -n '/\${datarootdir}/p' "$tmp/out"`; test -n "$ac_out"; } &&
+  { ac_out=`sed -n '/^[	 ]*datarootdir[	 ]*:*=/p' "$tmp/out"`; test -z "$ac_out"; } &&
   { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: $ac_file contains a reference to the variable \`datarootdir'
 which seems to be undefined.  Please make sure it is defined" >&5
 $as_echo "$as_me: WARNING: $ac_file contains a reference to the variable \`datarootdir'
 which seems to be undefined.  Please make sure it is defined" >&2;}
 
-  rm -f "$ac_tmp/stdin"
+  rm -f "$tmp/stdin"
   case $ac_file in
-  -) cat "$ac_tmp/out" && rm -f "$ac_tmp/out";;
-  *) rm -f "$ac_file" && mv "$ac_tmp/out" "$ac_file";;
+  -) cat "$tmp/out" && rm -f "$tmp/out";;
+  *) rm -f "$ac_file" && mv "$tmp/out" "$ac_file";;
   esac \
   || as_fn_error $? "could not create $ac_file" "$LINENO" 5
  ;;
@@ -15117,20 +15095,20 @@
   if test x"$ac_file" != x-; then
     {
       $as_echo "/* $configure_input  */" \
-      && eval '$AWK -f "$ac_tmp/defines.awk"' "$ac_file_inputs"
-    } >"$ac_tmp/config.h" \
+      && eval '$AWK -f "$tmp/defines.awk"' "$ac_file_inputs"
+    } >"$tmp/config.h" \
       || as_fn_error $? "could not create $ac_file" "$LINENO" 5
-    if diff "$ac_file" "$ac_tmp/config.h" >/dev/null 2>&1; then
+    if diff "$ac_file" "$tmp/config.h" >/dev/null 2>&1; then
       { $as_echo "$as_me:${as_lineno-$LINENO}: $ac_file is unchanged" >&5
 $as_echo "$as_me: $ac_file is unchanged" >&6;}
     else
       rm -f "$ac_file"
-      mv "$ac_tmp/config.h" "$ac_file" \
+      mv "$tmp/config.h" "$ac_file" \
 	|| as_fn_error $? "could not create $ac_file" "$LINENO" 5
     fi
   else
     $as_echo "/* $configure_input  */" \
-      && eval '$AWK -f "$ac_tmp/defines.awk"' "$ac_file_inputs" \
+      && eval '$AWK -f "$tmp/defines.awk"' "$ac_file_inputs" \
       || as_fn_error $? "could not create -" "$LINENO" 5
   fi
  ;;
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -2538,7 +2538,8 @@
  select sem_open sem_timedwait sem_getvalue sem_unlink sendfile setegid seteuid \
  setgid sethostname \
  setlocale setregid setreuid setresuid setresgid setsid setpgid setpgrp setpriority setuid setvbuf \
- sched_setaffinity sched_setscheduler sched_setparam sched_rr_get_interval \
+ sched_get_priority_max sched_setaffinity sched_setscheduler sched_setparam \
+ sched_rr_get_interval \
  sigaction sigaltstack siginterrupt sigpending sigrelse \
  sigtimedwait sigwait sigwaitinfo snprintf strftime strlcpy symlinkat sync \
  sysconf tcgetpgrp tcsetpgrp tempnam timegm times tmpfile tmpnam tmpnam_r \
diff --git a/pyconfig.h.in b/pyconfig.h.in
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -653,6 +653,9 @@
 /* Define to 1 if you have the `round' function. */
 #undef HAVE_ROUND
 
+/* Define to 1 if you have the `sched_get_priority_max' function. */
+#undef HAVE_SCHED_GET_PRIORITY_MAX
+
 /* Define to 1 if you have the <sched.h> header file. */
 #undef HAVE_SCHED_H
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Wed Sep  7 05:14:19 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Wed, 07 Sep 2011 05:14:19 +0200
Subject: [Python-checkins] Daily reference leaks (5c8b6e03ebfe): sum=0
Message-ID: <E1R18ah-0003XA-Sg@ap.vmr.nerim.net>

results for 5c8b6e03ebfe on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogeWZgwi', '-x']

From python-checkins at python.org  Wed Sep  7 16:19:07 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 07 Sep 2011 16:19:07 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312929=3A_faulthand?=
 =?utf8?q?ler_now_uses_char*_for_arithmetic_on_pointers?=
Message-ID: <E1R1Iy3-0005dq-Rl@dinsdale.python.org>

http://hg.python.org/cpython/rev/e91ad9669c08
changeset:   72311:e91ad9669c08
user:        Victor Stinner <vstinner at wyplay.com>
date:        Wed Sep 07 16:18:56 2011 +0200
summary:
  Issue #12929: faulthandler now uses char* for arithmetic on pointers

instead of void*

files:
  Modules/faulthandler.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Modules/faulthandler.c b/Modules/faulthandler.c
--- a/Modules/faulthandler.c
+++ b/Modules/faulthandler.c
@@ -904,7 +904,7 @@
 faulthandler_stack_overflow(PyObject *self)
 {
     size_t depth, size;
-    void *sp = &depth, *stop;
+    char *sp = (char *)&depth, *stop;
 
     depth = 0;
     stop = stack_overflow(sp - STACK_OVERFLOW_MAX_SIZE,

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep  7 21:41:18 2011
From: python-checkins at python.org (nadeem.vawda)
Date: Wed, 07 Sep 2011 21:41:18 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312909=3A_Make_PyLo?=
 =?utf8?q?ng=5FAs*_functions_consistent_in_their_use_of_exceptions=2E?=
Message-ID: <E1R1Nzq-0006GC-Sy@dinsdale.python.org>

http://hg.python.org/cpython/rev/4a66a35da3fd
changeset:   72312:4a66a35da3fd
user:        Nadeem Vawda <nadeem.vawda at gmail.com>
date:        Wed Sep 07 21:40:26 2011 +0200
summary:
  Issue #12909: Make PyLong_As* functions consistent in their use of exceptions.

PyLong_AsDouble() and PyLong_AsUnsignedLongLong() now raise TypeError (rather
than SystemError) when passed a non-integer argument, matching the behavior of
all the other PyLong_As*() functions.

files:
  Modules/_testcapimodule.c |  64 +++++++++++++++++++++++++++
  Modules/testcapi_long.h   |  26 ++++++++++
  Objects/longobject.c      |  12 ++++-
  3 files changed, 100 insertions(+), 2 deletions(-)


diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -769,6 +769,68 @@
     return Py_None;
 }
 
+/* Test the PyLong_As{Size,Ssize}_t API. At present this just tests that
+   non-integer arguments are handled correctly. It should be extended to
+   test overflow handling.
+ */
+
+static PyObject *
+test_long_as_size_t(PyObject *self)
+{
+    size_t out_u;
+    Py_ssize_t out_s;
+
+    Py_INCREF(Py_None);
+
+    out_u = PyLong_AsSize_t(Py_None);
+    if (out_u != (size_t)-1 || !PyErr_Occurred())
+        return raiseTestError("test_long_as_size_t",
+                              "PyLong_AsSize_t(None) didn't complain");
+    if (!PyErr_ExceptionMatches(PyExc_TypeError))
+        return raiseTestError("test_long_as_size_t",
+                              "PyLong_AsSize_t(None) raised "
+                              "something other than TypeError");
+    PyErr_Clear();
+
+    out_s = PyLong_AsSsize_t(Py_None);
+    if (out_s != (Py_ssize_t)-1 || !PyErr_Occurred())
+        return raiseTestError("test_long_as_size_t",
+                              "PyLong_AsSsize_t(None) didn't complain");
+    if (!PyErr_ExceptionMatches(PyExc_TypeError))
+        return raiseTestError("test_long_as_size_t",
+                              "PyLong_AsSsize_t(None) raised "
+                              "something other than TypeError");
+    PyErr_Clear();
+
+    /* Py_INCREF(Py_None) omitted - we already have a reference to it. */
+    return Py_None;
+}
+
+/* Test the PyLong_AsDouble API. At present this just tests that
+   non-integer arguments are handled correctly.
+ */
+
+static PyObject *
+test_long_as_double(PyObject *self)
+{
+    double out;
+
+    Py_INCREF(Py_None);
+
+    out = PyLong_AsDouble(Py_None);
+    if (out != -1.0 || !PyErr_Occurred())
+        return raiseTestError("test_long_as_double",
+                              "PyLong_AsDouble(None) didn't complain");
+    if (!PyErr_ExceptionMatches(PyExc_TypeError))
+        return raiseTestError("test_long_as_double",
+                              "PyLong_AsDouble(None) raised "
+                              "something other than TypeError");
+    PyErr_Clear();
+
+    /* Py_INCREF(Py_None) omitted - we already have a reference to it. */
+    return Py_None;
+}
+
 /* Test the L code for PyArg_ParseTuple.  This should deliver a PY_LONG_LONG
    for both long and int arguments.  The test may leak a little memory if
    it fails.
@@ -2267,6 +2329,8 @@
     {"test_long_api",           (PyCFunction)test_long_api,      METH_NOARGS},
     {"test_long_and_overflow", (PyCFunction)test_long_and_overflow,
      METH_NOARGS},
+    {"test_long_as_double",     (PyCFunction)test_long_as_double,METH_NOARGS},
+    {"test_long_as_size_t",     (PyCFunction)test_long_as_size_t,METH_NOARGS},
     {"test_long_numbits",       (PyCFunction)test_long_numbits,  METH_NOARGS},
     {"test_k_code",             (PyCFunction)test_k_code,        METH_NOARGS},
     {"test_empty_argparse", (PyCFunction)test_empty_argparse,METH_NOARGS},
diff --git a/Modules/testcapi_long.h b/Modules/testcapi_long.h
--- a/Modules/testcapi_long.h
+++ b/Modules/testcapi_long.h
@@ -177,6 +177,32 @@
         Py_DECREF(one);
     }
 
+    /* Test F_PY_TO_{S,U} on non-pylong input. This should raise a TypeError. */
+    {
+        TYPENAME out;
+        unsigned TYPENAME uout;
+
+        Py_INCREF(Py_None);
+
+        out = F_PY_TO_S(Py_None);
+        if (out != (TYPENAME)-1 || !PyErr_Occurred())
+            return error("PyLong_AsXXX(None) didn't complain");
+        if (!PyErr_ExceptionMatches(PyExc_TypeError))
+            return error("PyLong_AsXXX(None) raised "
+                         "something other than TypeError");
+        PyErr_Clear();
+
+        uout = F_PY_TO_U(Py_None);
+        if (uout != (unsigned TYPENAME)-1 || !PyErr_Occurred())
+            return error("PyLong_AsXXX(None) didn't complain");
+        if (!PyErr_ExceptionMatches(PyExc_TypeError))
+            return error("PyLong_AsXXX(None) raised "
+                         "something other than TypeError");
+        PyErr_Clear();
+
+        Py_DECREF(Py_None);
+    }
+
     Py_INCREF(Py_None);
     return Py_None;
 }
diff --git a/Objects/longobject.c b/Objects/longobject.c
--- a/Objects/longobject.c
+++ b/Objects/longobject.c
@@ -1193,10 +1193,14 @@
     int one = 1;
     int res;
 
-    if (vv == NULL || !PyLong_Check(vv)) {
+    if (vv == NULL) {
         PyErr_BadInternalCall();
         return (unsigned PY_LONG_LONG)-1;
     }
+    if (!PyLong_Check(vv)) {
+        PyErr_SetString(PyExc_TypeError, "an integer is required");
+        return (unsigned PY_LONG_LONG)-1;
+    }
 
     v = (PyLongObject*)vv;
     switch(Py_SIZE(v)) {
@@ -2481,10 +2485,14 @@
     Py_ssize_t exponent;
     double x;
 
-    if (v == NULL || !PyLong_Check(v)) {
+    if (v == NULL) {
         PyErr_BadInternalCall();
         return -1.0;
     }
+    if (!PyLong_Check(v)) {
+        PyErr_SetString(PyExc_TypeError, "an integer is required");
+        return -1.0;
+    }
     x = _PyLong_Frexp((PyLongObject *)v, &exponent);
     if ((x == -1.0 && PyErr_Occurred()) || exponent > DBL_MAX_EXP) {
         PyErr_SetString(PyExc_OverflowError,

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep  7 22:30:35 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 07 Sep 2011 22:30:35 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312852=3A_Set_=5FXO?=
 =?utf8?q?PEN=5FSOURCE_to_700_to_get_POSIX_2008?=
Message-ID: <E1R1OlX-0002ST-12@dinsdale.python.org>

http://hg.python.org/cpython/rev/cf66578d03d1
changeset:   72313:cf66578d03d1
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 07 22:29:43 2011 +0200
summary:
  Issue #12852: Set _XOPEN_SOURCE to 700 to get POSIX 2008

configure: Set _XOPEN_SOURCE to 700, instead of 600, to get POSIX 2008
functions on OpenBSD (e.g. fdopendir).

files:
  Misc/NEWS    |  3 +++
  configure    |  5 +++--
  configure.in |  3 ++-
  3 files changed, 8 insertions(+), 3 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1209,6 +1209,9 @@
 Build
 -----
 
+- Issue #12852: Set _XOPEN_SOURCE to 700, instead of 600, to get POSIX 2008
+  functions on OpenBSD (e.g. fdopendir).
+
 - Issue #11863: Remove support for legacy systems deprecated in Python 3.2
   (following PEP 11).  These systems are systems using Mach C Threads,
   SunOS lightweight processes, GNU pth threads and IRIX threads.
diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -3140,8 +3140,9 @@
 
 if test $define_xopen_source = yes
 then
-
-$as_echo "#define _XOPEN_SOURCE 600" >>confdefs.h
+  # X/Open 7, incorporating POSIX.1-2008
+
+$as_echo "#define _XOPEN_SOURCE 700" >>confdefs.h
 
 
   # On Tru64 Unix 4.0F, defining _XOPEN_SOURCE also requires
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -397,7 +397,8 @@
 
 if test $define_xopen_source = yes
 then
-  AC_DEFINE(_XOPEN_SOURCE, 600, 
+  # X/Open 7, incorporating POSIX.1-2008
+  AC_DEFINE(_XOPEN_SOURCE, 700,
             Define to the level of X/Open that your system supports)
 
   # On Tru64 Unix 4.0F, defining _XOPEN_SOURCE also requires

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep  8 00:56:48 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 08 Sep 2011 00:56:48 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312852=3A_Set_=5FPO?=
 =?utf8?q?SIX=5FC=5FSOURCE_to_200809_to_get_POSIX_2008?=
Message-ID: <E1R1R32-0006Re-A4@dinsdale.python.org>

http://hg.python.org/cpython/rev/92842e347d98
changeset:   72314:92842e347d98
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 08 00:56:17 2011 +0200
summary:
  Issue #12852: Set _POSIX_C_SOURCE to 200809 to get POSIX 2008

configure.in: Set _POSIX_C_SOURCE to 200809L, instead of 200112L, to activate
features from IEEE Stds 1003.1-2008.

files:
  configure    |  592 ++++++++++++++++++++------------------
  configure.in |    3 +-
  2 files changed, 308 insertions(+), 287 deletions(-)


diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -1,6 +1,6 @@
 #! /bin/sh
 # Guess values for system-dependent variables and create Makefiles.
-# Generated by GNU Autoconf 2.67 for python 3.3.
+# Generated by GNU Autoconf 2.68 for python 3.3.
 #
 # Report bugs to <http://bugs.python.org/>.
 #
@@ -91,6 +91,7 @@
 IFS=" ""	$as_nl"
 
 # Find who we are.  Look in the path if we contain no directory separator.
+as_myself=
 case $0 in #((
   *[\\/]* ) as_myself=$0 ;;
   *) as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
@@ -216,11 +217,18 @@
   # We cannot yet assume a decent shell, so we have to provide a
 	# neutralization value for shells without unset; and this also
 	# works around shells that cannot unset nonexistent variables.
+	# Preserve -v and -x to the replacement shell.
 	BASH_ENV=/dev/null
 	ENV=/dev/null
 	(unset BASH_ENV) >/dev/null 2>&1 && unset BASH_ENV ENV
 	export CONFIG_SHELL
-	exec "$CONFIG_SHELL" "$as_myself" ${1+"$@"}
+	case $- in # ((((
+	  *v*x* | *x*v* ) as_opts=-vx ;;
+	  *v* ) as_opts=-v ;;
+	  *x* ) as_opts=-x ;;
+	  * ) as_opts= ;;
+	esac
+	exec "$CONFIG_SHELL" $as_opts "$as_myself" ${1+"$@"}
 fi
 
     if test x$as_have_required = xno; then :
@@ -1175,7 +1183,7 @@
     $as_echo "$as_me: WARNING: you should use --build, --host, --target" >&2
     expr "x$ac_option" : ".*[^-._$as_cr_alnum]" >/dev/null &&
       $as_echo "$as_me: WARNING: invalid host type: $ac_option" >&2
-    : ${build_alias=$ac_option} ${host_alias=$ac_option} ${target_alias=$ac_option}
+    : "${build_alias=$ac_option} ${host_alias=$ac_option} ${target_alias=$ac_option}"
     ;;
 
   esac
@@ -1512,7 +1520,7 @@
 if $ac_init_version; then
   cat <<\_ACEOF
 python configure 3.3
-generated by GNU Autoconf 2.67
+generated by GNU Autoconf 2.68
 
 Copyright (C) 2010 Free Software Foundation, Inc.
 This configure script is free software; the Free Software Foundation
@@ -1558,7 +1566,7 @@
 
 	ac_retval=1
 fi
-  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
+  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
   as_fn_set_status $ac_retval
 
 } # ac_fn_c_try_compile
@@ -1604,7 +1612,7 @@
   # interfere with the next link command; also delete a directory that is
   # left behind by Apple's compiler.  We do this before executing the actions.
   rm -rf conftest.dSYM conftest_ipa8_conftest.oo
-  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
+  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
   as_fn_set_status $ac_retval
 
 } # ac_fn_c_try_link
@@ -1641,7 +1649,7 @@
 
     ac_retval=1
 fi
-  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
+  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
   as_fn_set_status $ac_retval
 
 } # ac_fn_c_try_cpp
@@ -1654,10 +1662,10 @@
 ac_fn_c_check_header_mongrel ()
 {
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
-  if eval "test \"\${$3+set}\"" = set; then :
+  if eval \${$3+:} false; then :
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2" >&5
 $as_echo_n "checking for $2... " >&6; }
-if eval "test \"\${$3+set}\"" = set; then :
+if eval \${$3+:} false; then :
   $as_echo_n "(cached) " >&6
 fi
 eval ac_res=\$$3
@@ -1724,7 +1732,7 @@
 esac
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2" >&5
 $as_echo_n "checking for $2... " >&6; }
-if eval "test \"\${$3+set}\"" = set; then :
+if eval \${$3+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   eval "$3=\$ac_header_compiler"
@@ -1733,7 +1741,7 @@
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
 fi
-  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
+  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
 
 } # ac_fn_c_check_header_mongrel
 
@@ -1774,7 +1782,7 @@
        ac_retval=$ac_status
 fi
   rm -rf conftest.dSYM conftest_ipa8_conftest.oo
-  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
+  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
   as_fn_set_status $ac_retval
 
 } # ac_fn_c_try_run
@@ -1788,7 +1796,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2" >&5
 $as_echo_n "checking for $2... " >&6; }
-if eval "test \"\${$3+set}\"" = set; then :
+if eval \${$3+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -1806,7 +1814,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
+  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
 
 } # ac_fn_c_check_header_compile
 
@@ -1819,7 +1827,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2" >&5
 $as_echo_n "checking for $2... " >&6; }
-if eval "test \"\${$3+set}\"" = set; then :
+if eval \${$3+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   eval "$3=no"
@@ -1860,7 +1868,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
+  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
 
 } # ac_fn_c_check_type
 
@@ -1873,7 +1881,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for uint$2_t" >&5
 $as_echo_n "checking for uint$2_t... " >&6; }
-if eval "test \"\${$3+set}\"" = set; then :
+if eval \${$3+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   eval "$3=no"
@@ -1913,7 +1921,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
+  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
 
 } # ac_fn_c_find_uintX_t
 
@@ -1926,7 +1934,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for int$2_t" >&5
 $as_echo_n "checking for int$2_t... " >&6; }
-if eval "test \"\${$3+set}\"" = set; then :
+if eval \${$3+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   eval "$3=no"
@@ -1987,7 +1995,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
+  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
 
 } # ac_fn_c_find_intX_t
 
@@ -2164,7 +2172,7 @@
 rm -f conftest.val
 
   fi
-  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
+  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
   as_fn_set_status $ac_retval
 
 } # ac_fn_c_compute_int
@@ -2177,7 +2185,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2" >&5
 $as_echo_n "checking for $2... " >&6; }
-if eval "test \"\${$3+set}\"" = set; then :
+if eval \${$3+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -2232,7 +2240,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
+  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
 
 } # ac_fn_c_check_func
 
@@ -2245,7 +2253,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2.$3" >&5
 $as_echo_n "checking for $2.$3... " >&6; }
-if eval "test \"\${$4+set}\"" = set; then :
+if eval \${$4+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -2289,7 +2297,7 @@
 eval ac_res=\$$4
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
+  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
 
 } # ac_fn_c_check_member
 
@@ -2304,7 +2312,7 @@
   as_decl_use=`echo $2|sed -e 's/(/((/' -e 's/)/) 0&/' -e 's/,/) 0& (/g'`
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $as_decl_name is declared" >&5
 $as_echo_n "checking whether $as_decl_name is declared... " >&6; }
-if eval "test \"\${$3+set}\"" = set; then :
+if eval \${$3+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -2335,7 +2343,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
+  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
 
 } # ac_fn_c_check_decl
 cat >config.log <<_ACEOF
@@ -2343,7 +2351,7 @@
 running configure, to aid debugging if configure makes a mistake.
 
 It was created by python $as_me 3.3, which was
-generated by GNU Autoconf 2.67.  Invocation command line was
+generated by GNU Autoconf 2.68.  Invocation command line was
 
   $ $0 $@
 
@@ -2601,7 +2609,7 @@
       || { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "failed to load site script $ac_site_file
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
   fi
 done
 
@@ -2701,7 +2709,7 @@
 set dummy hg; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if test "${ac_cv_prog_HAS_HG+set}" = set; then :
+if ${ac_cv_prog_HAS_HG+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$HAS_HG"; then
@@ -3154,8 +3162,7 @@
 
 
 
-$as_echo "#define _POSIX_C_SOURCE 200112L" >>confdefs.h
-
+$as_echo "#define _POSIX_C_SOURCE 200809L" >>confdefs.h
 
 fi
 
@@ -3251,7 +3258,7 @@
 set dummy ${ac_tool_prefix}gcc; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if test "${ac_cv_prog_CC+set}" = set; then :
+if ${ac_cv_prog_CC+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$CC"; then
@@ -3291,7 +3298,7 @@
 set dummy gcc; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if test "${ac_cv_prog_ac_ct_CC+set}" = set; then :
+if ${ac_cv_prog_ac_ct_CC+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$ac_ct_CC"; then
@@ -3344,7 +3351,7 @@
 set dummy ${ac_tool_prefix}cc; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if test "${ac_cv_prog_CC+set}" = set; then :
+if ${ac_cv_prog_CC+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$CC"; then
@@ -3384,7 +3391,7 @@
 set dummy cc; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if test "${ac_cv_prog_CC+set}" = set; then :
+if ${ac_cv_prog_CC+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$CC"; then
@@ -3443,7 +3450,7 @@
 set dummy $ac_tool_prefix$ac_prog; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if test "${ac_cv_prog_CC+set}" = set; then :
+if ${ac_cv_prog_CC+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$CC"; then
@@ -3487,7 +3494,7 @@
 set dummy $ac_prog; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if test "${ac_cv_prog_ac_ct_CC+set}" = set; then :
+if ${ac_cv_prog_ac_ct_CC+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$ac_ct_CC"; then
@@ -3542,7 +3549,7 @@
 test -z "$CC" && { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "no acceptable C compiler found in \$PATH
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
 
 # Provide some information about the compiler.
 $as_echo "$as_me:${as_lineno-$LINENO}: checking for C compiler version" >&5
@@ -3657,7 +3664,7 @@
 { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "C compiler cannot create executables
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
 $as_echo "yes" >&6; }
@@ -3700,7 +3707,7 @@
   { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "cannot compute suffix of executables: cannot compile and link
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
 fi
 rm -f conftest conftest$ac_cv_exeext
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_exeext" >&5
@@ -3759,7 +3766,7 @@
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "cannot run C compiled programs.
 If you meant to cross compile, use \`--host'.
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
     fi
   fi
 fi
@@ -3770,7 +3777,7 @@
 ac_clean_files=$ac_clean_files_save
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for suffix of object files" >&5
 $as_echo_n "checking for suffix of object files... " >&6; }
-if test "${ac_cv_objext+set}" = set; then :
+if ${ac_cv_objext+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -3811,7 +3818,7 @@
 { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "cannot compute suffix of object files: cannot compile
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
 fi
 rm -f conftest.$ac_cv_objext conftest.$ac_ext
 fi
@@ -3821,7 +3828,7 @@
 ac_objext=$OBJEXT
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether we are using the GNU C compiler" >&5
 $as_echo_n "checking whether we are using the GNU C compiler... " >&6; }
-if test "${ac_cv_c_compiler_gnu+set}" = set; then :
+if ${ac_cv_c_compiler_gnu+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -3858,7 +3865,7 @@
 ac_save_CFLAGS=$CFLAGS
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC accepts -g" >&5
 $as_echo_n "checking whether $CC accepts -g... " >&6; }
-if test "${ac_cv_prog_cc_g+set}" = set; then :
+if ${ac_cv_prog_cc_g+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_save_c_werror_flag=$ac_c_werror_flag
@@ -3936,7 +3943,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $CC option to accept ISO C89" >&5
 $as_echo_n "checking for $CC option to accept ISO C89... " >&6; }
-if test "${ac_cv_prog_cc_c89+set}" = set; then :
+if ${ac_cv_prog_cc_c89+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_cv_prog_cc_c89=no
@@ -4071,7 +4078,7 @@
 set dummy g++; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if test "${ac_cv_path_CXX+set}" = set; then :
+if ${ac_cv_path_CXX+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   case $CXX in
@@ -4112,7 +4119,7 @@
 set dummy c++; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if test "${ac_cv_path_CXX+set}" = set; then :
+if ${ac_cv_path_CXX+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   case $CXX in
@@ -4163,7 +4170,7 @@
 set dummy $ac_prog; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if test "${ac_cv_prog_CXX+set}" = set; then :
+if ${ac_cv_prog_CXX+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$CXX"; then
@@ -4264,7 +4271,7 @@
   CPP=
 fi
 if test -z "$CPP"; then
-  if test "${ac_cv_prog_CPP+set}" = set; then :
+  if ${ac_cv_prog_CPP+:} false; then :
   $as_echo_n "(cached) " >&6
 else
       # Double quotes because CPP needs to be expanded
@@ -4380,7 +4387,7 @@
   { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "C preprocessor \"$CPP\" fails sanity check
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
 fi
 
 ac_ext=c
@@ -4392,7 +4399,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for grep that handles long lines and -e" >&5
 $as_echo_n "checking for grep that handles long lines and -e... " >&6; }
-if test "${ac_cv_path_GREP+set}" = set; then :
+if ${ac_cv_path_GREP+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test -z "$GREP"; then
@@ -4455,7 +4462,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for egrep" >&5
 $as_echo_n "checking for egrep... " >&6; }
-if test "${ac_cv_path_EGREP+set}" = set; then :
+if ${ac_cv_path_EGREP+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if echo a | $GREP -E '(a|b)' >/dev/null 2>&1
@@ -4522,7 +4529,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ANSI C header files" >&5
 $as_echo_n "checking for ANSI C header files... " >&6; }
-if test "${ac_cv_header_stdc+set}" = set; then :
+if ${ac_cv_header_stdc+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -4651,7 +4658,7 @@
 
 
   ac_fn_c_check_header_mongrel "$LINENO" "minix/config.h" "ac_cv_header_minix_config_h" "$ac_includes_default"
-if test "x$ac_cv_header_minix_config_h" = x""yes; then :
+if test "x$ac_cv_header_minix_config_h" = xyes; then :
   MINIX=yes
 else
   MINIX=
@@ -4673,7 +4680,7 @@
 
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether it is safe to define __EXTENSIONS__" >&5
 $as_echo_n "checking whether it is safe to define __EXTENSIONS__... " >&6; }
-if test "${ac_cv_safe_to_define___extensions__+set}" = set; then :
+if ${ac_cv_safe_to_define___extensions__+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -4866,7 +4873,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for inline" >&5
 $as_echo_n "checking for inline... " >&6; }
-if test "${ac_cv_c_inline+set}" = set; then :
+if ${ac_cv_c_inline+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_cv_c_inline=no
@@ -5062,7 +5069,7 @@
 set dummy ${ac_tool_prefix}ranlib; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if test "${ac_cv_prog_RANLIB+set}" = set; then :
+if ${ac_cv_prog_RANLIB+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$RANLIB"; then
@@ -5102,7 +5109,7 @@
 set dummy ranlib; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if test "${ac_cv_prog_ac_ct_RANLIB+set}" = set; then :
+if ${ac_cv_prog_ac_ct_RANLIB+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$ac_ct_RANLIB"; then
@@ -5156,7 +5163,7 @@
 set dummy $ac_prog; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if test "${ac_cv_prog_AR+set}" = set; then :
+if ${ac_cv_prog_AR+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$AR"; then
@@ -5207,7 +5214,7 @@
 set dummy python; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if test "${ac_cv_prog_HAS_PYTHON+set}" = set; then :
+if ${ac_cv_prog_HAS_PYTHON+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$HAS_PYTHON"; then
@@ -5301,7 +5308,7 @@
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for a BSD-compatible install" >&5
 $as_echo_n "checking for a BSD-compatible install... " >&6; }
 if test -z "$INSTALL"; then
-if test "${ac_cv_path_install+set}" = set; then :
+if ${ac_cv_path_install+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
@@ -5487,7 +5494,7 @@
      ac_save_cc="$CC"
      CC="$CC -fno-strict-aliasing"
      save_CFLAGS="$CFLAGS"
-     if test "${ac_cv_no_strict_aliasing+set}" = set; then :
+     if ${ac_cv_no_strict_aliasing+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -5737,7 +5744,7 @@
 # options before we can check whether -Kpthread improves anything.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether pthreads are available without options" >&5
 $as_echo_n "checking whether pthreads are available without options... " >&6; }
-if test "${ac_cv_pthread_is_default+set}" = set; then :
+if ${ac_cv_pthread_is_default+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -5790,7 +5797,7 @@
 # function available.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC accepts -Kpthread" >&5
 $as_echo_n "checking whether $CC accepts -Kpthread... " >&6; }
-if test "${ac_cv_kpthread+set}" = set; then :
+if ${ac_cv_kpthread+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_save_cc="$CC"
@@ -5839,7 +5846,7 @@
 # function available.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC accepts -Kthread" >&5
 $as_echo_n "checking whether $CC accepts -Kthread... " >&6; }
-if test "${ac_cv_kthread+set}" = set; then :
+if ${ac_cv_kthread+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_save_cc="$CC"
@@ -5888,7 +5895,7 @@
 # function available.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC accepts -pthread" >&5
 $as_echo_n "checking whether $CC accepts -pthread... " >&6; }
-if test "${ac_cv_thread+set}" = set; then :
+if ${ac_cv_thread+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_save_cc="$CC"
@@ -5973,7 +5980,7 @@
 # checks for header files
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ANSI C header files" >&5
 $as_echo_n "checking for ANSI C header files... " >&6; }
-if test "${ac_cv_header_stdc+set}" = set; then :
+if ${ac_cv_header_stdc+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -6112,7 +6119,7 @@
   as_ac_Header=`$as_echo "ac_cv_header_dirent_$ac_hdr" | $as_tr_sh`
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_hdr that defines DIR" >&5
 $as_echo_n "checking for $ac_hdr that defines DIR... " >&6; }
-if eval "test \"\${$as_ac_Header+set}\"" = set; then :
+if eval \${$as_ac_Header+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -6152,7 +6159,7 @@
 if test $ac_header_dirent = dirent.h; then
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing opendir" >&5
 $as_echo_n "checking for library containing opendir... " >&6; }
-if test "${ac_cv_search_opendir+set}" = set; then :
+if ${ac_cv_search_opendir+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_func_search_save_LIBS=$LIBS
@@ -6186,11 +6193,11 @@
 fi
 rm -f core conftest.err conftest.$ac_objext \
     conftest$ac_exeext
-  if test "${ac_cv_search_opendir+set}" = set; then :
+  if ${ac_cv_search_opendir+:} false; then :
   break
 fi
 done
-if test "${ac_cv_search_opendir+set}" = set; then :
+if ${ac_cv_search_opendir+:} false; then :
 
 else
   ac_cv_search_opendir=no
@@ -6209,7 +6216,7 @@
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing opendir" >&5
 $as_echo_n "checking for library containing opendir... " >&6; }
-if test "${ac_cv_search_opendir+set}" = set; then :
+if ${ac_cv_search_opendir+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_func_search_save_LIBS=$LIBS
@@ -6243,11 +6250,11 @@
 fi
 rm -f core conftest.err conftest.$ac_objext \
     conftest$ac_exeext
-  if test "${ac_cv_search_opendir+set}" = set; then :
+  if ${ac_cv_search_opendir+:} false; then :
   break
 fi
 done
-if test "${ac_cv_search_opendir+set}" = set; then :
+if ${ac_cv_search_opendir+:} false; then :
 
 else
   ac_cv_search_opendir=no
@@ -6267,7 +6274,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether sys/types.h defines makedev" >&5
 $as_echo_n "checking whether sys/types.h defines makedev... " >&6; }
-if test "${ac_cv_header_sys_types_h_makedev+set}" = set; then :
+if ${ac_cv_header_sys_types_h_makedev+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -6295,7 +6302,7 @@
 
 if test $ac_cv_header_sys_types_h_makedev = no; then
 ac_fn_c_check_header_mongrel "$LINENO" "sys/mkdev.h" "ac_cv_header_sys_mkdev_h" "$ac_includes_default"
-if test "x$ac_cv_header_sys_mkdev_h" = x""yes; then :
+if test "x$ac_cv_header_sys_mkdev_h" = xyes; then :
 
 $as_echo "#define MAJOR_IN_MKDEV 1" >>confdefs.h
 
@@ -6305,7 +6312,7 @@
 
   if test $ac_cv_header_sys_mkdev_h = no; then
     ac_fn_c_check_header_mongrel "$LINENO" "sys/sysmacros.h" "ac_cv_header_sys_sysmacros_h" "$ac_includes_default"
-if test "x$ac_cv_header_sys_sysmacros_h" = x""yes; then :
+if test "x$ac_cv_header_sys_sysmacros_h" = xyes; then :
 
 $as_echo "#define MAJOR_IN_SYSMACROS 1" >>confdefs.h
 
@@ -6333,7 +6340,7 @@
 #endif
 
 "
-if test "x$ac_cv_header_net_if_h" = x""yes; then :
+if test "x$ac_cv_header_net_if_h" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_NET_IF_H 1
 _ACEOF
@@ -6353,7 +6360,7 @@
 #endif
 
 "
-if test "x$ac_cv_header_term_h" = x""yes; then :
+if test "x$ac_cv_header_term_h" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_TERM_H 1
 _ACEOF
@@ -6375,7 +6382,7 @@
 #endif
 
 "
-if test "x$ac_cv_header_linux_netlink_h" = x""yes; then :
+if test "x$ac_cv_header_linux_netlink_h" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_LINUX_NETLINK_H 1
 _ACEOF
@@ -6511,7 +6518,7 @@
 
 # Type availability checks
 ac_fn_c_check_type "$LINENO" "mode_t" "ac_cv_type_mode_t" "$ac_includes_default"
-if test "x$ac_cv_type_mode_t" = x""yes; then :
+if test "x$ac_cv_type_mode_t" = xyes; then :
 
 else
 
@@ -6522,7 +6529,7 @@
 fi
 
 ac_fn_c_check_type "$LINENO" "off_t" "ac_cv_type_off_t" "$ac_includes_default"
-if test "x$ac_cv_type_off_t" = x""yes; then :
+if test "x$ac_cv_type_off_t" = xyes; then :
 
 else
 
@@ -6533,7 +6540,7 @@
 fi
 
 ac_fn_c_check_type "$LINENO" "pid_t" "ac_cv_type_pid_t" "$ac_includes_default"
-if test "x$ac_cv_type_pid_t" = x""yes; then :
+if test "x$ac_cv_type_pid_t" = xyes; then :
 
 else
 
@@ -6549,7 +6556,7 @@
 _ACEOF
 
 ac_fn_c_check_type "$LINENO" "size_t" "ac_cv_type_size_t" "$ac_includes_default"
-if test "x$ac_cv_type_size_t" = x""yes; then :
+if test "x$ac_cv_type_size_t" = xyes; then :
 
 else
 
@@ -6561,7 +6568,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for uid_t in sys/types.h" >&5
 $as_echo_n "checking for uid_t in sys/types.h... " >&6; }
-if test "${ac_cv_type_uid_t+set}" = set; then :
+if ${ac_cv_type_uid_t+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -6640,7 +6647,7 @@
 esac
 
 ac_fn_c_check_type "$LINENO" "ssize_t" "ac_cv_type_ssize_t" "$ac_includes_default"
-if test "x$ac_cv_type_ssize_t" = x""yes; then :
+if test "x$ac_cv_type_ssize_t" = xyes; then :
 
 $as_echo "#define HAVE_SSIZE_T 1" >>confdefs.h
 
@@ -6655,7 +6662,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of int" >&5
 $as_echo_n "checking size of int... " >&6; }
-if test "${ac_cv_sizeof_int+set}" = set; then :
+if ${ac_cv_sizeof_int+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (int))" "ac_cv_sizeof_int"        "$ac_includes_default"; then :
@@ -6665,7 +6672,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (int)
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
    else
      ac_cv_sizeof_int=0
    fi
@@ -6688,7 +6695,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of long" >&5
 $as_echo_n "checking size of long... " >&6; }
-if test "${ac_cv_sizeof_long+set}" = set; then :
+if ${ac_cv_sizeof_long+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (long))" "ac_cv_sizeof_long"        "$ac_includes_default"; then :
@@ -6698,7 +6705,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (long)
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
    else
      ac_cv_sizeof_long=0
    fi
@@ -6721,7 +6728,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of void *" >&5
 $as_echo_n "checking size of void *... " >&6; }
-if test "${ac_cv_sizeof_void_p+set}" = set; then :
+if ${ac_cv_sizeof_void_p+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (void *))" "ac_cv_sizeof_void_p"        "$ac_includes_default"; then :
@@ -6731,7 +6738,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (void *)
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
    else
      ac_cv_sizeof_void_p=0
    fi
@@ -6754,7 +6761,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of short" >&5
 $as_echo_n "checking size of short... " >&6; }
-if test "${ac_cv_sizeof_short+set}" = set; then :
+if ${ac_cv_sizeof_short+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (short))" "ac_cv_sizeof_short"        "$ac_includes_default"; then :
@@ -6764,7 +6771,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (short)
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
    else
      ac_cv_sizeof_short=0
    fi
@@ -6787,7 +6794,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of float" >&5
 $as_echo_n "checking size of float... " >&6; }
-if test "${ac_cv_sizeof_float+set}" = set; then :
+if ${ac_cv_sizeof_float+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (float))" "ac_cv_sizeof_float"        "$ac_includes_default"; then :
@@ -6797,7 +6804,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (float)
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
    else
      ac_cv_sizeof_float=0
    fi
@@ -6820,7 +6827,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of double" >&5
 $as_echo_n "checking size of double... " >&6; }
-if test "${ac_cv_sizeof_double+set}" = set; then :
+if ${ac_cv_sizeof_double+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (double))" "ac_cv_sizeof_double"        "$ac_includes_default"; then :
@@ -6830,7 +6837,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (double)
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
    else
      ac_cv_sizeof_double=0
    fi
@@ -6853,7 +6860,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of fpos_t" >&5
 $as_echo_n "checking size of fpos_t... " >&6; }
-if test "${ac_cv_sizeof_fpos_t+set}" = set; then :
+if ${ac_cv_sizeof_fpos_t+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (fpos_t))" "ac_cv_sizeof_fpos_t"        "$ac_includes_default"; then :
@@ -6863,7 +6870,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (fpos_t)
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
    else
      ac_cv_sizeof_fpos_t=0
    fi
@@ -6886,7 +6893,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of size_t" >&5
 $as_echo_n "checking size of size_t... " >&6; }
-if test "${ac_cv_sizeof_size_t+set}" = set; then :
+if ${ac_cv_sizeof_size_t+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (size_t))" "ac_cv_sizeof_size_t"        "$ac_includes_default"; then :
@@ -6896,7 +6903,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (size_t)
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
    else
      ac_cv_sizeof_size_t=0
    fi
@@ -6919,7 +6926,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of pid_t" >&5
 $as_echo_n "checking size of pid_t... " >&6; }
-if test "${ac_cv_sizeof_pid_t+set}" = set; then :
+if ${ac_cv_sizeof_pid_t+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (pid_t))" "ac_cv_sizeof_pid_t"        "$ac_includes_default"; then :
@@ -6929,7 +6936,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (pid_t)
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
    else
      ac_cv_sizeof_pid_t=0
    fi
@@ -6979,7 +6986,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of long long" >&5
 $as_echo_n "checking size of long long... " >&6; }
-if test "${ac_cv_sizeof_long_long+set}" = set; then :
+if ${ac_cv_sizeof_long_long+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (long long))" "ac_cv_sizeof_long_long"        "$ac_includes_default"; then :
@@ -6989,7 +6996,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (long long)
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
    else
      ac_cv_sizeof_long_long=0
    fi
@@ -7040,7 +7047,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of long double" >&5
 $as_echo_n "checking size of long double... " >&6; }
-if test "${ac_cv_sizeof_long_double+set}" = set; then :
+if ${ac_cv_sizeof_long_double+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (long double))" "ac_cv_sizeof_long_double"        "$ac_includes_default"; then :
@@ -7050,7 +7057,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (long double)
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
    else
      ac_cv_sizeof_long_double=0
    fi
@@ -7102,7 +7109,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of _Bool" >&5
 $as_echo_n "checking size of _Bool... " >&6; }
-if test "${ac_cv_sizeof__Bool+set}" = set; then :
+if ${ac_cv_sizeof__Bool+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (_Bool))" "ac_cv_sizeof__Bool"        "$ac_includes_default"; then :
@@ -7112,7 +7119,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (_Bool)
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
    else
      ac_cv_sizeof__Bool=0
    fi
@@ -7138,7 +7145,7 @@
         #include <inttypes.h>
         #endif
 "
-if test "x$ac_cv_type_uintptr_t" = x""yes; then :
+if test "x$ac_cv_type_uintptr_t" = xyes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_UINTPTR_T 1
@@ -7150,7 +7157,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of uintptr_t" >&5
 $as_echo_n "checking size of uintptr_t... " >&6; }
-if test "${ac_cv_sizeof_uintptr_t+set}" = set; then :
+if ${ac_cv_sizeof_uintptr_t+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (uintptr_t))" "ac_cv_sizeof_uintptr_t"        "$ac_includes_default"; then :
@@ -7160,7 +7167,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (uintptr_t)
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
    else
      ac_cv_sizeof_uintptr_t=0
    fi
@@ -7186,7 +7193,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of off_t" >&5
 $as_echo_n "checking size of off_t... " >&6; }
-if test "${ac_cv_sizeof_off_t+set}" = set; then :
+if ${ac_cv_sizeof_off_t+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (off_t))" "ac_cv_sizeof_off_t"        "
@@ -7201,7 +7208,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (off_t)
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
    else
      ac_cv_sizeof_off_t=0
    fi
@@ -7245,7 +7252,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of time_t" >&5
 $as_echo_n "checking size of time_t... " >&6; }
-if test "${ac_cv_sizeof_time_t+set}" = set; then :
+if ${ac_cv_sizeof_time_t+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (time_t))" "ac_cv_sizeof_time_t"        "
@@ -7263,7 +7270,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (time_t)
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
    else
      ac_cv_sizeof_time_t=0
    fi
@@ -7320,7 +7327,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of pthread_t" >&5
 $as_echo_n "checking size of pthread_t... " >&6; }
-if test "${ac_cv_sizeof_pthread_t+set}" = set; then :
+if ${ac_cv_sizeof_pthread_t+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (pthread_t))" "ac_cv_sizeof_pthread_t"        "
@@ -7335,7 +7342,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (pthread_t)
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
    else
      ac_cv_sizeof_pthread_t=0
    fi
@@ -7766,7 +7773,7 @@
 # checks for libraries
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sendfile in -lsendfile" >&5
 $as_echo_n "checking for sendfile in -lsendfile... " >&6; }
-if test "${ac_cv_lib_sendfile_sendfile+set}" = set; then :
+if ${ac_cv_lib_sendfile_sendfile+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -7800,7 +7807,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_sendfile_sendfile" >&5
 $as_echo "$ac_cv_lib_sendfile_sendfile" >&6; }
-if test "x$ac_cv_lib_sendfile_sendfile" = x""yes; then :
+if test "x$ac_cv_lib_sendfile_sendfile" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_LIBSENDFILE 1
 _ACEOF
@@ -7811,7 +7818,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for dlopen in -ldl" >&5
 $as_echo_n "checking for dlopen in -ldl... " >&6; }
-if test "${ac_cv_lib_dl_dlopen+set}" = set; then :
+if ${ac_cv_lib_dl_dlopen+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -7845,7 +7852,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_dl_dlopen" >&5
 $as_echo "$ac_cv_lib_dl_dlopen" >&6; }
-if test "x$ac_cv_lib_dl_dlopen" = x""yes; then :
+if test "x$ac_cv_lib_dl_dlopen" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_LIBDL 1
 _ACEOF
@@ -7856,7 +7863,7 @@
 	# Dynamic linking for SunOS/Solaris and SYSV
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for shl_load in -ldld" >&5
 $as_echo_n "checking for shl_load in -ldld... " >&6; }
-if test "${ac_cv_lib_dld_shl_load+set}" = set; then :
+if ${ac_cv_lib_dld_shl_load+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -7890,7 +7897,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_dld_shl_load" >&5
 $as_echo "$ac_cv_lib_dld_shl_load" >&6; }
-if test "x$ac_cv_lib_dld_shl_load" = x""yes; then :
+if test "x$ac_cv_lib_dld_shl_load" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_LIBDLD 1
 _ACEOF
@@ -7904,7 +7911,7 @@
 if test "$with_threads" = "yes" -o -z "$with_threads"; then
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing sem_init" >&5
 $as_echo_n "checking for library containing sem_init... " >&6; }
-if test "${ac_cv_search_sem_init+set}" = set; then :
+if ${ac_cv_search_sem_init+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_func_search_save_LIBS=$LIBS
@@ -7938,11 +7945,11 @@
 fi
 rm -f core conftest.err conftest.$ac_objext \
     conftest$ac_exeext
-  if test "${ac_cv_search_sem_init+set}" = set; then :
+  if ${ac_cv_search_sem_init+:} false; then :
   break
 fi
 done
-if test "${ac_cv_search_sem_init+set}" = set; then :
+if ${ac_cv_search_sem_init+:} false; then :
 
 else
   ac_cv_search_sem_init=no
@@ -7965,7 +7972,7 @@
 # check if we need libintl for locale functions
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for textdomain in -lintl" >&5
 $as_echo_n "checking for textdomain in -lintl... " >&6; }
-if test "${ac_cv_lib_intl_textdomain+set}" = set; then :
+if ${ac_cv_lib_intl_textdomain+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -7999,7 +8006,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_intl_textdomain" >&5
 $as_echo "$ac_cv_lib_intl_textdomain" >&6; }
-if test "x$ac_cv_lib_intl_textdomain" = x""yes; then :
+if test "x$ac_cv_lib_intl_textdomain" = xyes; then :
 
 $as_echo "#define WITH_LIBINTL 1" >>confdefs.h
 
@@ -8046,7 +8053,7 @@
 # Most SVR4 platforms (e.g. Solaris) need -lsocket and -lnsl.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for t_open in -lnsl" >&5
 $as_echo_n "checking for t_open in -lnsl... " >&6; }
-if test "${ac_cv_lib_nsl_t_open+set}" = set; then :
+if ${ac_cv_lib_nsl_t_open+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8080,13 +8087,13 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_nsl_t_open" >&5
 $as_echo "$ac_cv_lib_nsl_t_open" >&6; }
-if test "x$ac_cv_lib_nsl_t_open" = x""yes; then :
+if test "x$ac_cv_lib_nsl_t_open" = xyes; then :
   LIBS="-lnsl $LIBS"
 fi
  # SVR4
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for socket in -lsocket" >&5
 $as_echo_n "checking for socket in -lsocket... " >&6; }
-if test "${ac_cv_lib_socket_socket+set}" = set; then :
+if ${ac_cv_lib_socket_socket+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8120,7 +8127,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_socket_socket" >&5
 $as_echo "$ac_cv_lib_socket_socket" >&6; }
-if test "x$ac_cv_lib_socket_socket" = x""yes; then :
+if test "x$ac_cv_lib_socket_socket" = xyes; then :
   LIBS="-lsocket $LIBS"
 fi
  # SVR4 sockets
@@ -8146,7 +8153,7 @@
 set dummy ${ac_tool_prefix}pkg-config; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if test "${ac_cv_path_PKG_CONFIG+set}" = set; then :
+if ${ac_cv_path_PKG_CONFIG+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   case $PKG_CONFIG in
@@ -8189,7 +8196,7 @@
 set dummy pkg-config; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if test "${ac_cv_path_ac_pt_PKG_CONFIG+set}" = set; then :
+if ${ac_cv_path_ac_pt_PKG_CONFIG+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   case $ac_pt_PKG_CONFIG in
@@ -8485,7 +8492,7 @@
 
     LIBS=$_libs
     ac_fn_c_check_func "$LINENO" "pthread_detach" "ac_cv_func_pthread_detach"
-if test "x$ac_cv_func_pthread_detach" = x""yes; then :
+if test "x$ac_cv_func_pthread_detach" = xyes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     posix_threads=yes
@@ -8494,7 +8501,7 @@
 
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread_create in -lpthreads" >&5
 $as_echo_n "checking for pthread_create in -lpthreads... " >&6; }
-if test "${ac_cv_lib_pthreads_pthread_create+set}" = set; then :
+if ${ac_cv_lib_pthreads_pthread_create+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8528,7 +8535,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_pthreads_pthread_create" >&5
 $as_echo "$ac_cv_lib_pthreads_pthread_create" >&6; }
-if test "x$ac_cv_lib_pthreads_pthread_create" = x""yes; then :
+if test "x$ac_cv_lib_pthreads_pthread_create" = xyes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     posix_threads=yes
@@ -8538,7 +8545,7 @@
 
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread_create in -lc_r" >&5
 $as_echo_n "checking for pthread_create in -lc_r... " >&6; }
-if test "${ac_cv_lib_c_r_pthread_create+set}" = set; then :
+if ${ac_cv_lib_c_r_pthread_create+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8572,7 +8579,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_c_r_pthread_create" >&5
 $as_echo "$ac_cv_lib_c_r_pthread_create" >&6; }
-if test "x$ac_cv_lib_c_r_pthread_create" = x""yes; then :
+if test "x$ac_cv_lib_c_r_pthread_create" = xyes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     posix_threads=yes
@@ -8582,7 +8589,7 @@
 
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for __pthread_create_system in -lpthread" >&5
 $as_echo_n "checking for __pthread_create_system in -lpthread... " >&6; }
-if test "${ac_cv_lib_pthread___pthread_create_system+set}" = set; then :
+if ${ac_cv_lib_pthread___pthread_create_system+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8616,7 +8623,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_pthread___pthread_create_system" >&5
 $as_echo "$ac_cv_lib_pthread___pthread_create_system" >&6; }
-if test "x$ac_cv_lib_pthread___pthread_create_system" = x""yes; then :
+if test "x$ac_cv_lib_pthread___pthread_create_system" = xyes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     posix_threads=yes
@@ -8626,7 +8633,7 @@
 
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread_create in -lcma" >&5
 $as_echo_n "checking for pthread_create in -lcma... " >&6; }
-if test "${ac_cv_lib_cma_pthread_create+set}" = set; then :
+if ${ac_cv_lib_cma_pthread_create+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8660,7 +8667,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_cma_pthread_create" >&5
 $as_echo "$ac_cv_lib_cma_pthread_create" >&6; }
-if test "x$ac_cv_lib_cma_pthread_create" = x""yes; then :
+if test "x$ac_cv_lib_cma_pthread_create" = xyes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     posix_threads=yes
@@ -8686,7 +8693,7 @@
 
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for usconfig in -lmpc" >&5
 $as_echo_n "checking for usconfig in -lmpc... " >&6; }
-if test "${ac_cv_lib_mpc_usconfig+set}" = set; then :
+if ${ac_cv_lib_mpc_usconfig+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8720,7 +8727,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_mpc_usconfig" >&5
 $as_echo "$ac_cv_lib_mpc_usconfig" >&6; }
-if test "x$ac_cv_lib_mpc_usconfig" = x""yes; then :
+if test "x$ac_cv_lib_mpc_usconfig" = xyes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     LIBS="$LIBS -lmpc"
@@ -8732,7 +8739,7 @@
     if test "$posix_threads" != "yes"; then
       { $as_echo "$as_me:${as_lineno-$LINENO}: checking for thr_create in -lthread" >&5
 $as_echo_n "checking for thr_create in -lthread... " >&6; }
-if test "${ac_cv_lib_thread_thr_create+set}" = set; then :
+if ${ac_cv_lib_thread_thr_create+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8766,7 +8773,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_thread_thr_create" >&5
 $as_echo "$ac_cv_lib_thread_thr_create" >&6; }
-if test "x$ac_cv_lib_thread_thr_create" = x""yes; then :
+if test "x$ac_cv_lib_thread_thr_create" = xyes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
       LIBS="$LIBS -lthread"
@@ -8802,7 +8809,7 @@
 
       { $as_echo "$as_me:${as_lineno-$LINENO}: checking if PTHREAD_SCOPE_SYSTEM is supported" >&5
 $as_echo_n "checking if PTHREAD_SCOPE_SYSTEM is supported... " >&6; }
-      if test "${ac_cv_pthread_system_supported+set}" = set; then :
+      if ${ac_cv_pthread_system_supported+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -8845,7 +8852,7 @@
       for ac_func in pthread_sigmask
 do :
   ac_fn_c_check_func "$LINENO" "pthread_sigmask" "ac_cv_func_pthread_sigmask"
-if test "x$ac_cv_func_pthread_sigmask" = x""yes; then :
+if test "x$ac_cv_func_pthread_sigmask" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_PTHREAD_SIGMASK 1
 _ACEOF
@@ -9237,7 +9244,7 @@
 $as_echo "$with_valgrind" >&6; }
 if test "$with_valgrind" != no; then
     ac_fn_c_check_header_mongrel "$LINENO" "valgrind/valgrind.h" "ac_cv_header_valgrind_valgrind_h" "$ac_includes_default"
-if test "x$ac_cv_header_valgrind_valgrind_h" = x""yes; then :
+if test "x$ac_cv_header_valgrind_valgrind_h" = xyes; then :
 
 $as_echo "#define WITH_VALGRIND 1" >>confdefs.h
 
@@ -9259,7 +9266,7 @@
 for ac_func in dlopen
 do :
   ac_fn_c_check_func "$LINENO" "dlopen" "ac_cv_func_dlopen"
-if test "x$ac_cv_func_dlopen" = x""yes; then :
+if test "x$ac_cv_func_dlopen" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_DLOPEN 1
 _ACEOF
@@ -9593,7 +9600,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for flock declaration" >&5
 $as_echo_n "checking for flock declaration... " >&6; }
-if test "${ac_cv_flock_decl+set}" = set; then :
+if ${ac_cv_flock_decl+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -9623,7 +9630,7 @@
   for ac_func in flock
 do :
   ac_fn_c_check_func "$LINENO" "flock" "ac_cv_func_flock"
-if test "x$ac_cv_func_flock" = x""yes; then :
+if test "x$ac_cv_func_flock" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_FLOCK 1
 _ACEOF
@@ -9631,7 +9638,7 @@
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for flock in -lbsd" >&5
 $as_echo_n "checking for flock in -lbsd... " >&6; }
-if test "${ac_cv_lib_bsd_flock+set}" = set; then :
+if ${ac_cv_lib_bsd_flock+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -9665,7 +9672,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_bsd_flock" >&5
 $as_echo "$ac_cv_lib_bsd_flock" >&6; }
-if test "x$ac_cv_lib_bsd_flock" = x""yes; then :
+if test "x$ac_cv_lib_bsd_flock" = xyes; then :
   $as_echo "#define HAVE_FLOCK 1" >>confdefs.h
 
 
@@ -9742,7 +9749,7 @@
 set dummy $ac_prog; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if test "${ac_cv_prog_TRUE+set}" = set; then :
+if ${ac_cv_prog_TRUE+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$TRUE"; then
@@ -9782,7 +9789,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for inet_aton in -lc" >&5
 $as_echo_n "checking for inet_aton in -lc... " >&6; }
-if test "${ac_cv_lib_c_inet_aton+set}" = set; then :
+if ${ac_cv_lib_c_inet_aton+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -9816,12 +9823,12 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_c_inet_aton" >&5
 $as_echo "$ac_cv_lib_c_inet_aton" >&6; }
-if test "x$ac_cv_lib_c_inet_aton" = x""yes; then :
+if test "x$ac_cv_lib_c_inet_aton" = xyes; then :
   $ac_cv_prog_TRUE
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for inet_aton in -lresolv" >&5
 $as_echo_n "checking for inet_aton in -lresolv... " >&6; }
-if test "${ac_cv_lib_resolv_inet_aton+set}" = set; then :
+if ${ac_cv_lib_resolv_inet_aton+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -9855,7 +9862,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_resolv_inet_aton" >&5
 $as_echo "$ac_cv_lib_resolv_inet_aton" >&6; }
-if test "x$ac_cv_lib_resolv_inet_aton" = x""yes; then :
+if test "x$ac_cv_lib_resolv_inet_aton" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_LIBRESOLV 1
 _ACEOF
@@ -9872,7 +9879,7 @@
 # exit Python
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for chflags" >&5
 $as_echo_n "checking for chflags... " >&6; }
-if test "${ac_cv_have_chflags+set}" = set; then :
+if ${ac_cv_have_chflags+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -9906,7 +9913,7 @@
 $as_echo "$ac_cv_have_chflags" >&6; }
 if test "$ac_cv_have_chflags" = cross ; then
   ac_fn_c_check_func "$LINENO" "chflags" "ac_cv_func_chflags"
-if test "x$ac_cv_func_chflags" = x""yes; then :
+if test "x$ac_cv_func_chflags" = xyes; then :
   ac_cv_have_chflags="yes"
 else
   ac_cv_have_chflags="no"
@@ -9921,7 +9928,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for lchflags" >&5
 $as_echo_n "checking for lchflags... " >&6; }
-if test "${ac_cv_have_lchflags+set}" = set; then :
+if ${ac_cv_have_lchflags+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -9955,7 +9962,7 @@
 $as_echo "$ac_cv_have_lchflags" >&6; }
 if test "$ac_cv_have_lchflags" = cross ; then
   ac_fn_c_check_func "$LINENO" "lchflags" "ac_cv_func_lchflags"
-if test "x$ac_cv_func_lchflags" = x""yes; then :
+if test "x$ac_cv_func_lchflags" = xyes; then :
   ac_cv_have_lchflags="yes"
 else
   ac_cv_have_lchflags="no"
@@ -9979,7 +9986,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for inflateCopy in -lz" >&5
 $as_echo_n "checking for inflateCopy in -lz... " >&6; }
-if test "${ac_cv_lib_z_inflateCopy+set}" = set; then :
+if ${ac_cv_lib_z_inflateCopy+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -10013,7 +10020,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_z_inflateCopy" >&5
 $as_echo "$ac_cv_lib_z_inflateCopy" >&6; }
-if test "x$ac_cv_lib_z_inflateCopy" = x""yes; then :
+if test "x$ac_cv_lib_z_inflateCopy" = xyes; then :
 
 $as_echo "#define HAVE_ZLIB_COPY 1" >>confdefs.h
 
@@ -10156,7 +10163,7 @@
 for ac_func in openpty
 do :
   ac_fn_c_check_func "$LINENO" "openpty" "ac_cv_func_openpty"
-if test "x$ac_cv_func_openpty" = x""yes; then :
+if test "x$ac_cv_func_openpty" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_OPENPTY 1
 _ACEOF
@@ -10164,7 +10171,7 @@
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for openpty in -lutil" >&5
 $as_echo_n "checking for openpty in -lutil... " >&6; }
-if test "${ac_cv_lib_util_openpty+set}" = set; then :
+if ${ac_cv_lib_util_openpty+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -10198,13 +10205,13 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_util_openpty" >&5
 $as_echo "$ac_cv_lib_util_openpty" >&6; }
-if test "x$ac_cv_lib_util_openpty" = x""yes; then :
+if test "x$ac_cv_lib_util_openpty" = xyes; then :
   $as_echo "#define HAVE_OPENPTY 1" >>confdefs.h
  LIBS="$LIBS -lutil"
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for openpty in -lbsd" >&5
 $as_echo_n "checking for openpty in -lbsd... " >&6; }
-if test "${ac_cv_lib_bsd_openpty+set}" = set; then :
+if ${ac_cv_lib_bsd_openpty+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -10238,7 +10245,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_bsd_openpty" >&5
 $as_echo "$ac_cv_lib_bsd_openpty" >&6; }
-if test "x$ac_cv_lib_bsd_openpty" = x""yes; then :
+if test "x$ac_cv_lib_bsd_openpty" = xyes; then :
   $as_echo "#define HAVE_OPENPTY 1" >>confdefs.h
  LIBS="$LIBS -lbsd"
 fi
@@ -10253,7 +10260,7 @@
 for ac_func in forkpty
 do :
   ac_fn_c_check_func "$LINENO" "forkpty" "ac_cv_func_forkpty"
-if test "x$ac_cv_func_forkpty" = x""yes; then :
+if test "x$ac_cv_func_forkpty" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_FORKPTY 1
 _ACEOF
@@ -10261,7 +10268,7 @@
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for forkpty in -lutil" >&5
 $as_echo_n "checking for forkpty in -lutil... " >&6; }
-if test "${ac_cv_lib_util_forkpty+set}" = set; then :
+if ${ac_cv_lib_util_forkpty+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -10295,13 +10302,13 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_util_forkpty" >&5
 $as_echo "$ac_cv_lib_util_forkpty" >&6; }
-if test "x$ac_cv_lib_util_forkpty" = x""yes; then :
+if test "x$ac_cv_lib_util_forkpty" = xyes; then :
   $as_echo "#define HAVE_FORKPTY 1" >>confdefs.h
  LIBS="$LIBS -lutil"
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for forkpty in -lbsd" >&5
 $as_echo_n "checking for forkpty in -lbsd... " >&6; }
-if test "${ac_cv_lib_bsd_forkpty+set}" = set; then :
+if ${ac_cv_lib_bsd_forkpty+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -10335,7 +10342,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_bsd_forkpty" >&5
 $as_echo "$ac_cv_lib_bsd_forkpty" >&6; }
-if test "x$ac_cv_lib_bsd_forkpty" = x""yes; then :
+if test "x$ac_cv_lib_bsd_forkpty" = xyes; then :
   $as_echo "#define HAVE_FORKPTY 1" >>confdefs.h
  LIBS="$LIBS -lbsd"
 fi
@@ -10352,7 +10359,7 @@
 for ac_func in memmove
 do :
   ac_fn_c_check_func "$LINENO" "memmove" "ac_cv_func_memmove"
-if test "x$ac_cv_func_memmove" = x""yes; then :
+if test "x$ac_cv_func_memmove" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_MEMMOVE 1
 _ACEOF
@@ -10376,7 +10383,7 @@
 
 
 ac_fn_c_check_func "$LINENO" "dup2" "ac_cv_func_dup2"
-if test "x$ac_cv_func_dup2" = x""yes; then :
+if test "x$ac_cv_func_dup2" = xyes; then :
   $as_echo "#define HAVE_DUP2 1" >>confdefs.h
 
 else
@@ -10389,7 +10396,7 @@
 fi
 
 ac_fn_c_check_func "$LINENO" "getcwd" "ac_cv_func_getcwd"
-if test "x$ac_cv_func_getcwd" = x""yes; then :
+if test "x$ac_cv_func_getcwd" = xyes; then :
   $as_echo "#define HAVE_GETCWD 1" >>confdefs.h
 
 else
@@ -10402,7 +10409,7 @@
 fi
 
 ac_fn_c_check_func "$LINENO" "strdup" "ac_cv_func_strdup"
-if test "x$ac_cv_func_strdup" = x""yes; then :
+if test "x$ac_cv_func_strdup" = xyes; then :
   $as_echo "#define HAVE_STRDUP 1" >>confdefs.h
 
 else
@@ -10418,7 +10425,7 @@
 for ac_func in getpgrp
 do :
   ac_fn_c_check_func "$LINENO" "getpgrp" "ac_cv_func_getpgrp"
-if test "x$ac_cv_func_getpgrp" = x""yes; then :
+if test "x$ac_cv_func_getpgrp" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_GETPGRP 1
 _ACEOF
@@ -10446,7 +10453,7 @@
 for ac_func in setpgrp
 do :
   ac_fn_c_check_func "$LINENO" "setpgrp" "ac_cv_func_setpgrp"
-if test "x$ac_cv_func_setpgrp" = x""yes; then :
+if test "x$ac_cv_func_setpgrp" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_SETPGRP 1
 _ACEOF
@@ -10474,7 +10481,7 @@
 for ac_func in gettimeofday
 do :
   ac_fn_c_check_func "$LINENO" "gettimeofday" "ac_cv_func_gettimeofday"
-if test "x$ac_cv_func_gettimeofday" = x""yes; then :
+if test "x$ac_cv_func_gettimeofday" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_GETTIMEOFDAY 1
 _ACEOF
@@ -10576,7 +10583,7 @@
 then
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking getaddrinfo bug" >&5
 $as_echo_n "checking getaddrinfo bug... " >&6; }
-  if test "${ac_cv_buggy_getaddrinfo+set}" = set; then :
+  if ${ac_cv_buggy_getaddrinfo+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -10705,7 +10712,7 @@
 for ac_func in getnameinfo
 do :
   ac_fn_c_check_func "$LINENO" "getnameinfo" "ac_cv_func_getnameinfo"
-if test "x$ac_cv_func_getnameinfo" = x""yes; then :
+if test "x$ac_cv_func_getnameinfo" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_GETNAMEINFO 1
 _ACEOF
@@ -10717,7 +10724,7 @@
 # checks for structures
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether time.h and sys/time.h may both be included" >&5
 $as_echo_n "checking whether time.h and sys/time.h may both be included... " >&6; }
-if test "${ac_cv_header_time+set}" = set; then :
+if ${ac_cv_header_time+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -10752,7 +10759,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether struct tm is in sys/time.h or time.h" >&5
 $as_echo_n "checking whether struct tm is in sys/time.h or time.h... " >&6; }
-if test "${ac_cv_struct_tm+set}" = set; then :
+if ${ac_cv_struct_tm+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -10789,7 +10796,7 @@
 #include <$ac_cv_struct_tm>
 
 "
-if test "x$ac_cv_member_struct_tm_tm_zone" = x""yes; then :
+if test "x$ac_cv_member_struct_tm_tm_zone" = xyes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_TM_TM_ZONE 1
@@ -10805,7 +10812,7 @@
 else
   ac_fn_c_check_decl "$LINENO" "tzname" "ac_cv_have_decl_tzname" "#include <time.h>
 "
-if test "x$ac_cv_have_decl_tzname" = x""yes; then :
+if test "x$ac_cv_have_decl_tzname" = xyes; then :
   ac_have_decl=1
 else
   ac_have_decl=0
@@ -10817,7 +10824,7 @@
 
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for tzname" >&5
 $as_echo_n "checking for tzname... " >&6; }
-if test "${ac_cv_var_tzname+set}" = set; then :
+if ${ac_cv_var_tzname+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -10853,7 +10860,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_rdev" "ac_cv_member_struct_stat_st_rdev" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_rdev" = x""yes; then :
+if test "x$ac_cv_member_struct_stat_st_rdev" = xyes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_RDEV 1
@@ -10863,7 +10870,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_blksize" "ac_cv_member_struct_stat_st_blksize" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_blksize" = x""yes; then :
+if test "x$ac_cv_member_struct_stat_st_blksize" = xyes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_BLKSIZE 1
@@ -10873,7 +10880,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_flags" "ac_cv_member_struct_stat_st_flags" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_flags" = x""yes; then :
+if test "x$ac_cv_member_struct_stat_st_flags" = xyes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_FLAGS 1
@@ -10883,7 +10890,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_gen" "ac_cv_member_struct_stat_st_gen" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_gen" = x""yes; then :
+if test "x$ac_cv_member_struct_stat_st_gen" = xyes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_GEN 1
@@ -10893,7 +10900,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_birthtime" "ac_cv_member_struct_stat_st_birthtime" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_birthtime" = x""yes; then :
+if test "x$ac_cv_member_struct_stat_st_birthtime" = xyes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_BIRTHTIME 1
@@ -10903,7 +10910,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_blocks" "ac_cv_member_struct_stat_st_blocks" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_blocks" = x""yes; then :
+if test "x$ac_cv_member_struct_stat_st_blocks" = xyes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_BLOCKS 1
@@ -10925,7 +10932,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for time.h that defines altzone" >&5
 $as_echo_n "checking for time.h that defines altzone... " >&6; }
-if test "${ac_cv_header_time_altzone+set}" = set; then :
+if ${ac_cv_header_time_altzone+:} false; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -10989,7 +10996,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for addrinfo" >&5
 $as_echo_n "checking for addrinfo... " >&6; }
-if test "${ac_cv_struct_addrinfo+set}" = set; then :
+if ${ac_cv_struct_addrinfo+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -11021,7 +11028,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sockaddr_storage" >&5
 $as_echo_n "checking for sockaddr_storage... " >&6; }
-if test "${ac_cv_struct_sockaddr_storage+set}" = set; then :
+if ${ac_cv_struct_sockaddr_storage+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -11057,7 +11064,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether char is unsigned" >&5
 $as_echo_n "checking whether char is unsigned... " >&6; }
-if test "${ac_cv_c_char_unsigned+set}" = set; then :
+if ${ac_cv_c_char_unsigned+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -11089,7 +11096,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for an ANSI C-conforming const" >&5
 $as_echo_n "checking for an ANSI C-conforming const... " >&6; }
-if test "${ac_cv_c_const+set}" = set; then :
+if ${ac_cv_c_const+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -11377,7 +11384,7 @@
 
 
 ac_fn_c_check_func "$LINENO" "gethostbyname_r" "ac_cv_func_gethostbyname_r"
-if test "x$ac_cv_func_gethostbyname_r" = x""yes; then :
+if test "x$ac_cv_func_gethostbyname_r" = xyes; then :
 
   $as_echo "#define HAVE_GETHOSTBYNAME_R 1" >>confdefs.h
 
@@ -11508,7 +11515,7 @@
   for ac_func in gethostbyname
 do :
   ac_fn_c_check_func "$LINENO" "gethostbyname" "ac_cv_func_gethostbyname"
-if test "x$ac_cv_func_gethostbyname" = x""yes; then :
+if test "x$ac_cv_func_gethostbyname" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_GETHOSTBYNAME 1
 _ACEOF
@@ -11530,12 +11537,12 @@
 
 # Linux requires this for correct f.p. operations
 ac_fn_c_check_func "$LINENO" "__fpu_control" "ac_cv_func___fpu_control"
-if test "x$ac_cv_func___fpu_control" = x""yes; then :
+if test "x$ac_cv_func___fpu_control" = xyes; then :
 
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for __fpu_control in -lieee" >&5
 $as_echo_n "checking for __fpu_control in -lieee... " >&6; }
-if test "${ac_cv_lib_ieee___fpu_control+set}" = set; then :
+if ${ac_cv_lib_ieee___fpu_control+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -11569,7 +11576,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_ieee___fpu_control" >&5
 $as_echo "$ac_cv_lib_ieee___fpu_control" >&6; }
-if test "x$ac_cv_lib_ieee___fpu_control" = x""yes; then :
+if test "x$ac_cv_lib_ieee___fpu_control" = xyes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_LIBIEEE 1
 _ACEOF
@@ -11663,7 +11670,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether C doubles are little-endian IEEE 754 binary64" >&5
 $as_echo_n "checking whether C doubles are little-endian IEEE 754 binary64... " >&6; }
-if test "${ac_cv_little_endian_double+set}" = set; then :
+if ${ac_cv_little_endian_double+:} false; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -11705,7 +11712,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether C doubles are big-endian IEEE 754 binary64" >&5
 $as_echo_n "checking whether C doubles are big-endian IEEE 754 binary64... " >&6; }
-if test "${ac_cv_big_endian_double+set}" = set; then :
+if ${ac_cv_big_endian_double+:} false; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -11751,7 +11758,7 @@
 # conversions work.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether C doubles are ARM mixed-endian IEEE 754 binary64" >&5
 $as_echo_n "checking whether C doubles are ARM mixed-endian IEEE 754 binary64... " >&6; }
-if test "${ac_cv_mixed_endian_double+set}" = set; then :
+if ${ac_cv_mixed_endian_double+:} false; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -11921,7 +11928,7 @@
 
 ac_fn_c_check_decl "$LINENO" "isinf" "ac_cv_have_decl_isinf" "#include <math.h>
 "
-if test "x$ac_cv_have_decl_isinf" = x""yes; then :
+if test "x$ac_cv_have_decl_isinf" = xyes; then :
   ac_have_decl=1
 else
   ac_have_decl=0
@@ -11932,7 +11939,7 @@
 _ACEOF
 ac_fn_c_check_decl "$LINENO" "isnan" "ac_cv_have_decl_isnan" "#include <math.h>
 "
-if test "x$ac_cv_have_decl_isnan" = x""yes; then :
+if test "x$ac_cv_have_decl_isnan" = xyes; then :
   ac_have_decl=1
 else
   ac_have_decl=0
@@ -11943,7 +11950,7 @@
 _ACEOF
 ac_fn_c_check_decl "$LINENO" "isfinite" "ac_cv_have_decl_isfinite" "#include <math.h>
 "
-if test "x$ac_cv_have_decl_isfinite" = x""yes; then :
+if test "x$ac_cv_have_decl_isfinite" = xyes; then :
   ac_have_decl=1
 else
   ac_have_decl=0
@@ -11958,7 +11965,7 @@
 # -0. on some architectures.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether tanh preserves the sign of zero" >&5
 $as_echo_n "checking whether tanh preserves the sign of zero... " >&6; }
-if test "${ac_cv_tanh_preserves_zero_sign+set}" = set; then :
+if ${ac_cv_tanh_preserves_zero_sign+:} false; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -12006,7 +12013,7 @@
     # -0.  See issue #9920.
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether log1p drops the sign of negative zero" >&5
 $as_echo_n "checking whether log1p drops the sign of negative zero... " >&6; }
-    if test "${ac_cv_log1p_drops_zero_sign+set}" = set; then :
+    if ${ac_cv_log1p_drops_zero_sign+:} false; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -12058,7 +12065,7 @@
 # sem_open results in a 'Signal 12' error.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether POSIX semaphores are enabled" >&5
 $as_echo_n "checking whether POSIX semaphores are enabled... " >&6; }
-if test "${ac_cv_posix_semaphores_enabled+set}" = set; then :
+if ${ac_cv_posix_semaphores_enabled+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -12109,7 +12116,7 @@
 # Multiprocessing check for broken sem_getvalue
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for broken sem_getvalue" >&5
 $as_echo_n "checking for broken sem_getvalue... " >&6; }
-if test "${ac_cv_broken_sem_getvalue+set}" = set; then :
+if ${ac_cv_broken_sem_getvalue+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -12174,7 +12181,7 @@
 15|30)
   ;;
 *)
-  as_fn_error $? "bad value $enable_big_digits for --enable-big-digits; value should be 15 or 30" "$LINENO" 5  ;;
+  as_fn_error $? "bad value $enable_big_digits for --enable-big-digits; value should be 15 or 30" "$LINENO" 5 ;;
 esac
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $enable_big_digits" >&5
 $as_echo "$enable_big_digits" >&6; }
@@ -12192,7 +12199,7 @@
 
 # check for wchar.h
 ac_fn_c_check_header_mongrel "$LINENO" "wchar.h" "ac_cv_header_wchar_h" "$ac_includes_default"
-if test "x$ac_cv_header_wchar_h" = x""yes; then :
+if test "x$ac_cv_header_wchar_h" = xyes; then :
 
 
 $as_echo "#define HAVE_WCHAR_H 1" >>confdefs.h
@@ -12215,7 +12222,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of wchar_t" >&5
 $as_echo_n "checking size of wchar_t... " >&6; }
-if test "${ac_cv_sizeof_wchar_t+set}" = set; then :
+if ${ac_cv_sizeof_wchar_t+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (wchar_t))" "ac_cv_sizeof_wchar_t"        "#include <wchar.h>
@@ -12226,7 +12233,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (wchar_t)
-See \`config.log' for more details" "$LINENO" 5 ; }
+See \`config.log' for more details" "$LINENO" 5; }
    else
      ac_cv_sizeof_wchar_t=0
    fi
@@ -12281,7 +12288,7 @@
   # check whether wchar_t is signed or not
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether wchar_t is signed" >&5
 $as_echo_n "checking whether wchar_t is signed... " >&6; }
-  if test "${ac_cv_wchar_t_signed+set}" = set; then :
+  if ${ac_cv_wchar_t_signed+:} false; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -12377,7 +12384,7 @@
 # check for endianness
  { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether byte ordering is bigendian" >&5
 $as_echo_n "checking whether byte ordering is bigendian... " >&6; }
-if test "${ac_cv_c_bigendian+set}" = set; then :
+if ${ac_cv_c_bigendian+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_cv_c_bigendian=unknown
@@ -12596,7 +12603,7 @@
      ;; #(
    *)
      as_fn_error $? "unknown endianness
- presetting ac_cv_c_bigendian=no (or yes) will help" "$LINENO" 5  ;;
+ presetting ac_cv_c_bigendian=no (or yes) will help" "$LINENO" 5 ;;
  esac
 
 
@@ -12668,7 +12675,7 @@
 # or fills with zeros (like the Cray J90, according to Tim Peters).
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether right shift extends the sign bit" >&5
 $as_echo_n "checking whether right shift extends the sign bit... " >&6; }
-if test "${ac_cv_rshift_extends_sign+set}" = set; then :
+if ${ac_cv_rshift_extends_sign+:} false; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -12707,7 +12714,7 @@
 # check for getc_unlocked and related locking functions
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for getc_unlocked() and friends" >&5
 $as_echo_n "checking for getc_unlocked() and friends... " >&6; }
-if test "${ac_cv_have_getc_unlocked+set}" = set; then :
+if ${ac_cv_have_getc_unlocked+:} false; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -12805,7 +12812,7 @@
 # check for readline 2.1
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for rl_callback_handler_install in -lreadline" >&5
 $as_echo_n "checking for rl_callback_handler_install in -lreadline... " >&6; }
-if test "${ac_cv_lib_readline_rl_callback_handler_install+set}" = set; then :
+if ${ac_cv_lib_readline_rl_callback_handler_install+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -12839,7 +12846,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_readline_rl_callback_handler_install" >&5
 $as_echo "$ac_cv_lib_readline_rl_callback_handler_install" >&6; }
-if test "x$ac_cv_lib_readline_rl_callback_handler_install" = x""yes; then :
+if test "x$ac_cv_lib_readline_rl_callback_handler_install" = xyes; then :
 
 $as_echo "#define HAVE_RL_CALLBACK 1" >>confdefs.h
 
@@ -12891,7 +12898,7 @@
 # check for readline 4.0
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for rl_pre_input_hook in -lreadline" >&5
 $as_echo_n "checking for rl_pre_input_hook in -lreadline... " >&6; }
-if test "${ac_cv_lib_readline_rl_pre_input_hook+set}" = set; then :
+if ${ac_cv_lib_readline_rl_pre_input_hook+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -12925,7 +12932,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_readline_rl_pre_input_hook" >&5
 $as_echo "$ac_cv_lib_readline_rl_pre_input_hook" >&6; }
-if test "x$ac_cv_lib_readline_rl_pre_input_hook" = x""yes; then :
+if test "x$ac_cv_lib_readline_rl_pre_input_hook" = xyes; then :
 
 $as_echo "#define HAVE_RL_PRE_INPUT_HOOK 1" >>confdefs.h
 
@@ -12935,7 +12942,7 @@
 # also in 4.0
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for rl_completion_display_matches_hook in -lreadline" >&5
 $as_echo_n "checking for rl_completion_display_matches_hook in -lreadline... " >&6; }
-if test "${ac_cv_lib_readline_rl_completion_display_matches_hook+set}" = set; then :
+if ${ac_cv_lib_readline_rl_completion_display_matches_hook+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -12969,7 +12976,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_readline_rl_completion_display_matches_hook" >&5
 $as_echo "$ac_cv_lib_readline_rl_completion_display_matches_hook" >&6; }
-if test "x$ac_cv_lib_readline_rl_completion_display_matches_hook" = x""yes; then :
+if test "x$ac_cv_lib_readline_rl_completion_display_matches_hook" = xyes; then :
 
 $as_echo "#define HAVE_RL_COMPLETION_DISPLAY_MATCHES_HOOK 1" >>confdefs.h
 
@@ -12979,7 +12986,7 @@
 # check for readline 4.2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for rl_completion_matches in -lreadline" >&5
 $as_echo_n "checking for rl_completion_matches in -lreadline... " >&6; }
-if test "${ac_cv_lib_readline_rl_completion_matches+set}" = set; then :
+if ${ac_cv_lib_readline_rl_completion_matches+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -13013,7 +13020,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_readline_rl_completion_matches" >&5
 $as_echo "$ac_cv_lib_readline_rl_completion_matches" >&6; }
-if test "x$ac_cv_lib_readline_rl_completion_matches" = x""yes; then :
+if test "x$ac_cv_lib_readline_rl_completion_matches" = xyes; then :
 
 $as_echo "#define HAVE_RL_COMPLETION_MATCHES 1" >>confdefs.h
 
@@ -13054,7 +13061,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for broken nice()" >&5
 $as_echo_n "checking for broken nice()... " >&6; }
-if test "${ac_cv_broken_nice+set}" = set; then :
+if ${ac_cv_broken_nice+:} false; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -13095,7 +13102,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for broken poll()" >&5
 $as_echo_n "checking for broken poll()... " >&6; }
-if test "${ac_cv_broken_poll+set}" = set; then :
+if ${ac_cv_broken_poll+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -13150,7 +13157,7 @@
 #include <$ac_cv_struct_tm>
 
 "
-if test "x$ac_cv_member_struct_tm_tm_zone" = x""yes; then :
+if test "x$ac_cv_member_struct_tm_tm_zone" = xyes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_TM_TM_ZONE 1
@@ -13166,7 +13173,7 @@
 else
   ac_fn_c_check_decl "$LINENO" "tzname" "ac_cv_have_decl_tzname" "#include <time.h>
 "
-if test "x$ac_cv_have_decl_tzname" = x""yes; then :
+if test "x$ac_cv_have_decl_tzname" = xyes; then :
   ac_have_decl=1
 else
   ac_have_decl=0
@@ -13178,7 +13185,7 @@
 
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for tzname" >&5
 $as_echo_n "checking for tzname... " >&6; }
-if test "${ac_cv_var_tzname+set}" = set; then :
+if ${ac_cv_var_tzname+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -13217,7 +13224,7 @@
 # check tzset(3) exists and works like we expect it to
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for working tzset()" >&5
 $as_echo_n "checking for working tzset()... " >&6; }
-if test "${ac_cv_working_tzset+set}" = set; then :
+if ${ac_cv_working_tzset+:} false; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -13314,7 +13321,7 @@
 # Look for subsecond timestamps in struct stat
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for tv_nsec in struct stat" >&5
 $as_echo_n "checking for tv_nsec in struct stat... " >&6; }
-if test "${ac_cv_stat_tv_nsec+set}" = set; then :
+if ${ac_cv_stat_tv_nsec+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -13351,7 +13358,7 @@
 # Look for BSD style subsecond timestamps in struct stat
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for tv_nsec2 in struct stat" >&5
 $as_echo_n "checking for tv_nsec2 in struct stat... " >&6; }
-if test "${ac_cv_stat_tv_nsec2+set}" = set; then :
+if ${ac_cv_stat_tv_nsec2+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -13388,7 +13395,7 @@
 # On HP/UX 11.0, mvwdelch is a block with a return statement
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether mvwdelch is an expression" >&5
 $as_echo_n "checking whether mvwdelch is an expression... " >&6; }
-if test "${ac_cv_mvwdelch_is_expression+set}" = set; then :
+if ${ac_cv_mvwdelch_is_expression+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -13425,7 +13432,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether WINDOW has _flags" >&5
 $as_echo_n "checking whether WINDOW has _flags... " >&6; }
-if test "${ac_cv_window_has_flags+set}" = set; then :
+if ${ac_cv_window_has_flags+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -13573,7 +13580,7 @@
 then
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for %lld and %llu printf() format support" >&5
 $as_echo_n "checking for %lld and %llu printf() format support... " >&6; }
-  if test "${ac_cv_have_long_long_format+set}" = set; then :
+  if ${ac_cv_have_long_long_format+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -13643,7 +13650,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for %zd printf() format support" >&5
 $as_echo_n "checking for %zd printf() format support... " >&6; }
-if test "${ac_cv_have_size_t_format+set}" = set; then :
+if ${ac_cv_have_size_t_format+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -13716,7 +13723,7 @@
 #endif
 
 "
-if test "x$ac_cv_type_socklen_t" = x""yes; then :
+if test "x$ac_cv_type_socklen_t" = xyes; then :
 
 else
 
@@ -13727,7 +13734,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for broken mbstowcs" >&5
 $as_echo_n "checking for broken mbstowcs... " >&6; }
-if test "${ac_cv_broken_mbstowcs+set}" = set; then :
+if ${ac_cv_broken_mbstowcs+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -13767,7 +13774,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC supports computed gotos" >&5
 $as_echo_n "checking whether $CC supports computed gotos... " >&6; }
-if test "${ac_cv_computed_gotos+set}" = set; then :
+if ${ac_cv_computed_gotos+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -13934,10 +13941,21 @@
      :end' >>confcache
 if diff "$cache_file" confcache >/dev/null 2>&1; then :; else
   if test -w "$cache_file"; then
-    test "x$cache_file" != "x/dev/null" &&
+    if test "x$cache_file" != "x/dev/null"; then
       { $as_echo "$as_me:${as_lineno-$LINENO}: updating cache $cache_file" >&5
 $as_echo "$as_me: updating cache $cache_file" >&6;}
-    cat confcache >$cache_file
+      if test ! -f "$cache_file" || test -h "$cache_file"; then
+	cat confcache >"$cache_file"
+      else
+        case $cache_file in #(
+        */* | ?:*)
+	  mv -f confcache "$cache_file"$$ &&
+	  mv -f "$cache_file"$$ "$cache_file" ;; #(
+        *)
+	  mv -f confcache "$cache_file" ;;
+	esac
+      fi
+    fi
   else
     { $as_echo "$as_me:${as_lineno-$LINENO}: not updating unwritable cache $cache_file" >&5
 $as_echo "$as_me: not updating unwritable cache $cache_file" >&6;}
@@ -13970,7 +13988,7 @@
 
 
 
-: ${CONFIG_STATUS=./config.status}
+: "${CONFIG_STATUS=./config.status}"
 ac_write_fail=0
 ac_clean_files_save=$ac_clean_files
 ac_clean_files="$ac_clean_files $CONFIG_STATUS"
@@ -14071,6 +14089,7 @@
 IFS=" ""	$as_nl"
 
 # Find who we are.  Look in the path if we contain no directory separator.
+as_myself=
 case $0 in #((
   *[\\/]* ) as_myself=$0 ;;
   *) as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
@@ -14378,7 +14397,7 @@
 # values after options handling.
 ac_log="
 This file was extended by python $as_me 3.3, which was
-generated by GNU Autoconf 2.67.  Invocation command line was
+generated by GNU Autoconf 2.68.  Invocation command line was
 
   CONFIG_FILES    = $CONFIG_FILES
   CONFIG_HEADERS  = $CONFIG_HEADERS
@@ -14440,7 +14459,7 @@
 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
 ac_cs_version="\\
 python config.status 3.3
-configured by $0, generated by GNU Autoconf 2.67,
+configured by $0, generated by GNU Autoconf 2.68,
   with options \\"\$ac_cs_config\\"
 
 Copyright (C) 2010 Free Software Foundation, Inc.
@@ -14571,7 +14590,7 @@
     "Misc/python.pc") CONFIG_FILES="$CONFIG_FILES Misc/python.pc" ;;
     "Modules/ld_so_aix") CONFIG_FILES="$CONFIG_FILES Modules/ld_so_aix" ;;
 
-  *) as_fn_error $? "invalid argument: \`$ac_config_target'" "$LINENO" 5 ;;
+  *) as_fn_error $? "invalid argument: \`$ac_config_target'" "$LINENO" 5;;
   esac
 done
 
@@ -14593,9 +14612,10 @@
 # after its creation but before its name has been assigned to `$tmp'.
 $debug ||
 {
-  tmp=
+  tmp= ac_tmp=
   trap 'exit_status=$?
-  { test -z "$tmp" || test ! -d "$tmp" || rm -fr "$tmp"; } && exit $exit_status
+  : "${ac_tmp:=$tmp}"
+  { test ! -d "$ac_tmp" || rm -fr "$ac_tmp"; } && exit $exit_status
 ' 0
   trap 'as_fn_exit 1' 1 2 13 15
 }
@@ -14603,12 +14623,13 @@
 
 {
   tmp=`(umask 077 && mktemp -d "./confXXXXXX") 2>/dev/null` &&
-  test -n "$tmp" && test -d "$tmp"
+  test -d "$tmp"
 }  ||
 {
   tmp=./conf$$-$RANDOM
   (umask 077 && mkdir "$tmp")
 } || as_fn_error $? "cannot create a temporary directory in ." "$LINENO" 5
+ac_tmp=$tmp
 
 # Set up the scripts for CONFIG_FILES section.
 # No need to generate them if there are no CONFIG_FILES.
@@ -14630,7 +14651,7 @@
   ac_cs_awk_cr=$ac_cr
 fi
 
-echo 'BEGIN {' >"$tmp/subs1.awk" &&
+echo 'BEGIN {' >"$ac_tmp/subs1.awk" &&
 _ACEOF
 
 
@@ -14658,7 +14679,7 @@
 rm -f conf$$subs.sh
 
 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
-cat >>"\$tmp/subs1.awk" <<\\_ACAWK &&
+cat >>"\$ac_tmp/subs1.awk" <<\\_ACAWK &&
 _ACEOF
 sed -n '
 h
@@ -14706,7 +14727,7 @@
 rm -f conf$$subs.awk
 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
 _ACAWK
-cat >>"\$tmp/subs1.awk" <<_ACAWK &&
+cat >>"\$ac_tmp/subs1.awk" <<_ACAWK &&
   for (key in S) S_is_set[key] = 1
   FS = ""
 
@@ -14738,7 +14759,7 @@
   sed "s/$ac_cr\$//; s/$ac_cr/$ac_cs_awk_cr/g"
 else
   cat
-fi < "$tmp/subs1.awk" > "$tmp/subs.awk" \
+fi < "$ac_tmp/subs1.awk" > "$ac_tmp/subs.awk" \
   || as_fn_error $? "could not setup config files machinery" "$LINENO" 5
 _ACEOF
 
@@ -14772,7 +14793,7 @@
 # No need to generate them if there are no CONFIG_HEADERS.
 # This happens for instance with `./config.status Makefile'.
 if test -n "$CONFIG_HEADERS"; then
-cat >"$tmp/defines.awk" <<\_ACAWK ||
+cat >"$ac_tmp/defines.awk" <<\_ACAWK ||
 BEGIN {
 _ACEOF
 
@@ -14784,8 +14805,8 @@
 # handling of long lines.
 ac_delim='%!_!# '
 for ac_last_try in false false :; do
-  ac_t=`sed -n "/$ac_delim/p" confdefs.h`
-  if test -z "$ac_t"; then
+  ac_tt=`sed -n "/$ac_delim/p" confdefs.h`
+  if test -z "$ac_tt"; then
     break
   elif $ac_last_try; then
     as_fn_error $? "could not make $CONFIG_HEADERS" "$LINENO" 5
@@ -14886,7 +14907,7 @@
   esac
   case $ac_mode$ac_tag in
   :[FHL]*:*);;
-  :L* | :C*:*) as_fn_error $? "invalid tag \`$ac_tag'" "$LINENO" 5 ;;
+  :L* | :C*:*) as_fn_error $? "invalid tag \`$ac_tag'" "$LINENO" 5;;
   :[FH]-) ac_tag=-:-;;
   :[FH]*) ac_tag=$ac_tag:$ac_tag.in;;
   esac
@@ -14905,7 +14926,7 @@
     for ac_f
     do
       case $ac_f in
-      -) ac_f="$tmp/stdin";;
+      -) ac_f="$ac_tmp/stdin";;
       *) # Look for the file first in the build tree, then in the source tree
 	 # (if the path is not absolute).  The absolute path cannot be DOS-style,
 	 # because $ac_f cannot contain `:'.
@@ -14914,7 +14935,7 @@
 	   [\\/$]*) false;;
 	   *) test -f "$srcdir/$ac_f" && ac_f="$srcdir/$ac_f";;
 	   esac ||
-	   as_fn_error 1 "cannot find input file: \`$ac_f'" "$LINENO" 5 ;;
+	   as_fn_error 1 "cannot find input file: \`$ac_f'" "$LINENO" 5;;
       esac
       case $ac_f in *\'*) ac_f=`$as_echo "$ac_f" | sed "s/'/'\\\\\\\\''/g"`;; esac
       as_fn_append ac_file_inputs " '$ac_f'"
@@ -14940,8 +14961,8 @@
     esac
 
     case $ac_tag in
-    *:-:* | *:-) cat >"$tmp/stdin" \
-      || as_fn_error $? "could not create $ac_file" "$LINENO" 5  ;;
+    *:-:* | *:-) cat >"$ac_tmp/stdin" \
+      || as_fn_error $? "could not create $ac_file" "$LINENO" 5 ;;
     esac
     ;;
   esac
@@ -15071,21 +15092,22 @@
 s&@INSTALL@&$ac_INSTALL&;t t
 $ac_datarootdir_hack
 "
-eval sed \"\$ac_sed_extra\" "$ac_file_inputs" | $AWK -f "$tmp/subs.awk" >$tmp/out \
-  || as_fn_error $? "could not create $ac_file" "$LINENO" 5
+eval sed \"\$ac_sed_extra\" "$ac_file_inputs" | $AWK -f "$ac_tmp/subs.awk" \
+  >$ac_tmp/out || as_fn_error $? "could not create $ac_file" "$LINENO" 5
 
 test -z "$ac_datarootdir_hack$ac_datarootdir_seen" &&
-  { ac_out=`sed -n '/\${datarootdir}/p' "$tmp/out"`; test -n "$ac_out"; } &&
-  { ac_out=`sed -n '/^[	 ]*datarootdir[	 ]*:*=/p' "$tmp/out"`; test -z "$ac_out"; } &&
+  { ac_out=`sed -n '/\${datarootdir}/p' "$ac_tmp/out"`; test -n "$ac_out"; } &&
+  { ac_out=`sed -n '/^[	 ]*datarootdir[	 ]*:*=/p' \
+      "$ac_tmp/out"`; test -z "$ac_out"; } &&
   { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: $ac_file contains a reference to the variable \`datarootdir'
 which seems to be undefined.  Please make sure it is defined" >&5
 $as_echo "$as_me: WARNING: $ac_file contains a reference to the variable \`datarootdir'
 which seems to be undefined.  Please make sure it is defined" >&2;}
 
-  rm -f "$tmp/stdin"
+  rm -f "$ac_tmp/stdin"
   case $ac_file in
-  -) cat "$tmp/out" && rm -f "$tmp/out";;
-  *) rm -f "$ac_file" && mv "$tmp/out" "$ac_file";;
+  -) cat "$ac_tmp/out" && rm -f "$ac_tmp/out";;
+  *) rm -f "$ac_file" && mv "$ac_tmp/out" "$ac_file";;
   esac \
   || as_fn_error $? "could not create $ac_file" "$LINENO" 5
  ;;
@@ -15096,20 +15118,20 @@
   if test x"$ac_file" != x-; then
     {
       $as_echo "/* $configure_input  */" \
-      && eval '$AWK -f "$tmp/defines.awk"' "$ac_file_inputs"
-    } >"$tmp/config.h" \
+      && eval '$AWK -f "$ac_tmp/defines.awk"' "$ac_file_inputs"
+    } >"$ac_tmp/config.h" \
       || as_fn_error $? "could not create $ac_file" "$LINENO" 5
-    if diff "$ac_file" "$tmp/config.h" >/dev/null 2>&1; then
+    if diff "$ac_file" "$ac_tmp/config.h" >/dev/null 2>&1; then
       { $as_echo "$as_me:${as_lineno-$LINENO}: $ac_file is unchanged" >&5
 $as_echo "$as_me: $ac_file is unchanged" >&6;}
     else
       rm -f "$ac_file"
-      mv "$tmp/config.h" "$ac_file" \
+      mv "$ac_tmp/config.h" "$ac_file" \
 	|| as_fn_error $? "could not create $ac_file" "$LINENO" 5
     fi
   else
     $as_echo "/* $configure_input  */" \
-      && eval '$AWK -f "$tmp/defines.awk"' "$ac_file_inputs" \
+      && eval '$AWK -f "$ac_tmp/defines.awk"' "$ac_file_inputs" \
       || as_fn_error $? "could not create -" "$LINENO" 5
   fi
  ;;
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -408,8 +408,7 @@
   AC_DEFINE(_XOPEN_SOURCE_EXTENDED, 1,
    	    Define to activate Unix95-and-earlier features)
 
-  AC_DEFINE(_POSIX_C_SOURCE, 200112L, Define to activate features from IEEE Stds 1003.1-2001)
-  
+  AC_DEFINE(_POSIX_C_SOURCE, 200809L, Define to activate features from IEEE Stds 1003.1-2008)
 fi
 
 #

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Thu Sep  8 05:26:03 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Thu, 08 Sep 2011 05:26:03 +0200
Subject: [Python-checkins] Daily reference leaks (92842e347d98): sum=0
Message-ID: <E1R1VFb-0003Oe-D6@ap.vmr.nerim.net>

results for 92842e347d98 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogWnfF8W', '-x']

From ezio.melotti at gmail.com  Thu Sep  8 11:11:52 2011
From: ezio.melotti at gmail.com (Ezio Melotti)
Date: Thu, 8 Sep 2011 12:11:52 +0300
Subject: [Python-checkins] cpython: Issue #12567: Fix curses.unget_wch()
	tests
In-Reply-To: <E1R0qi0-0002Zh-I3@dinsdale.python.org>
References: <E1R0qi0-0002Zh-I3@dinsdale.python.org>
Message-ID: <CACBhJdEVwVEBJVfn8BX5G19NwfZzN6oTEwXs=q0=mJOcCNtsHA@mail.gmail.com>

Hi,

On Tue, Sep 6, 2011 at 11:08 AM, victor.stinner
<python-checkins at python.org>wrote:

> http://hg.python.org/cpython/rev/786668a4fb6b
> changeset:   72301:786668a4fb6b
> user:        Victor Stinner <vstinner at wyplay.com>
> date:        Tue Sep 06 10:08:28 2011 +0200
> summary:
>  Issue #12567: Fix curses.unget_wch() tests
>
> Skip the test if the function is missing. Use U+0061 (a) instead of U+00E9
> (?)
> because U+00E9 raises a _curses.error('unget_wch() returned ERR') on some
> buildbots. It's maybe because of the locale encoding.
>
> files:
>  Lib/test/test_curses.py |  6 ++++--
>  1 files changed, 4 insertions(+), 2 deletions(-)
>
>
> diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py
> --- a/Lib/test/test_curses.py
> +++ b/Lib/test/test_curses.py
> @@ -265,14 +265,16 @@
>     stdscr.getkey()
>
>  def test_unget_wch(stdscr):
> -    ch = '\xe9'
> +    if not hasattr(curses, 'unget_wch'):
> +        return
>

This should be a skip, not a bare return.


> +    ch = 'a'
>     curses.unget_wch(ch)
>     read = stdscr.get_wch()
>     read = chr(read)
>     if read != ch:
>         raise AssertionError("%r != %r" % (read, ch))
>

Why not just assertEqual?


>
> -    ch = ord('\xe9')
> +    ch = ord('a')
>     curses.unget_wch(ch)
>     read = stdscr.get_wch()
>     if read != ch:
>
>
>
Best Regards,
Ezio Melotti
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-checkins/attachments/20110908/5d5e68fd/attachment.html>

From python-checkins at python.org  Fri Sep  9 04:29:04 2011
From: python-checkins at python.org (larry.hastings)
Date: Fri, 09 Sep 2011 04:29:04 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312904=3A_os=2Eutim?=
 =?utf8?q?e=2C_os=2Efutimes=2C_os=2Elutimes=2C_and_os=2Efutimesat_now_writ?=
 =?utf8?q?e?=
Message-ID: <E1R1qq0-0004YG-U3@dinsdale.python.org>

http://hg.python.org/cpython/rev/1de6619733d9
changeset:   72315:1de6619733d9
user:        Larry Hastings <larry at hastings.org>
date:        Thu Sep 08 19:29:07 2011 -0700
summary:
  Issue #12904: os.utime, os.futimes, os.lutimes, and os.futimesat now write
atime and mtime with nanosecond precision on modern POSIX platforms.

files:
  Misc/NEWS             |    3 +
  Modules/posixmodule.c |  154 +++++++++++++++++++++--------
  2 files changed, 114 insertions(+), 43 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #12904: os.utime, os.futimes, os.lutimes, and os.futimesat now write
+  atime and mtime with nanosecond precision on modern POSIX platforms.
+
 - Issue #12802: the Windows error ERROR_DIRECTORY (numbered 267) is now
   mapped to POSIX errno ENOTDIR (previously EINVAL).
 
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -3417,6 +3417,30 @@
 }
 #endif /* HAVE_UNAME */
 
+
+/*
+ * Classic POSIX utime functions supported microseconds (1m/sec).
+ * Newer POSIX functions support nanoseconds (1 billion per sec).
+ * posixmodule now uses the new functions where possible.
+ * This improves accuracy in many situations, for example shutil.copy2().
+ *
+ * The implementation isn't currently sophisticated enough to handle
+ * a platform where HAVE_UTIMENSAT is true but HAVE_FUTIMENS is false.
+ * Specifically, posix_futimes() would break.
+ *
+ * Supporting such a platform wouldn't be impossible; you'd need two
+ * extract_time() functions, or make its precision a parameter.
+ * Since such a platform seems unlikely we haven't bothered.
+ */
+#if defined(HAVE_UTIMENSAT)
+#define EXTRACT_TIME_PRECISION (1e9)
+#if !defined(HAVE_FUTIMENS)
+#error You HAVE_UTIMENSAT but not HAVE_FUTIMENS... please see accompanying comment.
+#endif
+#else
+#define EXTRACT_TIME_PRECISION (1e6)
+#endif
+
 static int
 extract_time(PyObject *t, time_t* sec, long* usec)
 {
@@ -3435,7 +3459,8 @@
         if (intval == -1 && PyErr_Occurred())
             return -1;
         *sec = intval;
-        *usec = (long)((tval - intval) * 1e6); /* can't exceed 1000000 */
+
+        *usec = (long)((tval - intval) * EXTRACT_TIME_PRECISION);
         if (*usec < 0)
             /* If rounding gave us a negative number,
                truncate.  */
@@ -3553,24 +3578,6 @@
     int res;
     PyObject* arg;
 
-#if defined(HAVE_UTIMES)
-    struct timeval buf[2];
-#define ATIME buf[0].tv_sec
-#define MTIME buf[1].tv_sec
-#elif defined(HAVE_UTIME_H)
-/* XXX should define struct utimbuf instead, above */
-    struct utimbuf buf;
-#define ATIME buf.actime
-#define MTIME buf.modtime
-#define UTIME_ARG &buf
-#else /* HAVE_UTIMES */
-    time_t buf[2];
-#define ATIME buf[0]
-#define MTIME buf[1]
-#define UTIME_ARG buf
-#endif /* HAVE_UTIMES */
-
-
     if (!PyArg_ParseTuple(args, "O&O:utime",
                           PyUnicode_FSConverter, &opath, &arg))
         return NULL;
@@ -3598,19 +3605,37 @@
             Py_DECREF(opath);
             return NULL;
         }
-        ATIME = atime;
-        MTIME = mtime;
-#ifdef HAVE_UTIMES
+
+        Py_BEGIN_ALLOW_THREADS
+        {
+#ifdef HAVE_UTIMENSAT
+        struct timespec buf[2];
+        buf[0].tv_sec = atime;
+        buf[0].tv_nsec = ausec;
+        buf[1].tv_sec = mtime;
+        buf[1].tv_nsec = musec;
+        res = utimensat(AT_FDCWD, path, buf, 0);
+#elif defined(HAVE_UTIMES)
+        struct timeval buf[2];
+        buf[0].tv_sec = atime;
         buf[0].tv_usec = ausec;
+        buf[1].tv_sec = mtime;
         buf[1].tv_usec = musec;
-        Py_BEGIN_ALLOW_THREADS
         res = utimes(path, buf);
+#elif defined(HAVE_UTIME_H)
+        /* XXX should define struct utimbuf instead, above */
+        struct utimbuf buf;
+        buf.actime = atime;
+        buf.modtime = mtime;
+        res = utime(path, &buf);
+#else
+        time_t buf[2];
+        buf[0] = atime;
+        buf[1] = mtime;
+        res = utime(path, buf);
+#endif
+        }
         Py_END_ALLOW_THREADS
-#else
-        Py_BEGIN_ALLOW_THREADS
-        res = utime(path, UTIME_ARG);
-        Py_END_ALLOW_THREADS
-#endif /* HAVE_UTIMES */
     }
     if (res < 0) {
         return posix_error_with_allocated_filename(opath);
@@ -3618,9 +3643,7 @@
     Py_DECREF(opath);
     Py_INCREF(Py_None);
     return Py_None;
-#undef UTIME_ARG
-#undef ATIME
-#undef MTIME
+#undef UTIME_EXTRACT
 #endif /* MS_WINDOWS */
 }
 
@@ -3637,7 +3660,7 @@
 {
     int res, fd;
     PyObject* arg;
-    struct timeval buf[2];
+    time_t atime, mtime;
     long ausec, musec;
 
     if (!PyArg_ParseTuple(args, "iO:futimes", &fd, &arg))
@@ -3656,17 +3679,31 @@
     }
     else {
         if (extract_time(PyTuple_GET_ITEM(arg, 0),
-                &(buf[0].tv_sec), &ausec) == -1) {
+                &atime, &ausec) == -1) {
             return NULL;
         }
         if (extract_time(PyTuple_GET_ITEM(arg, 1),
-                &(buf[1].tv_sec), &musec) == -1) {
+                &mtime, &musec) == -1) {
             return NULL;
         }
+        Py_BEGIN_ALLOW_THREADS
+        {
+#ifdef HAVE_FUTIMENS
+        struct timespec buf[2];
+        buf[0].tv_sec = atime;
+        buf[0].tv_nsec = ausec;
+        buf[1].tv_sec = mtime;
+        buf[1].tv_nsec = musec;
+        res = futimens(fd, buf);
+#else
+        struct timeval buf[2];
+        buf[0].tv_sec = atime;
         buf[0].tv_usec = ausec;
+        buf[1].tv_sec = mtime;
         buf[1].tv_usec = musec;
-        Py_BEGIN_ALLOW_THREADS
         res = futimes(fd, buf);
+#endif
+        }
         Py_END_ALLOW_THREADS
     }
     if (res < 0)
@@ -3687,7 +3724,7 @@
     PyObject *opath, *arg;
     const char *path;
     int res;
-    struct timeval buf[2];
+    time_t atime, mtime;
     long ausec, musec;
 
     if (!PyArg_ParseTuple(args, "O&O:lutimes",
@@ -3708,19 +3745,33 @@
     }
     else {
         if (extract_time(PyTuple_GET_ITEM(arg, 0),
-                &(buf[0].tv_sec), &ausec) == -1) {
+                &atime, &ausec) == -1) {
             Py_DECREF(opath);
             return NULL;
         }
         if (extract_time(PyTuple_GET_ITEM(arg, 1),
-                &(buf[1].tv_sec), &musec) == -1) {
+                &mtime, &musec) == -1) {
             Py_DECREF(opath);
             return NULL;
         }
+        Py_BEGIN_ALLOW_THREADS
+        {
+#ifdef HAVE_UTIMENSAT
+        struct timespec buf[2];
+        buf[0].tv_sec = atime;
+        buf[0].tv_nsec = ausec;
+        buf[1].tv_sec = mtime;
+        buf[1].tv_nsec = musec;
+        res = utimensat(AT_FDCWD, path, buf, AT_SYMLINK_NOFOLLOW);
+#else
+        struct timeval buf[2];
+        buf[0].tv_sec = atime;
         buf[0].tv_usec = ausec;
+        buf[1].tv_sec = mtime;
         buf[1].tv_usec = musec;
-        Py_BEGIN_ALLOW_THREADS
         res = lutimes(path, buf);
+#endif
+        }
         Py_END_ALLOW_THREADS
     }
     Py_DECREF(opath);
@@ -9558,8 +9609,8 @@
     char *path;
     int res, dirfd;
     PyObject* arg;
-
-    struct timeval buf[2];
+    time_t atime, mtime;
+    long ausec, musec;
 
     if (!PyArg_ParseTuple(args, "iO&O:futimesat",
             &dirfd, PyUnicode_FSConverter, &opath, &arg))
@@ -9579,17 +9630,34 @@
     }
     else {
         if (extract_time(PyTuple_GET_ITEM(arg, 0),
-                         &(buf[0].tv_sec), &(buf[0].tv_usec)) == -1) {
+                         &atime, &ausec) == -1) {
             Py_DECREF(opath);
             return NULL;
         }
         if (extract_time(PyTuple_GET_ITEM(arg, 1),
-                         &(buf[1].tv_sec), &(buf[1].tv_usec)) == -1) {
+                         &mtime, &musec) == -1) {
             Py_DECREF(opath);
             return NULL;
         }
+
         Py_BEGIN_ALLOW_THREADS
+        {
+#ifdef HAVE_UTIMENSAT
+        struct timespec buf[2];
+        buf[0].tv_sec = atime;
+        buf[0].tv_nsec = ausec;
+        buf[1].tv_sec = mtime;
+        buf[1].tv_nsec = musec;
+        res = utimensat(dirfd, path, buf, 0);
+#else
+        struct timeval buf[2];
+        buf[0].tv_sec = atime;
+        buf[0].tv_usec = ausec;
+        buf[1].tv_sec = mtime;
+        buf[1].tv_usec = musec;
         res = futimesat(dirfd, path, buf);
+#endif
+        }
         Py_END_ALLOW_THREADS
     }
     Py_DECREF(opath);

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Fri Sep  9 05:17:59 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Fri, 09 Sep 2011 05:17:59 +0200
Subject: [Python-checkins] Daily reference leaks (92842e347d98): sum=0
Message-ID: <E1R1rbL-0003Uw-HN@ap.vmr.nerim.net>

results for 92842e347d98 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogL1fBE_', '-x']

From python-checkins at python.org  Fri Sep  9 18:51:24 2011
From: python-checkins at python.org (jesus.cea)
Date: Fri, 09 Sep 2011 18:51:24 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyMzMz?=
 =?utf8?q?=3A_fix_test=5Fdistutils_failures_under_Solaris_and_derivatives?=
 =?utf8?q?=2E_Patch?=
Message-ID: <E1R24IW-00057R-OF@dinsdale.python.org>

http://hg.python.org/cpython/rev/a74483c12740
changeset:   72316:a74483c12740
branch:      2.7
parent:      72307:299ea19c3197
user:        Jesus Cea <jcea at jcea.es>
date:        Fri Sep 09 18:50:59 2011 +0200
summary:
  Issue #12333: fix test_distutils failures under Solaris and derivatives. Patch by Antoine Pitrou

files:
  Lib/distutils/tests/support.py |  4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Lib/distutils/tests/support.py b/Lib/distutils/tests/support.py
--- a/Lib/distutils/tests/support.py
+++ b/Lib/distutils/tests/support.py
@@ -63,9 +63,13 @@
 
     def setUp(self):
         super(TempdirManager, self).setUp()
+        self.old_cwd = os.getcwd()
         self.tempdirs = []
 
     def tearDown(self):
+        # Restore working dir, for Solaris and derivatives, where rmdir()
+        # on the current directory fails.
+        os.chdir(self.old_cwd)
         super(TempdirManager, self).tearDown()
         while self.tempdirs:
             d = self.tempdirs.pop()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  9 19:11:27 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 09 Sep 2011 19:11:27 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_current_nam?=
 =?utf8?q?e_of_the_Python_3_binary_on_Unix_=28=2312896=29=2E?=
Message-ID: <E1R24bv-0007Cl-4Y@dinsdale.python.org>

http://hg.python.org/cpython/rev/4c7566669be5
changeset:   72317:4c7566669be5
branch:      3.2
parent:      72308:cf811943046b
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Sep 09 19:03:41 2011 +0200
summary:
  Fix current name of the Python 3 binary on Unix (#12896).

Also fix some markup and typos.

files:
  Doc/using/unix.rst |  27 ++++++++++++---------------
  1 files changed, 12 insertions(+), 15 deletions(-)


diff --git a/Doc/using/unix.rst b/Doc/using/unix.rst
--- a/Doc/using/unix.rst
+++ b/Doc/using/unix.rst
@@ -1,4 +1,4 @@
-.. highlightlang:: none
+.. highlightlang:: sh
 
 .. _using-on-unix:
 
@@ -55,8 +55,8 @@
 On OpenSolaris
 --------------
 
-To install the newest Python versions on OpenSolaris, install blastwave
-(http://www.blastwave.org/howto.html) and type "pkg_get -i python" at the
+To install the newest Python versions on OpenSolaris, install `blastwave
+<http://www.blastwave.org/howto.html>`_ and type ``pkg_get -i python`` at the
 prompt.
 
 
@@ -65,22 +65,23 @@
 
 If you want to compile CPython yourself, first thing you should do is get the
 `source <http://python.org/download/source/>`_. You can download either the
-latest release's source or just grab a fresh `checkout
-<http://www.python.org/dev/faq/#how-do-i-get-a-checkout-of-the-repository-read-only-and-read-write>`_.
+latest release's source or just grab a fresh `clone
+<http://docs.python.org/devguide/setup#getting-the-source-code>`_.  (If you want
+to contribute patches, you will need a clone.)
 
-The build process consists the usual ::
+The build process consists in the usual ::
 
    ./configure
    make
    make install
 
 invocations. Configuration options and caveats for specific Unix platforms are
-extensively documented in the :file:`README` file in the root of the Python
+extensively documented in the :source:`README` file in the root of the Python
 source tree.
 
 .. warning::
 
-   ``make install`` can overwrite or masquerade the :file:`python` binary.
+   ``make install`` can overwrite or masquerade the :file:`python3` binary.
    ``make altinstall`` is therefore recommended instead of ``make install``
    since it only installs :file:`{exec_prefix}/bin/python{version}`.
 
@@ -98,7 +99,7 @@
 +-----------------------------------------------+------------------------------------------+
 | File/directory                                | Meaning                                  |
 +===============================================+==========================================+
-| :file:`{exec_prefix}/bin/python`              | Recommended location of the interpreter. |
+| :file:`{exec_prefix}/bin/python3`             | Recommended location of the interpreter. |
 +-----------------------------------------------+------------------------------------------+
 | :file:`{prefix}/lib/python{version}`,         | Recommended locations of the directories |
 | :file:`{exec_prefix}/lib/python{version}`     | containing the standard modules.         |
@@ -108,10 +109,6 @@
 |                                               | developing Python extensions and         |
 |                                               | embedding the interpreter.               |
 +-----------------------------------------------+------------------------------------------+
-| :file:`~/.pythonrc.py`                        | User-specific initialization file loaded |
-|                                               | by the user module; not used by default  |
-|                                               | or by most applications.                 |
-+-----------------------------------------------+------------------------------------------+
 
 
 Miscellaneous
@@ -125,11 +122,11 @@
 and put an appropriate Shebang line at the top of the script.  A good choice is
 usually ::
 
-   #!/usr/bin/env python
+   #!/usr/bin/env python3
 
 which searches for the Python interpreter in the whole :envvar:`PATH`.  However,
 some Unices may not have the :program:`env` command, so you may need to hardcode
-``/usr/bin/python`` as the interpreter path.
+``/usr/bin/python3`` as the interpreter path.
 
 To use shell commands in your Python scripts, look at the :mod:`subprocess` module.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  9 19:11:27 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 09 Sep 2011 19:11:27 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge_3=2E2?=
Message-ID: <E1R24bv-0007Cp-SP@dinsdale.python.org>

http://hg.python.org/cpython/rev/272c8e9183db
changeset:   72318:272c8e9183db
parent:      72315:1de6619733d9
parent:      72317:4c7566669be5
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Sep 09 19:11:04 2011 +0200
summary:
  Merge 3.2

files:
  Doc/using/unix.rst |  27 ++++++++++++---------------
  1 files changed, 12 insertions(+), 15 deletions(-)


diff --git a/Doc/using/unix.rst b/Doc/using/unix.rst
--- a/Doc/using/unix.rst
+++ b/Doc/using/unix.rst
@@ -1,4 +1,4 @@
-.. highlightlang:: none
+.. highlightlang:: sh
 
 .. _using-on-unix:
 
@@ -55,8 +55,8 @@
 On OpenSolaris
 --------------
 
-To install the newest Python versions on OpenSolaris, install blastwave
-(http://www.blastwave.org/howto.html) and type "pkg_get -i python" at the
+To install the newest Python versions on OpenSolaris, install `blastwave
+<http://www.blastwave.org/howto.html>`_ and type ``pkg_get -i python`` at the
 prompt.
 
 
@@ -65,22 +65,23 @@
 
 If you want to compile CPython yourself, first thing you should do is get the
 `source <http://python.org/download/source/>`_. You can download either the
-latest release's source or just grab a fresh `checkout
-<http://docs.python.org/devguide/setup#checking-out-the-code>`_.
+latest release's source or just grab a fresh `clone
+<http://docs.python.org/devguide/setup#getting-the-source-code>`_.  (If you want
+to contribute patches, you will need a clone.)
 
-The build process consists the usual ::
+The build process consists in the usual ::
 
    ./configure
    make
    make install
 
 invocations. Configuration options and caveats for specific Unix platforms are
-extensively documented in the :file:`README` file in the root of the Python
+extensively documented in the :source:`README` file in the root of the Python
 source tree.
 
 .. warning::
 
-   ``make install`` can overwrite or masquerade the :file:`python` binary.
+   ``make install`` can overwrite or masquerade the :file:`python3` binary.
    ``make altinstall`` is therefore recommended instead of ``make install``
    since it only installs :file:`{exec_prefix}/bin/python{version}`.
 
@@ -98,7 +99,7 @@
 +-----------------------------------------------+------------------------------------------+
 | File/directory                                | Meaning                                  |
 +===============================================+==========================================+
-| :file:`{exec_prefix}/bin/python`              | Recommended location of the interpreter. |
+| :file:`{exec_prefix}/bin/python3`             | Recommended location of the interpreter. |
 +-----------------------------------------------+------------------------------------------+
 | :file:`{prefix}/lib/python{version}`,         | Recommended locations of the directories |
 | :file:`{exec_prefix}/lib/python{version}`     | containing the standard modules.         |
@@ -108,10 +109,6 @@
 |                                               | developing Python extensions and         |
 |                                               | embedding the interpreter.               |
 +-----------------------------------------------+------------------------------------------+
-| :file:`~/.pythonrc.py`                        | User-specific initialization file loaded |
-|                                               | by the user module; not used by default  |
-|                                               | or by most applications.                 |
-+-----------------------------------------------+------------------------------------------+
 
 
 Miscellaneous
@@ -125,11 +122,11 @@
 and put an appropriate Shebang line at the top of the script.  A good choice is
 usually ::
 
-   #!/usr/bin/env python
+   #!/usr/bin/env python3
 
 which searches for the Python interpreter in the whole :envvar:`PATH`.  However,
 some Unices may not have the :program:`env` command, so you may need to hardcode
-``/usr/bin/python`` as the interpreter path.
+``/usr/bin/python3`` as the interpreter path.
 
 To use shell commands in your Python scripts, look at the :mod:`subprocess` module.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  9 19:22:17 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 09 Sep 2011 19:22:17 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Port_a_few_mark?=
	=?utf8?q?up_edits_from_3=2Ex?=
Message-ID: <E1R24mP-000843-ST@dinsdale.python.org>

http://hg.python.org/cpython/rev/f3bd49d07a53
changeset:   72319:f3bd49d07a53
branch:      2.7
parent:      72316:a74483c12740
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Sep 09 19:20:27 2011 +0200
summary:
  Port a few markup edits from 3.x

files:
  Doc/using/unix.rst |  15 ++++++++-------
  1 files changed, 8 insertions(+), 7 deletions(-)


diff --git a/Doc/using/unix.rst b/Doc/using/unix.rst
--- a/Doc/using/unix.rst
+++ b/Doc/using/unix.rst
@@ -1,4 +1,4 @@
-.. highlightlang:: none
+.. highlightlang:: sh
 
 .. _using-on-unix:
 
@@ -55,8 +55,8 @@
 On OpenSolaris
 --------------
 
-To install the newest Python versions on OpenSolaris, install blastwave
-(http://www.blastwave.org/howto.html) and type "pkg_get -i python" at the
+To install the newest Python versions on OpenSolaris, install `blastwave
+<http://www.blastwave.org/howto.html>`_ and type ``pkg_get -i python`` at the
 prompt.
 
 
@@ -65,17 +65,18 @@
 
 If you want to compile CPython yourself, first thing you should do is get the
 `source <http://python.org/download/source/>`_. You can download either the
-latest release's source or just grab a fresh `checkout
-<http://www.python.org/dev/faq/#how-do-i-get-a-checkout-of-the-repository-read-only-and-read-write>`_.
+latest release's source or just grab a fresh `clone
+<http://docs.python.org/devguide/setup#getting-the-source-code>`_.  (If you want
+to contribute patches, you will need a clone.)
 
-The build process consists the usual ::
+The build process consists in the usual ::
 
    ./configure
    make
    make install
 
 invocations. Configuration options and caveats for specific Unix platforms are
-extensively documented in the :file:`README` file in the root of the Python
+extensively documented in the :source:`README` file in the root of the Python
 source tree.
 
 .. warning::

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  9 20:35:04 2011
From: python-checkins at python.org (jesus.cea)
Date: Fri, 09 Sep 2011 20:35:04 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Close_issue_=23?=
 =?utf8?q?12948=3A_multiprocessing_test_failures_can_hang_the_buildbots?=
Message-ID: <E1R25uq-0006H0-HB@dinsdale.python.org>

http://hg.python.org/cpython/rev/ebeb58f3de39
changeset:   72320:ebeb58f3de39
branch:      3.2
parent:      72317:4c7566669be5
user:        Jesus Cea <jcea at jcea.es>
date:        Fri Sep 09 20:26:57 2011 +0200
summary:
  Close issue #12948: multiprocessing test failures can hang the buildbots

files:
  Lib/test/test_multiprocessing.py |  30 ++++++++++++++++++--
  1 files changed, 27 insertions(+), 3 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -267,6 +267,7 @@
         p = self.Process(target=time.sleep, args=(DELTA,))
         self.assertNotIn(p, self.active_children())
 
+        p.daemon = True
         p.start()
         self.assertIn(p, self.active_children())
 
@@ -337,6 +338,7 @@
 
     def test_subclassing(self):
         uppercaser = _UpperCaser()
+        uppercaser.daemon = True
         uppercaser.start()
         self.assertEqual(uppercaser.submit('hello'), 'HELLO')
         self.assertEqual(uppercaser.submit('world'), 'WORLD')
@@ -515,6 +517,7 @@
 
         # fork process
         p = self.Process(target=self._test_fork, args=(queue,))
+        p.daemon = True
         p.start()
 
         # check that all expected items are in the queue
@@ -555,6 +558,7 @@
                    for i in range(4)]
 
         for p in workers:
+            p.daemon = True
             p.start()
 
         for i in range(10):
@@ -825,7 +829,9 @@
 
         #self.assertEqual(event.is_set(), False)
 
-        self.Process(target=self._test_event, args=(event,)).start()
+        p = self.Process(target=self._test_event, args=(event,))
+        p.daemon = True
+        p.start()
         self.assertEqual(wait(), True)
 
 #
@@ -865,6 +871,7 @@
             self.assertEqual(sv.value, cv[1])
 
         proc = self.Process(target=self._test, args=(values,))
+        proc.daemon = True
         proc.start()
         proc.join()
 
@@ -928,6 +935,7 @@
         self.f(seq)
 
         p = self.Process(target=self.f, args=(arr,))
+        p.daemon = True
         p.start()
         p.join()
 
@@ -1332,6 +1340,7 @@
         manager.start()
 
         p = self.Process(target=self._putter, args=(manager.address, authkey))
+        p.daemon = True
         p.start()
 
         manager2 = QueueManager2(
@@ -1373,6 +1382,7 @@
         manager.start()
 
         p = self.Process(target=self._putter, args=(manager.address, authkey))
+        p.daemon = True
         p.start()
         queue = manager.get_queue()
         self.assertEqual(queue.get(), 'hello world')
@@ -1505,6 +1515,7 @@
         conn, child_conn = self.Pipe()
 
         p = self.Process(target=self._echo, args=(child_conn,))
+        p.daemon = True
         p.start()
         child_conn.close()    # this might complete before child initializes
 
@@ -1577,6 +1588,7 @@
         conn, child_conn = self.Pipe(duplex=True)
 
         p = self.Process(target=self._writefd, args=(child_conn, b"foo"))
+        p.daemon = True
         p.start()
         with open(test.support.TESTFN, "wb") as f:
             fd = f.fileno()
@@ -1600,6 +1612,7 @@
         conn, child_conn = self.Pipe(duplex=True)
 
         p = self.Process(target=self._writefd, args=(child_conn, b"bar", True))
+        p.daemon = True
         p.start()
         with open(test.support.TESTFN, "wb") as f:
             fd = f.fileno()
@@ -1688,11 +1701,13 @@
 
         lconn, lconn0 = self.Pipe()
         lp = self.Process(target=self._listener, args=(lconn0, families))
+        lp.daemon = True
         lp.start()
         lconn0.close()
 
         rconn, rconn0 = self.Pipe()
         rp = self.Process(target=self._remote, args=(rconn0,))
+        rp.daemon = True
         rp.start()
         rconn0.close()
 
@@ -1830,6 +1845,7 @@
         string.value = latin('hello')
 
         p = self.Process(target=self._double, args=(x, y, foo, arr, string))
+        p.daemon = True
         p.start()
         p.join()
 
@@ -1902,6 +1918,7 @@
         conn, child_conn = self.Pipe()
 
         p = self.Process(target=self._test_finalize, args=(child_conn,))
+        p.daemon = True
         p.start()
         p.join()
 
@@ -1971,12 +1988,16 @@
         reader, writer = multiprocessing.Pipe(duplex=False)
 
         logger.setLevel(LEVEL1)
-        self.Process(target=self._test_level, args=(writer,)).start()
+        p = self.Process(target=self._test_level, args=(writer,))
+        p.daemon = True
+        p.start()
         self.assertEqual(LEVEL1, reader.recv())
 
         logger.setLevel(logging.NOTSET)
         root_logger.setLevel(LEVEL2)
-        self.Process(target=self._test_level, args=(writer,)).start()
+        p = self.Process(target=self._test_level, args=(writer,))
+        p.daemon = True
+        p.start()
         self.assertEqual(LEVEL2, reader.recv())
 
         root_logger.setLevel(root_level)
@@ -2162,6 +2183,7 @@
 def _TestProcess(q):
     queue = multiprocessing.Queue()
     subProc = multiprocessing.Process(target=_ThisSubProcess, args=(queue,))
+    subProc.daemon = True
     subProc.start()
     subProc.join()
 
@@ -2198,11 +2220,13 @@
     def test_queue_in_process(self):
         queue = multiprocessing.Queue()
         proc = multiprocessing.Process(target=_TestProcess, args=(queue,))
+        proc.daemon = True
         proc.start()
         proc.join()
 
     def test_pool_in_process(self):
         p = multiprocessing.Process(target=pool_in_process)
+        p.daemon = True
         p.start()
         p.join()
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  9 20:35:05 2011
From: python-checkins at python.org (jesus.cea)
Date: Fri, 09 Sep 2011 20:35:05 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Close_issue_=2312948=3A_multiprocessing_test_failures_can_ha?=
 =?utf8?q?ng_the_buildbots?=
Message-ID: <E1R25ur-0006KE-Kz@dinsdale.python.org>

http://hg.python.org/cpython/rev/1e189d55721c
changeset:   72321:1e189d55721c
parent:      72318:272c8e9183db
parent:      72320:ebeb58f3de39
user:        Jesus Cea <jcea at jcea.es>
date:        Fri Sep 09 20:29:01 2011 +0200
summary:
  Close issue #12948: multiprocessing test failures can hang the buildbots

files:
  Lib/test/test_multiprocessing.py |  30 ++++++++++++++++++--
  1 files changed, 27 insertions(+), 3 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -296,6 +296,7 @@
         p = self.Process(target=time.sleep, args=(DELTA,))
         self.assertNotIn(p, self.active_children())
 
+        p.daemon = True
         p.start()
         self.assertIn(p, self.active_children())
 
@@ -386,6 +387,7 @@
 
     def test_subclassing(self):
         uppercaser = _UpperCaser()
+        uppercaser.daemon = True
         uppercaser.start()
         self.assertEqual(uppercaser.submit('hello'), 'HELLO')
         self.assertEqual(uppercaser.submit('world'), 'WORLD')
@@ -564,6 +566,7 @@
 
         # fork process
         p = self.Process(target=self._test_fork, args=(queue,))
+        p.daemon = True
         p.start()
 
         # check that all expected items are in the queue
@@ -604,6 +607,7 @@
                    for i in range(4)]
 
         for p in workers:
+            p.daemon = True
             p.start()
 
         for i in range(10):
@@ -874,7 +878,9 @@
 
         #self.assertEqual(event.is_set(), False)
 
-        self.Process(target=self._test_event, args=(event,)).start()
+        p = self.Process(target=self._test_event, args=(event,))
+        p.daemon = True
+        p.start()
         self.assertEqual(wait(), True)
 
 #
@@ -914,6 +920,7 @@
             self.assertEqual(sv.value, cv[1])
 
         proc = self.Process(target=self._test, args=(values,))
+        proc.daemon = True
         proc.start()
         proc.join()
 
@@ -977,6 +984,7 @@
         self.f(seq)
 
         p = self.Process(target=self.f, args=(arr,))
+        p.daemon = True
         p.start()
         p.join()
 
@@ -1381,6 +1389,7 @@
         manager.start()
 
         p = self.Process(target=self._putter, args=(manager.address, authkey))
+        p.daemon = True
         p.start()
 
         manager2 = QueueManager2(
@@ -1422,6 +1431,7 @@
         manager.start()
 
         p = self.Process(target=self._putter, args=(manager.address, authkey))
+        p.daemon = True
         p.start()
         queue = manager.get_queue()
         self.assertEqual(queue.get(), 'hello world')
@@ -1554,6 +1564,7 @@
         conn, child_conn = self.Pipe()
 
         p = self.Process(target=self._echo, args=(child_conn,))
+        p.daemon = True
         p.start()
         child_conn.close()    # this might complete before child initializes
 
@@ -1626,6 +1637,7 @@
         conn, child_conn = self.Pipe(duplex=True)
 
         p = self.Process(target=self._writefd, args=(child_conn, b"foo"))
+        p.daemon = True
         p.start()
         with open(test.support.TESTFN, "wb") as f:
             fd = f.fileno()
@@ -1649,6 +1661,7 @@
         conn, child_conn = self.Pipe(duplex=True)
 
         p = self.Process(target=self._writefd, args=(child_conn, b"bar", True))
+        p.daemon = True
         p.start()
         with open(test.support.TESTFN, "wb") as f:
             fd = f.fileno()
@@ -1737,11 +1750,13 @@
 
         lconn, lconn0 = self.Pipe()
         lp = self.Process(target=self._listener, args=(lconn0, families))
+        lp.daemon = True
         lp.start()
         lconn0.close()
 
         rconn, rconn0 = self.Pipe()
         rp = self.Process(target=self._remote, args=(rconn0,))
+        rp.daemon = True
         rp.start()
         rconn0.close()
 
@@ -1879,6 +1894,7 @@
         string.value = latin('hello')
 
         p = self.Process(target=self._double, args=(x, y, foo, arr, string))
+        p.daemon = True
         p.start()
         p.join()
 
@@ -1951,6 +1967,7 @@
         conn, child_conn = self.Pipe()
 
         p = self.Process(target=self._test_finalize, args=(child_conn,))
+        p.daemon = True
         p.start()
         p.join()
 
@@ -2020,12 +2037,16 @@
         reader, writer = multiprocessing.Pipe(duplex=False)
 
         logger.setLevel(LEVEL1)
-        self.Process(target=self._test_level, args=(writer,)).start()
+        p = self.Process(target=self._test_level, args=(writer,))
+        p.daemon = True
+        p.start()
         self.assertEqual(LEVEL1, reader.recv())
 
         logger.setLevel(logging.NOTSET)
         root_logger.setLevel(LEVEL2)
-        self.Process(target=self._test_level, args=(writer,)).start()
+        p = self.Process(target=self._test_level, args=(writer,))
+        p.daemon = True
+        p.start()
         self.assertEqual(LEVEL2, reader.recv())
 
         root_logger.setLevel(root_level)
@@ -2217,6 +2238,7 @@
 def _TestProcess(q):
     queue = multiprocessing.Queue()
     subProc = multiprocessing.Process(target=_ThisSubProcess, args=(queue,))
+    subProc.daemon = True
     subProc.start()
     subProc.join()
 
@@ -2253,11 +2275,13 @@
     def test_queue_in_process(self):
         queue = multiprocessing.Queue()
         proc = multiprocessing.Process(target=_TestProcess, args=(queue,))
+        proc.daemon = True
         proc.start()
         proc.join()
 
     def test_pool_in_process(self):
         p = multiprocessing.Process(target=pool_in_process)
+        p.daemon = True
         p.start()
         p.join()
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  9 20:35:06 2011
From: python-checkins at python.org (jesus.cea)
Date: Fri, 09 Sep 2011 20:35:06 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Close_issue_=23?=
 =?utf8?q?12948=3A_multiprocessing_test_failures_can_hang_the_buildbots?=
Message-ID: <E1R25us-0006Mu-Cb@dinsdale.python.org>

http://hg.python.org/cpython/rev/b376534856a3
changeset:   72322:b376534856a3
branch:      2.7
parent:      72319:f3bd49d07a53
user:        Jesus Cea <jcea at jcea.es>
date:        Fri Sep 09 20:26:57 2011 +0200
summary:
  Close issue #12948: multiprocessing test failures can hang the buildbots

files:
  Lib/test/test_multiprocessing.py |  30 ++++++++++++++++++--
  1 files changed, 27 insertions(+), 3 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -264,6 +264,7 @@
         p = self.Process(target=time.sleep, args=(DELTA,))
         self.assertNotIn(p, self.active_children())
 
+        p.daemon = True
         p.start()
         self.assertIn(p, self.active_children())
 
@@ -334,6 +335,7 @@
 
     def test_subclassing(self):
         uppercaser = _UpperCaser()
+        uppercaser.daemon = True
         uppercaser.start()
         self.assertEqual(uppercaser.submit('hello'), 'HELLO')
         self.assertEqual(uppercaser.submit('world'), 'WORLD')
@@ -512,6 +514,7 @@
 
         # fork process
         p = self.Process(target=self._test_fork, args=(queue,))
+        p.daemon = True
         p.start()
 
         # check that all expected items are in the queue
@@ -552,6 +555,7 @@
                    for i in xrange(4)]
 
         for p in workers:
+            p.daemon = True
             p.start()
 
         for i in xrange(10):
@@ -816,7 +820,9 @@
 
         #self.assertEqual(event.is_set(), False)
 
-        self.Process(target=self._test_event, args=(event,)).start()
+        p = self.Process(target=self._test_event, args=(event,))
+        p.daemon = True
+        p.start()
         self.assertEqual(wait(), True)
 
 #
@@ -856,6 +862,7 @@
             self.assertEqual(sv.value, cv[1])
 
         proc = self.Process(target=self._test, args=(values,))
+        proc.daemon = True
         proc.start()
         proc.join()
 
@@ -919,6 +926,7 @@
         self.f(seq)
 
         p = self.Process(target=self.f, args=(arr,))
+        p.daemon = True
         p.start()
         p.join()
 
@@ -1285,6 +1293,7 @@
         manager.start()
 
         p = self.Process(target=self._putter, args=(manager.address, authkey))
+        p.daemon = True
         p.start()
 
         manager2 = QueueManager2(
@@ -1326,6 +1335,7 @@
         manager.start()
 
         p = self.Process(target=self._putter, args=(manager.address, authkey))
+        p.daemon = True
         p.start()
         queue = manager.get_queue()
         self.assertEqual(queue.get(), 'hello world')
@@ -1449,6 +1459,7 @@
         conn, child_conn = self.Pipe()
 
         p = self.Process(target=self._echo, args=(child_conn,))
+        p.daemon = True
         p.start()
         child_conn.close()    # this might complete before child initializes
 
@@ -1521,6 +1532,7 @@
         conn, child_conn = self.Pipe(duplex=True)
 
         p = self.Process(target=self._writefd, args=(child_conn, b"foo"))
+        p.daemon = True
         p.start()
         with open(test_support.TESTFN, "wb") as f:
             fd = f.fileno()
@@ -1544,6 +1556,7 @@
         conn, child_conn = self.Pipe(duplex=True)
 
         p = self.Process(target=self._writefd, args=(child_conn, b"bar", True))
+        p.daemon = True
         p.start()
         with open(test_support.TESTFN, "wb") as f:
             fd = f.fileno()
@@ -1632,11 +1645,13 @@
 
         lconn, lconn0 = self.Pipe()
         lp = self.Process(target=self._listener, args=(lconn0, families))
+        lp.daemon = True
         lp.start()
         lconn0.close()
 
         rconn, rconn0 = self.Pipe()
         rp = self.Process(target=self._remote, args=(rconn0,))
+        rp.daemon = True
         rp.start()
         rconn0.close()
 
@@ -1774,6 +1789,7 @@
         string.value = latin('hello')
 
         p = self.Process(target=self._double, args=(x, y, foo, arr, string))
+        p.daemon = True
         p.start()
         p.join()
 
@@ -1846,6 +1862,7 @@
         conn, child_conn = self.Pipe()
 
         p = self.Process(target=self._test_finalize, args=(child_conn,))
+        p.daemon = True
         p.start()
         p.join()
 
@@ -1915,12 +1932,16 @@
         reader, writer = multiprocessing.Pipe(duplex=False)
 
         logger.setLevel(LEVEL1)
-        self.Process(target=self._test_level, args=(writer,)).start()
+        p = self.Process(target=self._test_level, args=(writer,))
+        p.daemon = True
+        p.start()
         self.assertEqual(LEVEL1, reader.recv())
 
         logger.setLevel(logging.NOTSET)
         root_logger.setLevel(LEVEL2)
-        self.Process(target=self._test_level, args=(writer,)).start()
+        p = self.Process(target=self._test_level, args=(writer,))
+        p.daemon = True
+        p.start()
         self.assertEqual(LEVEL2, reader.recv())
 
         root_logger.setLevel(root_level)
@@ -2106,6 +2127,7 @@
 def _TestProcess(q):
     queue = multiprocessing.Queue()
     subProc = multiprocessing.Process(target=_ThisSubProcess, args=(queue,))
+    subProc.daemon = True
     subProc.start()
     subProc.join()
 
@@ -2142,11 +2164,13 @@
     def test_queue_in_process(self):
         queue = multiprocessing.Queue()
         proc = multiprocessing.Process(target=_TestProcess, args=(queue,))
+        proc.daemon = True
         proc.start()
         proc.join()
 
     def test_pool_in_process(self):
         p = multiprocessing.Process(target=pool_in_process)
+        p.daemon = True
         p.start()
         p.join()
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  9 22:17:16 2011
From: python-checkins at python.org (jesus.cea)
Date: Fri, 09 Sep 2011 22:17:16 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_issue_=2312?=
 =?utf8?q?948=3A_multiprocessing_test_failures_can_hang_the_buildbots?=
Message-ID: <E1R27Vk-0006yk-Ar@dinsdale.python.org>

http://hg.python.org/cpython/rev/9f4d72da69a8
changeset:   72323:9f4d72da69a8
branch:      2.7
user:        Jesus Cea <jcea at jcea.es>
date:        Fri Sep 09 22:15:16 2011 +0200
summary:
  Fix issue #12948: multiprocessing test failures can hang the buildbots

files:
  Lib/test/test_multiprocessing.py |  2 --
  1 files changed, 0 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -2164,13 +2164,11 @@
     def test_queue_in_process(self):
         queue = multiprocessing.Queue()
         proc = multiprocessing.Process(target=_TestProcess, args=(queue,))
-        proc.daemon = True
         proc.start()
         proc.join()
 
     def test_pool_in_process(self):
         p = multiprocessing.Process(target=pool_in_process)
-        p.daemon = True
         p.start()
         p.join()
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  9 22:17:17 2011
From: python-checkins at python.org (jesus.cea)
Date: Fri, 09 Sep 2011 22:17:17 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_issue_=2312?=
 =?utf8?q?948=3A_multiprocessing_test_failures_can_hang_the_buildbots?=
Message-ID: <E1R27Vl-0006yo-1x@dinsdale.python.org>

http://hg.python.org/cpython/rev/559ea53e25df
changeset:   72324:559ea53e25df
branch:      3.2
parent:      72320:ebeb58f3de39
user:        Jesus Cea <jcea at jcea.es>
date:        Fri Sep 09 22:15:16 2011 +0200
summary:
  Fix issue #12948: multiprocessing test failures can hang the buildbots

files:
  Lib/test/test_multiprocessing.py |  2 --
  1 files changed, 0 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -2220,13 +2220,11 @@
     def test_queue_in_process(self):
         queue = multiprocessing.Queue()
         proc = multiprocessing.Process(target=_TestProcess, args=(queue,))
-        proc.daemon = True
         proc.start()
         proc.join()
 
     def test_pool_in_process(self):
         p = multiprocessing.Process(target=pool_in_process)
-        p.daemon = True
         p.start()
         p.join()
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  9 22:17:18 2011
From: python-checkins at python.org (jesus.cea)
Date: Fri, 09 Sep 2011 22:17:18 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Fix_issue_=2312948=3A_multiprocessing_test_failures_can_hang?=
 =?utf8?q?_the_buildbots?=
Message-ID: <E1R27Vm-0006ys-5g@dinsdale.python.org>

http://hg.python.org/cpython/rev/c9a72fb5968d
changeset:   72325:c9a72fb5968d
parent:      72321:1e189d55721c
parent:      72324:559ea53e25df
user:        Jesus Cea <jcea at jcea.es>
date:        Fri Sep 09 22:16:57 2011 +0200
summary:
  Fix issue #12948: multiprocessing test failures can hang the buildbots

files:
  Lib/test/test_multiprocessing.py |  2 --
  1 files changed, 0 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -2275,13 +2275,11 @@
     def test_queue_in_process(self):
         queue = multiprocessing.Queue()
         proc = multiprocessing.Process(target=_TestProcess, args=(queue,))
-        proc.daemon = True
         proc.start()
         proc.join()
 
     def test_pool_in_process(self):
         p = multiprocessing.Process(target=pool_in_process)
-        p.daemon = True
         p.start()
         p.join()
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep  9 23:55:46 2011
From: python-checkins at python.org (jesus.cea)
Date: Fri, 09 Sep 2011 23:55:46 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Close_issue_12952=3A_Solari?=
 =?utf8?q?s/Illumos_=28OpenIndiana=29_Scheduling_policies?=
Message-ID: <E1R2934-0006gb-6N@dinsdale.python.org>

http://hg.python.org/cpython/rev/e1644c8edc3e
changeset:   72326:e1644c8edc3e
user:        Jesus Cea <jcea at jcea.es>
date:        Fri Sep 09 23:55:42 2011 +0200
summary:
  Close issue 12952: Solaris/Illumos (OpenIndiana) Scheduling policies

files:
  Modules/posixmodule.c |  14 +++++++++++++-
  1 files changed, 13 insertions(+), 1 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -11301,12 +11301,12 @@
 #endif
 
 #ifdef HAVE_SCHED_H
+    if (ins(d, "SCHED_OTHER", (long)SCHED_OTHER)) return -1;
     if (ins(d, "SCHED_FIFO", (long)SCHED_FIFO)) return -1;
     if (ins(d, "SCHED_RR", (long)SCHED_RR)) return -1;
 #ifdef SCHED_SPORADIC
     if (ins(d, "SCHED_SPORADIC", (long)SCHED_SPORADIC) return -1;
 #endif
-    if (ins(d, "SCHED_OTHER", (long)SCHED_OTHER)) return -1;
 #ifdef SCHED_BATCH
     if (ins(d, "SCHED_BATCH", (long)SCHED_BATCH)) return -1;
 #endif
@@ -11316,6 +11316,18 @@
 #ifdef SCHED_RESET_ON_FORK
     if (ins(d, "SCHED_RESET_ON_FORK", (long)SCHED_RESET_ON_FORK)) return -1;
 #endif
+#ifdef SCHED_SYS
+    if (ins(d, "SCHED_SYS", (long)SCHED_SYS)) return -1;
+#endif
+#ifdef SCHED_IA
+    if (ins(d, "SCHED_IA", (long)SCHED_IA)) return -1;
+#endif
+#ifdef SCHED_FSS
+    if (ins(d, "SCHED_FSS", (long)SCHED_FSS)) return -1;
+#endif
+#ifdef SCHED_FX
+    if (ins(d, "SCHED_FX", (long)SCHED_FSS)) return -1;
+#endif
 #endif
 
 #ifdef HAVE_ATTR_XATTR_H

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 10 01:17:29 2011
From: python-checkins at python.org (jesus.cea)
Date: Sat, 10 Sep 2011 01:17:29 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Better_fix_for_=2312763=3A_?=
 =?utf8?q?test=5Fposix_failure_on_OpenIndiana?=
Message-ID: <E1R2AK9-0004NP-Km@dinsdale.python.org>

http://hg.python.org/cpython/rev/dba886806eb3
changeset:   72327:dba886806eb3
user:        Jesus Cea <jcea at jcea.es>
date:        Sat Sep 10 01:16:55 2011 +0200
summary:
  Better fix for #12763: test_posix failure on OpenIndiana

files:
  Lib/test/test_posix.py |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py
--- a/Lib/test/test_posix.py
+++ b/Lib/test/test_posix.py
@@ -862,12 +862,12 @@
         mine = posix.sched_getscheduler(0)
         self.assertIn(mine, possible_schedulers)
         try:
-            init = posix.sched_getscheduler(1)
+            parent = posix.sched_getscheduler(os.getppid())
         except OSError as e:
-            if e.errno != errno.EPERM and e.errno != errno.ESRCH:
+            if e.errno != errno.EPERM:
                 raise
         else:
-            self.assertIn(init, possible_schedulers)
+            self.assertIn(parent, possible_schedulers)
         self.assertRaises(OSError, posix.sched_getscheduler, -1)
         self.assertRaises(OSError, posix.sched_getparam, -1)
         param = posix.sched_getparam(0)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 10 01:40:59 2011
From: python-checkins at python.org (jesus.cea)
Date: Sat, 10 Sep 2011 01:40:59 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Yet_another_fix_for_=231276?=
 =?utf8?q?3=3A_test=5Fposix_failure_on_OpenIndiana?=
Message-ID: <E1R2Agt-0005us-Q4@dinsdale.python.org>

http://hg.python.org/cpython/rev/9ef10328180b
changeset:   72328:9ef10328180b
user:        Jesus Cea <jcea at jcea.es>
date:        Sat Sep 10 01:40:52 2011 +0200
summary:
  Yet another fix for #12763: test_posix failure on OpenIndiana

files:
  Modules/posixmodule.c |  8 +++++++-
  1 files changed, 7 insertions(+), 1 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -4741,7 +4741,13 @@
     if (!PyArg_ParseTuple(args, _Py_PARSE_PID "iO&:sched_setscheduler",
                           &pid, &policy, &convert_sched_param, &param))
         return NULL;
-    if (sched_setscheduler(pid, policy, &param))
+
+    /*
+    ** sched_setscheduler() returns 0 in Linux, but
+    ** the previous scheduling policy.
+    ** On error, -1 is returned in all Operative Systems.
+    */
+    if (sched_setscheduler(pid, policy, &param) == -1)
         return posix_error();
     Py_RETURN_NONE;
 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 10 01:53:22 2011
From: python-checkins at python.org (jesus.cea)
Date: Sat, 10 Sep 2011 01:53:22 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Typo?=
Message-ID: <E1R2Ass-0006lQ-WC@dinsdale.python.org>

http://hg.python.org/cpython/rev/36923c7e72f2
changeset:   72329:36923c7e72f2
user:        Jesus Cea <jcea at jcea.es>
date:        Sat Sep 10 01:53:19 2011 +0200
summary:
  Typo

files:
  Modules/posixmodule.c |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -4743,9 +4743,9 @@
         return NULL;
 
     /*
-    ** sched_setscheduler() returns 0 in Linux, but
-    ** the previous scheduling policy.
-    ** On error, -1 is returned in all Operative Systems.
+    ** sched_setscheduler() returns 0 in Linux, but the previous
+    ** scheduling policy under Solaris/Illumos, and others.
+    ** On error, -1 is returned in all Operating Systems.
     */
     if (sched_setscheduler(pid, policy, &param) == -1)
         return posix_error();

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 10 04:04:16 2011
From: python-checkins at python.org (jesus.cea)
Date: Sat, 10 Sep 2011 04:04:16 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Close_=2312950=3A_multiproc?=
 =?utf8?q?essing_=22test=5Ffd=5Ftransfer=22_fails_under_OpenIndiana?=
Message-ID: <E1R2CvY-0006Sp-AH@dinsdale.python.org>

http://hg.python.org/cpython/rev/1fde7cf94c76
changeset:   72330:1fde7cf94c76
user:        Jesus Cea <jcea at jcea.es>
date:        Sat Sep 10 04:04:09 2011 +0200
summary:
  Close #12950: multiprocessing "test_fd_transfer" fails under OpenIndiana

files:
  Modules/_multiprocessing/multiprocessing.c |  34 ++++++---
  1 files changed, 23 insertions(+), 11 deletions(-)


diff --git a/Modules/_multiprocessing/multiprocessing.c b/Modules/_multiprocessing/multiprocessing.c
--- a/Modules/_multiprocessing/multiprocessing.c
+++ b/Modules/_multiprocessing/multiprocessing.c
@@ -86,31 +86,37 @@
 /* Functions for transferring file descriptors between processes.
    Reimplements some of the functionality of the fdcred
    module at http://www.mca-ltd.com/resources/fdcred_1.tgz. */
+/* Based in http://resin.csoft.net/cgi-bin/man.cgi?section=3&topic=CMSG_DATA */
 
 static PyObject *
 multiprocessing_sendfd(PyObject *self, PyObject *args)
 {
     int conn, fd, res;
+    struct iovec dummy_iov;
     char dummy_char;
-    char buf[CMSG_SPACE(sizeof(int))];
-    struct msghdr msg = {0};
-    struct iovec dummy_iov;
+    struct msghdr msg;
     struct cmsghdr *cmsg;
+    union {
+        struct cmsghdr hdr;
+        unsigned char buf[CMSG_SPACE(sizeof(int))];
+    } cmsgbuf;
 
     if (!PyArg_ParseTuple(args, "ii", &conn, &fd))
         return NULL;
 
     dummy_iov.iov_base = &dummy_char;
     dummy_iov.iov_len = 1;
-    msg.msg_control = buf;
-    msg.msg_controllen = sizeof(buf);
+
+    memset(&msg, 0, sizeof(msg));
+    msg.msg_control = &cmsgbuf.buf;
+    msg.msg_controllen = sizeof(cmsgbuf.buf);
     msg.msg_iov = &dummy_iov;
     msg.msg_iovlen = 1;
+
     cmsg = CMSG_FIRSTHDR(&msg);
+    cmsg->cmsg_len = CMSG_LEN(sizeof(int));
     cmsg->cmsg_level = SOL_SOCKET;
     cmsg->cmsg_type = SCM_RIGHTS;
-    cmsg->cmsg_len = CMSG_LEN(sizeof(int));
-    msg.msg_controllen = cmsg->cmsg_len;
     * (int *) CMSG_DATA(cmsg) = fd;
 
     Py_BEGIN_ALLOW_THREADS
@@ -127,20 +133,26 @@
 {
     int conn, fd, res;
     char dummy_char;
-    char buf[CMSG_SPACE(sizeof(int))];
+    struct iovec dummy_iov;
     struct msghdr msg = {0};
-    struct iovec dummy_iov;
     struct cmsghdr *cmsg;
+    union {
+        struct cmsghdr hdr;
+        unsigned char buf[CMSG_SPACE(sizeof(int))];
+    } cmsgbuf;
 
     if (!PyArg_ParseTuple(args, "i", &conn))
         return NULL;
 
     dummy_iov.iov_base = &dummy_char;
     dummy_iov.iov_len = 1;
-    msg.msg_control = buf;
-    msg.msg_controllen = sizeof(buf);
+
+    memset(&msg, 0, sizeof(msg));
+    msg.msg_control = &cmsgbuf.buf;
+    msg.msg_controllen = sizeof(cmsgbuf.buf);
     msg.msg_iov = &dummy_iov;
     msg.msg_iovlen = 1;
+
     cmsg = CMSG_FIRSTHDR(&msg);
     cmsg->cmsg_level = SOL_SOCKET;
     cmsg->cmsg_type = SCM_RIGHTS;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 10 04:09:21 2011
From: python-checkins at python.org (jesus.cea)
Date: Sat, 10 Sep 2011 04:09:21 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogQ2xvc2UgIzEyOTUw?=
 =?utf8?q?=3A_multiprocessing_=22test=5Ffd=5Ftransfer=22_fails_under_OpenI?=
 =?utf8?q?ndiana?=
Message-ID: <E1R2D0T-0006je-D7@dinsdale.python.org>

http://hg.python.org/cpython/rev/cd15473a9de2
changeset:   72331:cd15473a9de2
branch:      2.7
parent:      72323:9f4d72da69a8
user:        Jesus Cea <jcea at jcea.es>
date:        Sat Sep 10 04:04:09 2011 +0200
summary:
  Close #12950: multiprocessing "test_fd_transfer" fails under OpenIndiana

files:
  Modules/_multiprocessing/multiprocessing.c |  34 ++++++---
  1 files changed, 23 insertions(+), 11 deletions(-)


diff --git a/Modules/_multiprocessing/multiprocessing.c b/Modules/_multiprocessing/multiprocessing.c
--- a/Modules/_multiprocessing/multiprocessing.c
+++ b/Modules/_multiprocessing/multiprocessing.c
@@ -97,31 +97,37 @@
 /* Functions for transferring file descriptors between processes.
    Reimplements some of the functionality of the fdcred
    module at http://www.mca-ltd.com/resources/fdcred_1.tgz. */
+/* Based in http://resin.csoft.net/cgi-bin/man.cgi?section=3&topic=CMSG_DATA */
 
 static PyObject *
 multiprocessing_sendfd(PyObject *self, PyObject *args)
 {
     int conn, fd, res;
+    struct iovec dummy_iov;
     char dummy_char;
-    char buf[CMSG_SPACE(sizeof(int))];
-    struct msghdr msg = {0};
-    struct iovec dummy_iov;
+    struct msghdr msg;
     struct cmsghdr *cmsg;
+    union {
+        struct cmsghdr hdr;
+        unsigned char buf[CMSG_SPACE(sizeof(int))];
+    } cmsgbuf;
 
     if (!PyArg_ParseTuple(args, "ii", &conn, &fd))
         return NULL;
 
     dummy_iov.iov_base = &dummy_char;
     dummy_iov.iov_len = 1;
-    msg.msg_control = buf;
-    msg.msg_controllen = sizeof(buf);
+
+    memset(&msg, 0, sizeof(msg));
+    msg.msg_control = &cmsgbuf.buf;
+    msg.msg_controllen = sizeof(cmsgbuf.buf);
     msg.msg_iov = &dummy_iov;
     msg.msg_iovlen = 1;
+
     cmsg = CMSG_FIRSTHDR(&msg);
+    cmsg->cmsg_len = CMSG_LEN(sizeof(int));
     cmsg->cmsg_level = SOL_SOCKET;
     cmsg->cmsg_type = SCM_RIGHTS;
-    cmsg->cmsg_len = CMSG_LEN(sizeof(int));
-    msg.msg_controllen = cmsg->cmsg_len;
     * (int *) CMSG_DATA(cmsg) = fd;
 
     Py_BEGIN_ALLOW_THREADS
@@ -138,20 +144,26 @@
 {
     int conn, fd, res;
     char dummy_char;
-    char buf[CMSG_SPACE(sizeof(int))];
+    struct iovec dummy_iov;
     struct msghdr msg = {0};
-    struct iovec dummy_iov;
     struct cmsghdr *cmsg;
+    union {
+        struct cmsghdr hdr;
+        unsigned char buf[CMSG_SPACE(sizeof(int))];
+    } cmsgbuf;
 
     if (!PyArg_ParseTuple(args, "i", &conn))
         return NULL;
 
     dummy_iov.iov_base = &dummy_char;
     dummy_iov.iov_len = 1;
-    msg.msg_control = buf;
-    msg.msg_controllen = sizeof(buf);
+
+    memset(&msg, 0, sizeof(msg));
+    msg.msg_control = &cmsgbuf.buf;
+    msg.msg_controllen = sizeof(cmsgbuf.buf);
     msg.msg_iov = &dummy_iov;
     msg.msg_iovlen = 1;
+
     cmsg = CMSG_FIRSTHDR(&msg);
     cmsg->cmsg_level = SOL_SOCKET;
     cmsg->cmsg_type = SCM_RIGHTS;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 10 04:14:10 2011
From: python-checkins at python.org (jesus.cea)
Date: Sat, 10 Sep 2011 04:14:10 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogQ2xvc2UgIzEyOTUw?=
 =?utf8?q?=3A_multiprocessing_=22test=5Ffd=5Ftransfer=22_fails_under_OpenI?=
 =?utf8?q?ndiana?=
Message-ID: <E1R2D58-0006yj-9p@dinsdale.python.org>

http://hg.python.org/cpython/rev/0f50b8379614
changeset:   72332:0f50b8379614
branch:      3.2
parent:      72324:559ea53e25df
user:        Jesus Cea <jcea at jcea.es>
date:        Sat Sep 10 04:04:09 2011 +0200
summary:
  Close #12950: multiprocessing "test_fd_transfer" fails under OpenIndiana

files:
  Modules/_multiprocessing/multiprocessing.c |  34 ++++++---
  1 files changed, 23 insertions(+), 11 deletions(-)


diff --git a/Modules/_multiprocessing/multiprocessing.c b/Modules/_multiprocessing/multiprocessing.c
--- a/Modules/_multiprocessing/multiprocessing.c
+++ b/Modules/_multiprocessing/multiprocessing.c
@@ -97,31 +97,37 @@
 /* Functions for transferring file descriptors between processes.
    Reimplements some of the functionality of the fdcred
    module at http://www.mca-ltd.com/resources/fdcred_1.tgz. */
+/* Based in http://resin.csoft.net/cgi-bin/man.cgi?section=3&topic=CMSG_DATA */
 
 static PyObject *
 multiprocessing_sendfd(PyObject *self, PyObject *args)
 {
     int conn, fd, res;
+    struct iovec dummy_iov;
     char dummy_char;
-    char buf[CMSG_SPACE(sizeof(int))];
-    struct msghdr msg = {0};
-    struct iovec dummy_iov;
+    struct msghdr msg;
     struct cmsghdr *cmsg;
+    union {
+        struct cmsghdr hdr;
+        unsigned char buf[CMSG_SPACE(sizeof(int))];
+    } cmsgbuf;
 
     if (!PyArg_ParseTuple(args, "ii", &conn, &fd))
         return NULL;
 
     dummy_iov.iov_base = &dummy_char;
     dummy_iov.iov_len = 1;
-    msg.msg_control = buf;
-    msg.msg_controllen = sizeof(buf);
+
+    memset(&msg, 0, sizeof(msg));
+    msg.msg_control = &cmsgbuf.buf;
+    msg.msg_controllen = sizeof(cmsgbuf.buf);
     msg.msg_iov = &dummy_iov;
     msg.msg_iovlen = 1;
+
     cmsg = CMSG_FIRSTHDR(&msg);
+    cmsg->cmsg_len = CMSG_LEN(sizeof(int));
     cmsg->cmsg_level = SOL_SOCKET;
     cmsg->cmsg_type = SCM_RIGHTS;
-    cmsg->cmsg_len = CMSG_LEN(sizeof(int));
-    msg.msg_controllen = cmsg->cmsg_len;
     * (int *) CMSG_DATA(cmsg) = fd;
 
     Py_BEGIN_ALLOW_THREADS
@@ -138,20 +144,26 @@
 {
     int conn, fd, res;
     char dummy_char;
-    char buf[CMSG_SPACE(sizeof(int))];
+    struct iovec dummy_iov;
     struct msghdr msg = {0};
-    struct iovec dummy_iov;
     struct cmsghdr *cmsg;
+    union {
+        struct cmsghdr hdr;
+        unsigned char buf[CMSG_SPACE(sizeof(int))];
+    } cmsgbuf;
 
     if (!PyArg_ParseTuple(args, "i", &conn))
         return NULL;
 
     dummy_iov.iov_base = &dummy_char;
     dummy_iov.iov_len = 1;
-    msg.msg_control = buf;
-    msg.msg_controllen = sizeof(buf);
+
+    memset(&msg, 0, sizeof(msg));
+    msg.msg_control = &cmsgbuf.buf;
+    msg.msg_controllen = sizeof(cmsgbuf.buf);
     msg.msg_iov = &dummy_iov;
     msg.msg_iovlen = 1;
+
     cmsg = CMSG_FIRSTHDR(&msg);
     cmsg->cmsg_level = SOL_SOCKET;
     cmsg->cmsg_type = SCM_RIGHTS;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 10 04:14:11 2011
From: python-checkins at python.org (jesus.cea)
Date: Sat, 10 Sep 2011 04:14:11 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_MERGE=3A_Close_=2312950=3A_multiprocessing_=22test=5Ffd=5Ftr?=
 =?utf8?q?ansfer=22_fails_under_OpenIndiana?=
Message-ID: <E1R2D59-0006yn-0x@dinsdale.python.org>

http://hg.python.org/cpython/rev/e37488e78cfa
changeset:   72333:e37488e78cfa
parent:      72330:1fde7cf94c76
parent:      72332:0f50b8379614
user:        Jesus Cea <jcea at jcea.es>
date:        Sat Sep 10 04:14:04 2011 +0200
summary:
  MERGE: Close #12950: multiprocessing "test_fd_transfer" fails under OpenIndiana

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 10 04:40:30 2011
From: python-checkins at python.org (jesus.cea)
Date: Sat, 10 Sep 2011 04:40:30 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_NEWS?=
Message-ID: <E1R2DUc-0000Fe-Ty@dinsdale.python.org>

http://hg.python.org/cpython/rev/8812996e4062
changeset:   72334:8812996e4062
branch:      2.7
parent:      72331:cd15473a9de2
user:        Jesus Cea <jcea at jcea.es>
date:        Sat Sep 10 04:35:18 2011 +0200
summary:
  NEWS

files:
  Misc/NEWS |  3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -188,6 +188,9 @@
 Extension Modules
 -----------------
 
+- Issue #12950: Fix passing file descriptors in multiprocessing, under
+  OpenIndiana/Illumos.
+
 - Issue #12764: Fix a crash in ctypes when the name of a Structure field is not
   a string.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 10 04:40:31 2011
From: python-checkins at python.org (jesus.cea)
Date: Sat, 10 Sep 2011 04:40:31 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_NEWS?=
Message-ID: <E1R2DUd-0000Fo-M2@dinsdale.python.org>

http://hg.python.org/cpython/rev/1c5463cad0ae
changeset:   72335:1c5463cad0ae
branch:      3.2
parent:      72332:0f50b8379614
user:        Jesus Cea <jcea at jcea.es>
date:        Sat Sep 10 04:37:07 2011 +0200
summary:
  NEWS

files:
  Misc/NEWS |  5 +++++
  1 files changed, 5 insertions(+), 0 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -58,6 +58,11 @@
 
 - Issue #12821: Fix test_fcntl failures on OpenBSD 5.
 
+Extension Modules
+-----------------
+ 
+- Issue #12950: Fix passing file descriptors in multiprocessing, under
+  OpenIndiana/Illumos.
 
 What's New in Python 3.2.2?
 ===========================

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 10 04:40:32 2011
From: python-checkins at python.org (jesus.cea)
Date: Sat, 10 Sep 2011 04:40:32 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_NEWS?=
Message-ID: <E1R2DUe-0000Fs-E3@dinsdale.python.org>

http://hg.python.org/cpython/rev/622ecd28e5d1
changeset:   72336:622ecd28e5d1
parent:      72333:e37488e78cfa
parent:      72335:1c5463cad0ae
user:        Jesus Cea <jcea at jcea.es>
date:        Sat Sep 10 04:40:13 2011 +0200
summary:
  NEWS

files:
  Misc/NEWS |  3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1265,6 +1265,9 @@
 Extension Modules
 -----------------
 
+- Issue #12950: Fix passing file descriptors in multiprocessing, under
+  OpenIndiana/Illumos.
+
 - Issue #12764: Fix a crash in ctypes when the name of a Structure field is not
   a string.
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Sat Sep 10 05:18:48 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sat, 10 Sep 2011 05:18:48 +0200
Subject: [Python-checkins] Daily reference leaks (36923c7e72f2): sum=0
Message-ID: <E1R2E5g-0003fA-EB@ap.vmr.nerim.net>

results for 36923c7e72f2 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog5hFxMm', '-x']

From python-checkins at python.org  Sat Sep 10 09:13:23 2011
From: python-checkins at python.org (ezio.melotti)
Date: Sat, 10 Sep 2011 09:13:23 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzEyOTQwOiBmaXgg?=
 =?utf8?q?cmd_example=2E__Patch_by_Tim_Chase=2E?=
Message-ID: <E1R2Hkh-0001uH-KX@dinsdale.python.org>

http://hg.python.org/cpython/rev/1ab62d3b96d2
changeset:   72337:1ab62d3b96d2
branch:      3.2
parent:      72335:1c5463cad0ae
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Sat Sep 10 10:06:01 2011 +0300
summary:
  #12940: fix cmd example.  Patch by Tim Chase.

files:
  Doc/library/cmd.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/cmd.rst b/Doc/library/cmd.rst
--- a/Doc/library/cmd.rst
+++ b/Doc/library/cmd.rst
@@ -247,7 +247,7 @@
             right(*parse(arg))
         def do_left(self, arg):
             'Turn turtle left by given number of degrees:  LEFT 90'
-            right(*parse(arg))
+            left(*parse(arg))
         def do_goto(self, arg):
             'Move turtle to an absolute position with changing orientation.  GOTO 100 200'
             goto(*parse(arg))

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 10 09:13:24 2011
From: python-checkins at python.org (ezio.melotti)
Date: Sat, 10 Sep 2011 09:13:24 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_=2312940=3A_merge_with_3=2E2=2E?=
Message-ID: <E1R2Hki-0001uL-Ap@dinsdale.python.org>

http://hg.python.org/cpython/rev/b239b7d9f752
changeset:   72338:b239b7d9f752
parent:      72336:622ecd28e5d1
parent:      72337:1ab62d3b96d2
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Sat Sep 10 10:08:13 2011 +0300
summary:
  #12940: merge with 3.2.

files:
  Doc/library/cmd.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/cmd.rst b/Doc/library/cmd.rst
--- a/Doc/library/cmd.rst
+++ b/Doc/library/cmd.rst
@@ -247,7 +247,7 @@
             right(*parse(arg))
         def do_left(self, arg):
             'Turn turtle left by given number of degrees:  LEFT 90'
-            right(*parse(arg))
+            left(*parse(arg))
         def do_goto(self, arg):
             'Move turtle to an absolute position with changing orientation.  GOTO 100 200'
             goto(*parse(arg))

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Sun Sep 11 05:19:03 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sun, 11 Sep 2011 05:19:03 +0200
Subject: [Python-checkins] Daily reference leaks (b239b7d9f752): sum=0
Message-ID: <E1R2aZT-0003ox-0c@ap.vmr.nerim.net>

results for b239b7d9f752 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog9ME5J4', '-x']

From python-checkins at python.org  Sun Sep 11 18:55:40 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Sun, 11 Sep 2011 18:55:40 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_add_ChainMap_to_=5F=5Fall?=
	=?utf8?b?X18gKGNsb3NlcyAjMTI5NTkp?=
Message-ID: <E1R2nJk-0000zC-OY@dinsdale.python.org>

http://hg.python.org/cpython/rev/12bb3cd873c8
changeset:   72339:12bb3cd873c8
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Sep 11 12:55:34 2011 -0400
summary:
  add ChainMap to __all__ (closes #12959)

Thanks July Tikhonov.

files:
  Lib/collections/__init__.py |  2 +-
  Misc/NEWS                   |  2 ++
  2 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py
--- a/Lib/collections/__init__.py
+++ b/Lib/collections/__init__.py
@@ -1,5 +1,5 @@
 __all__ = ['deque', 'defaultdict', 'namedtuple', 'UserDict', 'UserList',
-            'UserString', 'Counter', 'OrderedDict']
+            'UserString', 'Counter', 'OrderedDict', 'ChainMap']
 
 # For backwards compatibility, continue to make the collections ABCs
 # available through the collections module.
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -274,6 +274,8 @@
 Library
 -------
 
+- Issue #12959: Add collections.ChainMap to collections.__all__.
+
 - Issue #12567: Add curses.unget_wch() function. Push a character so the next
   get_wch() will return it.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep 11 22:38:35 2011
From: python-checkins at python.org (nadeem.vawda)
Date: Sun, 11 Sep 2011 22:38:35 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_BZ2File_now_uses_the_compre?=
 =?utf8?q?sslevel_argument_given_by_the_caller=2C?=
Message-ID: <E1R2qnT-00040o-Sz@dinsdale.python.org>

http://hg.python.org/cpython/rev/d3ff5109f5fd
changeset:   72340:d3ff5109f5fd
user:        Nadeem Vawda <nadeem.vawda at gmail.com>
date:        Sun Sep 11 22:38:11 2011 +0200
summary:
  BZ2File now uses the compresslevel argument given by the caller,
instead of ignoring it and always using a compression level of 9.

files:
  Lib/bz2.py           |  4 ++--
  Lib/test/test_bz2.py |  7 +++++++
  2 files changed, 9 insertions(+), 2 deletions(-)


diff --git a/Lib/bz2.py b/Lib/bz2.py
--- a/Lib/bz2.py
+++ b/Lib/bz2.py
@@ -75,11 +75,11 @@
         elif mode in ("w", "wb"):
             mode = "wb"
             mode_code = _MODE_WRITE
-            self._compressor = BZ2Compressor()
+            self._compressor = BZ2Compressor(compresslevel)
         elif mode in ("a", "ab"):
             mode = "ab"
             mode_code = _MODE_WRITE
-            self._compressor = BZ2Compressor()
+            self._compressor = BZ2Compressor(compresslevel)
         else:
             raise ValueError("Invalid mode: {!r}".format(mode))
 
diff --git a/Lib/test/test_bz2.py b/Lib/test/test_bz2.py
--- a/Lib/test/test_bz2.py
+++ b/Lib/test/test_bz2.py
@@ -224,6 +224,13 @@
         with open(self.filename, 'rb') as f:
             self.assertEqual(self.decompress(f.read()), self.TEXT)
 
+    def testWriteNonDefaultCompressLevel(self):
+        expected = bz2.compress(self.TEXT, compresslevel=5)
+        with BZ2File(self.filename, "w", compresslevel=5) as bz2f:
+            bz2f.write(self.TEXT)
+        with open(self.filename, "rb") as f:
+            self.assertEqual(f.read(), expected)
+
     def testWriteLines(self):
         with BZ2File(self.filename, "w") as bz2f:
             self.assertRaises(TypeError, bz2f.writelines)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 00:07:13 2011
From: python-checkins at python.org (nadeem.vawda)
Date: Mon, 12 Sep 2011 00:07:13 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312306=3A_Add_ZLIB?=
 =?utf8?q?=5FRUNTIME=5FVERSION_to_the_zlib_module=2E?=
Message-ID: <E1R2sBF-0001XN-Sc@dinsdale.python.org>

http://hg.python.org/cpython/rev/b21d1de6d78e
changeset:   72341:b21d1de6d78e
user:        Nadeem Vawda <nadeem.vawda at gmail.com>
date:        Mon Sep 12 00:04:13 2011 +0200
summary:
  Issue #12306: Add ZLIB_RUNTIME_VERSION to the zlib module.

While we're at it, also document ZLIB_VERSION.

Patch by Torsten Landschoff.

files:
  Doc/library/zlib.rst  |  21 +++++++++++++++++++++
  Lib/test/test_zlib.py |  12 ++++++++++++
  Misc/NEWS             |   3 +++
  Modules/zlibmodule.c  |   4 ++++
  4 files changed, 40 insertions(+), 0 deletions(-)


diff --git a/Doc/library/zlib.rst b/Doc/library/zlib.rst
--- a/Doc/library/zlib.rst
+++ b/Doc/library/zlib.rst
@@ -122,6 +122,7 @@
    won't fit into memory at once.  The *wbits* parameter controls the size of the
    window buffer.
 
+
 Compression objects support the following methods:
 
 
@@ -217,6 +218,26 @@
    seeks into the stream at a future point.
 
 
+Information about the version of the zlib library in use is available through
+the following constants:
+
+
+.. data:: ZLIB_VERSION
+
+   The version string of the zlib library that was used for building the module.
+   This may be different from the zlib library actually used at runtime, which
+   is available as :const:`ZLIB_RUNTIME_VERSION`.
+
+   .. versionadded:: 3.3
+
+
+.. data:: ZLIB_RUNTIME_VERSION
+
+   The version string of the zlib library actually loaded by the interpreter.
+
+   .. versionadded:: 3.3
+
+
 .. seealso::
 
    Module :mod:`gzip`
diff --git a/Lib/test/test_zlib.py b/Lib/test/test_zlib.py
--- a/Lib/test/test_zlib.py
+++ b/Lib/test/test_zlib.py
@@ -13,6 +13,17 @@
     mmap = None
 
 
+class VersionTestCase(unittest.TestCase):
+
+    def test_library_version(self):
+        # On the build system, ZLIB_RUNTIME_VERSION should match ZLIB_VERSION.
+        # ZLIB_RUNTIME_VERSION is the actual library version while ZLIB_VERSION
+        # is the version from the header file. On the build system, the headers
+        # should match with the library exactly. At runtime, only the first
+        # digit is required to match.
+        self.assertEqual(zlib.ZLIB_RUNTIME_VERSION, zlib.ZLIB_VERSION)
+
+
 class ChecksumTestCase(unittest.TestCase):
     # checksum test cases
     def test_crc32start(self):
@@ -647,6 +658,7 @@
 
 def test_main():
     support.run_unittest(
+        VersionTestCase,
         ChecksumTestCase,
         ChecksumBigBufferTestCase,
         ExceptionTestCase,
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -274,6 +274,9 @@
 Library
 -------
 
+- Issue #12306: Expose the runtime version of the zlib C library as a constant,
+  ZLIB_RUNTIME_VERSION, in the zlib module. Patch by Torsten Landschoff.
+
 - Issue #12959: Add collections.ChainMap to collections.__all__.
 
 - Issue #12567: Add curses.unget_wch() function. Push a character so the next
diff --git a/Modules/zlibmodule.c b/Modules/zlibmodule.c
--- a/Modules/zlibmodule.c
+++ b/Modules/zlibmodule.c
@@ -1169,6 +1169,10 @@
     if (ver != NULL)
         PyModule_AddObject(m, "ZLIB_VERSION", ver);
 
+    ver = PyUnicode_FromString(zlibVersion());
+    if (ver != NULL)
+        PyModule_AddObject(m, "ZLIB_RUNTIME_VERSION", ver);
+
     PyModule_AddStringConstant(m, "__version__", "1.0");
 
     return m;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 00:07:14 2011
From: python-checkins at python.org (nadeem.vawda)
Date: Mon, 12 Sep 2011 00:07:14 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Terminology_fix=3A_=2Egz_an?=
 =?utf8?q?d_=2Ebz2_are_not_archive_formats=2E?=
Message-ID: <E1R2sBG-0001XR-MQ@dinsdale.python.org>

http://hg.python.org/cpython/rev/97a0da942d9c
changeset:   72342:97a0da942d9c
user:        Nadeem Vawda <nadeem.vawda at gmail.com>
date:        Mon Sep 12 00:06:49 2011 +0200
summary:
  Terminology fix: .gz and .bz2 are not archive formats.

files:
  Doc/library/zlib.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/zlib.rst b/Doc/library/zlib.rst
--- a/Doc/library/zlib.rst
+++ b/Doc/library/zlib.rst
@@ -19,7 +19,7 @@
 information.
 
 For reading and writing ``.gz`` files see the :mod:`gzip` module. For
-other archive formats, see the :mod:`bz2`, :mod:`zipfile`, and
+other related file formats, see the :mod:`bz2`, :mod:`zipfile`, and
 :mod:`tarfile` modules.
 
 The available exception and functions in this module are:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 00:09:46 2011
From: python-checkins at python.org (nadeem.vawda)
Date: Mon, 12 Sep 2011 00:09:46 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Oops=2C_zlib=2EZLIB=5FVERSI?=
 =?utf8?q?ON_isn=27t_new_in_3=2E3_-_just_newly-documented=2E=2E=2E?=
Message-ID: <E1R2sDi-0001di-Gr@dinsdale.python.org>

http://hg.python.org/cpython/rev/800a43a3869c
changeset:   72343:800a43a3869c
user:        Nadeem Vawda <nadeem.vawda at gmail.com>
date:        Mon Sep 12 00:09:27 2011 +0200
summary:
  Oops, zlib.ZLIB_VERSION isn't new in 3.3 - just newly-documented...

files:
  Doc/library/zlib.rst |  2 --
  1 files changed, 0 insertions(+), 2 deletions(-)


diff --git a/Doc/library/zlib.rst b/Doc/library/zlib.rst
--- a/Doc/library/zlib.rst
+++ b/Doc/library/zlib.rst
@@ -228,8 +228,6 @@
    This may be different from the zlib library actually used at runtime, which
    is available as :const:`ZLIB_RUNTIME_VERSION`.
 
-   .. versionadded:: 3.3
-
 
 .. data:: ZLIB_RUNTIME_VERSION
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 00:26:30 2011
From: python-checkins at python.org (nadeem.vawda)
Date: Mon, 12 Sep 2011 00:26:30 +0200
Subject: [Python-checkins] =?utf8?q?devguide=3A_experts=2Erst=3A_Apparentl?=
 =?utf8?q?y_the_zlib_module_likes_me=2E?=
Message-ID: <E1R2sTu-0002vW-S4@dinsdale.python.org>

http://hg.python.org/devguide/rev/cd391387854c
changeset:   450:cd391387854c
user:        Nadeem Vawda <nadeem.vawda at gmail.com>
date:        Mon Sep 12 00:26:22 2011 +0200
summary:
  experts.rst: Apparently the zlib module likes me.

files:
  experts.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/experts.rst b/experts.rst
--- a/experts.rst
+++ b/experts.rst
@@ -253,7 +253,7 @@
 xmlrpc                loewis
 zipfile               alanmcintyre
 zipimport
-zlib
+zlib                  nadeem.vawda
 ====================  =============================================
 
 

-- 
Repository URL: http://hg.python.org/devguide

From solipsis at pitrou.net  Mon Sep 12 05:21:14 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Mon, 12 Sep 2011 05:21:14 +0200
Subject: [Python-checkins] Daily reference leaks (800a43a3869c): sum=0
Message-ID: <E1R2x58-0004oZ-Hm@ap.vmr.nerim.net>

results for 800a43a3869c on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog4FYpAW', '-x']

From python-checkins at python.org  Mon Sep 12 16:27:41 2011
From: python-checkins at python.org (stefan.krah)
Date: Mon, 12 Sep 2011 16:27:41 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyOTYz?=
 =?utf8?q?=3A_PyLong=5FAsSize=5Ft=28=29_now_returns_=28size=5Ft=29-1_in_al?=
 =?utf8?q?l_error_cases=2E?=
Message-ID: <E1R37U5-0006I1-Fa@dinsdale.python.org>

http://hg.python.org/cpython/rev/d14f717b5e3d
changeset:   72344:d14f717b5e3d
branch:      3.2
parent:      72337:1ab62d3b96d2
user:        Stefan Krah <skrah at bytereef.org>
date:        Mon Sep 12 16:22:47 2011 +0200
summary:
  Issue #12963: PyLong_AsSize_t() now returns (size_t)-1 in all error cases.

files:
  Objects/longobject.c |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Objects/longobject.c b/Objects/longobject.c
--- a/Objects/longobject.c
+++ b/Objects/longobject.c
@@ -525,8 +525,8 @@
     return x;
 }
 
-/* Get a C unsigned long int from a long int object.
-   Returns -1 and sets an error condition if overflow occurs. */
+/* Get a C size_t from a long int object. Returns (size_t)-1 and sets
+   an error condition if overflow occurs. */
 
 size_t
 PyLong_AsSize_t(PyObject *vv)
@@ -562,7 +562,7 @@
         if ((x >> PyLong_SHIFT) != prev) {
             PyErr_SetString(PyExc_OverflowError,
                 "Python int too large to convert to C size_t");
-            return (unsigned long) -1;
+            return (size_t) -1;
         }
     }
     return x;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 16:27:42 2011
From: python-checkins at python.org (stefan.krah)
Date: Mon, 12 Sep 2011 16:27:42 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_fix_for_issue_=2312963=2E?=
Message-ID: <E1R37U6-0006I6-93@dinsdale.python.org>

http://hg.python.org/cpython/rev/c91900e4e805
changeset:   72345:c91900e4e805
parent:      72343:800a43a3869c
parent:      72344:d14f717b5e3d
user:        Stefan Krah <skrah at bytereef.org>
date:        Mon Sep 12 16:24:48 2011 +0200
summary:
  Merge fix for issue #12963.

files:
  Objects/longobject.c |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Objects/longobject.c b/Objects/longobject.c
--- a/Objects/longobject.c
+++ b/Objects/longobject.c
@@ -525,8 +525,8 @@
     return x;
 }
 
-/* Get a C unsigned long int from a long int object.
-   Returns -1 and sets an error condition if overflow occurs. */
+/* Get a C size_t from a long int object. Returns (size_t)-1 and sets
+   an error condition if overflow occurs. */
 
 size_t
 PyLong_AsSize_t(PyObject *vv)
@@ -562,7 +562,7 @@
         if ((x >> PyLong_SHIFT) != prev) {
             PyErr_SetString(PyExc_OverflowError,
                 "Python int too large to convert to C size_t");
-            return (unsigned long) -1;
+            return (size_t) -1;
         }
     }
     return x;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:42:15 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:42:15 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Slight_cleanup_?=
 =?utf8?q?in_distutils_test=5Fdist=2E?=
Message-ID: <E1R38eF-0005yH-3Y@dinsdale.python.org>

http://hg.python.org/cpython/rev/ab98ab93f715
changeset:   72346:ab98ab93f715
branch:      3.2
parent:      72317:4c7566669be5
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 10 01:34:44 2011 +0200
summary:
  Slight cleanup in distutils test_dist.

I have tests to add in this file and it?s always nice to start from a
clean base.

files:
  Lib/distutils/tests/test_dist.py |  98 +++++++++----------
  1 files changed, 47 insertions(+), 51 deletions(-)


diff --git a/Lib/distutils/tests/test_dist.py b/Lib/distutils/tests/test_dist.py
--- a/Lib/distutils/tests/test_dist.py
+++ b/Lib/distutils/tests/test_dist.py
@@ -74,7 +74,7 @@
         self.assertEqual(d.get_command_packages(),
                          ["distutils.command", "foo.bar", "distutils.tests"])
         cmd = d.get_command_obj("test_dist")
-        self.assertTrue(isinstance(cmd, test_dist))
+        self.assertIsInstance(cmd, test_dist)
         self.assertEqual(cmd.sample_option, "sometext")
 
     def test_command_packages_configfile(self):
@@ -106,28 +106,23 @@
     def test_empty_options(self):
         # an empty options dictionary should not stay in the
         # list of attributes
-        klass = Distribution
 
         # catching warnings
         warns = []
+
         def _warn(msg):
             warns.append(msg)
 
-        old_warn = warnings.warn
+        self.addCleanup(setattr, warnings, 'warn', warnings.warn)
         warnings.warn = _warn
-        try:
-            dist = klass(attrs={'author': 'xxx',
-                                'name': 'xxx',
-                                'version': 'xxx',
-                                'url': 'xxxx',
-                                'options': {}})
-        finally:
-            warnings.warn = old_warn
+        dist = Distribution(attrs={'author': 'xxx', 'name': 'xxx',
+                                   'version': 'xxx', 'url': 'xxxx',
+                                   'options': {}})
 
         self.assertEqual(len(warns), 0)
+        self.assertNotIn('options', dir(dist))
 
     def test_finalize_options(self):
-
         attrs = {'keywords': 'one,two',
                  'platforms': 'one,two'}
 
@@ -150,7 +145,6 @@
         cmds = dist.get_command_packages()
         self.assertEqual(cmds, ['distutils.command', 'one', 'two'])
 
-
     def test_announce(self):
         # make sure the level is known
         dist = Distribution()
@@ -158,6 +152,7 @@
         kwargs = {'level': 'ok2'}
         self.assertRaises(ValueError, dist.announce, args, kwargs)
 
+
 class MetadataTestCase(support.TempdirManager, support.EnvironGuard,
                        unittest.TestCase):
 
@@ -170,15 +165,20 @@
         sys.argv[:] = self.argv[1]
         super(MetadataTestCase, self).tearDown()
 
+    def format_metadata(self, dist):
+        sio = io.StringIO()
+        dist.metadata.write_pkg_file(sio)
+        return sio.getvalue()
+
     def test_simple_metadata(self):
         attrs = {"name": "package",
                  "version": "1.0"}
         dist = Distribution(attrs)
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.0" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.0", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertNotIn("requires:", meta.lower())
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_provides(self):
         attrs = {"name": "package",
@@ -190,9 +190,9 @@
         self.assertEqual(dist.get_provides(),
                          ["package", "package.sub"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("requires:", meta.lower())
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_provides_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -210,11 +210,11 @@
         self.assertEqual(dist.get_requires(),
                          ["other", "another (==1.0)"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("Requires: other" in meta)
-        self.assertTrue("Requires: another (==1.0)" in meta)
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertIn("Requires: other", meta)
+        self.assertIn("Requires: another (==1.0)", meta)
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_requires_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -232,11 +232,11 @@
         self.assertEqual(dist.get_obsoletes(),
                          ["other", "another (<1.0)"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("Obsoletes: other" in meta)
-        self.assertTrue("Obsoletes: another (<1.0)" in meta)
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertNotIn("requires:", meta.lower())
+        self.assertIn("Obsoletes: other", meta)
+        self.assertIn("Obsoletes: another (<1.0)", meta)
 
     def test_obsoletes_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -244,10 +244,20 @@
                            "version": "1.0",
                            "obsoletes": ["my.pkg (splat)"]})
 
-    def format_metadata(self, dist):
-        sio = io.StringIO()
-        dist.metadata.write_pkg_file(sio)
-        return sio.getvalue()
+    def test_long_description(self):
+        long_desc = textwrap.dedent("""\
+        example::
+              We start here
+            and continue here
+          and end here.""")
+        attrs = {"name": "package",
+                 "version": "1.0",
+                 "long_description": long_desc}
+
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        meta = meta.replace('\n' + 8 * ' ', '\n')
+        self.assertIn(long_desc, meta)
 
     def test_custom_pydistutils(self):
         # fixes #2166
@@ -272,14 +282,14 @@
             if sys.platform in ('linux', 'darwin'):
                 os.environ['HOME'] = temp_dir
                 files = dist.find_config_files()
-                self.assertTrue(user_filename in files)
+                self.assertIn(user_filename, files)
 
             # win32-style
             if sys.platform == 'win32':
                 # home drive should be found
                 os.environ['HOME'] = temp_dir
                 files = dist.find_config_files()
-                self.assertTrue(user_filename in files,
+                self.assertIn(user_filename, files,
                              '%r not found in %r' % (user_filename, files))
         finally:
             os.remove(user_filename)
@@ -301,22 +311,8 @@
 
         output = [line for line in s.getvalue().split('\n')
                   if line.strip() != '']
-        self.assertTrue(len(output) > 0)
+        self.assertTrue(output)
 
-    def test_long_description(self):
-        long_desc = textwrap.dedent("""\
-        example::
-              We start here
-            and continue here
-          and end here.""")
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "long_description": long_desc}
-
-        dist = Distribution(attrs)
-        meta = self.format_metadata(dist)
-        meta = meta.replace('\n' + 8 * ' ', '\n')
-        self.assertTrue(long_desc in meta)
 
 def test_suite():
     suite = unittest.TestSuite()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:42:15 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:42:15 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_determinati?=
 =?utf8?q?on_of_Metadata_version_=28=238933=29=2E__Patch_by_Filip_Gruszczy?=
 =?utf8?b?xYRza2ku?=
Message-ID: <E1R38eF-0005yL-Qy@dinsdale.python.org>

http://hg.python.org/cpython/rev/e5c1de856828
changeset:   72347:e5c1de856828
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 10 01:51:40 2011 +0200
summary:
  Fix determination of Metadata version (#8933).  Patch by Filip Gruszczy?ski.

files:
  Lib/distutils/dist.py            |   3 ++-
  Lib/distutils/tests/test_dist.py |  14 ++++++++++++++
  Misc/NEWS                        |   4 ++++
  3 files changed, 20 insertions(+), 1 deletions(-)


diff --git a/Lib/distutils/dist.py b/Lib/distutils/dist.py
--- a/Lib/distutils/dist.py
+++ b/Lib/distutils/dist.py
@@ -1018,7 +1018,8 @@
         """Write the PKG-INFO format data to a file object.
         """
         version = '1.0'
-        if self.provides or self.requires or self.obsoletes:
+        if (self.provides or self.requires or self.obsoletes or
+            self.classifiers or self.download_url):
             version = '1.1'
 
         file.write('Metadata-Version: %s\n' % version)
diff --git a/Lib/distutils/tests/test_dist.py b/Lib/distutils/tests/test_dist.py
--- a/Lib/distutils/tests/test_dist.py
+++ b/Lib/distutils/tests/test_dist.py
@@ -244,6 +244,20 @@
                            "version": "1.0",
                            "obsoletes": ["my.pkg (splat)"]})
 
+    def test_classifier(self):
+        attrs = {'name': 'Boa', 'version': '3.0',
+                 'classifiers': ['Programming Language :: Python :: 3']}
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        self.assertIn('Metadata-Version: 1.1', meta)
+
+    def test_download_url(self):
+        attrs = {'name': 'Boa', 'version': '3.0',
+                 'download_url': 'http://example.org/boa'}
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        self.assertIn('Metadata-Version: 1.1', meta)
+
     def test_long_description(self):
         long_desc = textwrap.dedent("""\
         example::
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -25,6 +25,10 @@
 Library
 -------
 
+- Issue #8933: distutils' PKG-INFO files will now correctly report
+  Metadata-Version: 1.1 instead of 1.0 if a Classifier or Download-URL field is
+  present.
+
 - Issue #9561: distutils now reads and writes egg-info files using UTF-8,
   instead of the locale encoding.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:42:16 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:42:16 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_fix_for_=238933_from_3=2E2?=
Message-ID: <E1R38eG-0005yP-Uv@dinsdale.python.org>

http://hg.python.org/cpython/rev/70298cdc48cd
changeset:   72348:70298cdc48cd
parent:      72318:272c8e9183db
parent:      72347:e5c1de856828
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 10 04:56:44 2011 +0200
summary:
  Merge fix for #8933 from 3.2

files:
  Lib/distutils/dist.py            |    3 +-
  Lib/distutils/tests/test_dist.py |  112 ++++++++++--------
  Misc/NEWS                        |    4 +
  3 files changed, 67 insertions(+), 52 deletions(-)


diff --git a/Lib/distutils/dist.py b/Lib/distutils/dist.py
--- a/Lib/distutils/dist.py
+++ b/Lib/distutils/dist.py
@@ -1018,7 +1018,8 @@
         """Write the PKG-INFO format data to a file object.
         """
         version = '1.0'
-        if self.provides or self.requires or self.obsoletes:
+        if (self.provides or self.requires or self.obsoletes or
+            self.classifiers or self.download_url):
             version = '1.1'
 
         file.write('Metadata-Version: %s\n' % version)
diff --git a/Lib/distutils/tests/test_dist.py b/Lib/distutils/tests/test_dist.py
--- a/Lib/distutils/tests/test_dist.py
+++ b/Lib/distutils/tests/test_dist.py
@@ -74,7 +74,7 @@
         self.assertEqual(d.get_command_packages(),
                          ["distutils.command", "foo.bar", "distutils.tests"])
         cmd = d.get_command_obj("test_dist")
-        self.assertTrue(isinstance(cmd, test_dist))
+        self.assertIsInstance(cmd, test_dist)
         self.assertEqual(cmd.sample_option, "sometext")
 
     def test_command_packages_configfile(self):
@@ -106,28 +106,23 @@
     def test_empty_options(self):
         # an empty options dictionary should not stay in the
         # list of attributes
-        klass = Distribution
 
         # catching warnings
         warns = []
+
         def _warn(msg):
             warns.append(msg)
 
-        old_warn = warnings.warn
+        self.addCleanup(setattr, warnings, 'warn', warnings.warn)
         warnings.warn = _warn
-        try:
-            dist = klass(attrs={'author': 'xxx',
-                                'name': 'xxx',
-                                'version': 'xxx',
-                                'url': 'xxxx',
-                                'options': {}})
-        finally:
-            warnings.warn = old_warn
+        dist = Distribution(attrs={'author': 'xxx', 'name': 'xxx',
+                                   'version': 'xxx', 'url': 'xxxx',
+                                   'options': {}})
 
         self.assertEqual(len(warns), 0)
+        self.assertNotIn('options', dir(dist))
 
     def test_finalize_options(self):
-
         attrs = {'keywords': 'one,two',
                  'platforms': 'one,two'}
 
@@ -150,7 +145,6 @@
         cmds = dist.get_command_packages()
         self.assertEqual(cmds, ['distutils.command', 'one', 'two'])
 
-
     def test_announce(self):
         # make sure the level is known
         dist = Distribution()
@@ -158,6 +152,7 @@
         kwargs = {'level': 'ok2'}
         self.assertRaises(ValueError, dist.announce, args, kwargs)
 
+
 class MetadataTestCase(support.TempdirManager, support.EnvironGuard,
                        unittest.TestCase):
 
@@ -170,15 +165,20 @@
         sys.argv[:] = self.argv[1]
         super(MetadataTestCase, self).tearDown()
 
+    def format_metadata(self, dist):
+        sio = io.StringIO()
+        dist.metadata.write_pkg_file(sio)
+        return sio.getvalue()
+
     def test_simple_metadata(self):
         attrs = {"name": "package",
                  "version": "1.0"}
         dist = Distribution(attrs)
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.0" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.0", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertNotIn("requires:", meta.lower())
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_provides(self):
         attrs = {"name": "package",
@@ -190,9 +190,9 @@
         self.assertEqual(dist.get_provides(),
                          ["package", "package.sub"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("requires:", meta.lower())
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_provides_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -210,11 +210,11 @@
         self.assertEqual(dist.get_requires(),
                          ["other", "another (==1.0)"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("Requires: other" in meta)
-        self.assertTrue("Requires: another (==1.0)" in meta)
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertIn("Requires: other", meta)
+        self.assertIn("Requires: another (==1.0)", meta)
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_requires_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -232,11 +232,11 @@
         self.assertEqual(dist.get_obsoletes(),
                          ["other", "another (<1.0)"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("Obsoletes: other" in meta)
-        self.assertTrue("Obsoletes: another (<1.0)" in meta)
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertNotIn("requires:", meta.lower())
+        self.assertIn("Obsoletes: other", meta)
+        self.assertIn("Obsoletes: another (<1.0)", meta)
 
     def test_obsoletes_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -244,10 +244,34 @@
                            "version": "1.0",
                            "obsoletes": ["my.pkg (splat)"]})
 
-    def format_metadata(self, dist):
-        sio = io.StringIO()
-        dist.metadata.write_pkg_file(sio)
-        return sio.getvalue()
+    def test_classifier(self):
+        attrs = {'name': 'Boa', 'version': '3.0',
+                 'classifiers': ['Programming Language :: Python :: 3']}
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        self.assertIn('Metadata-Version: 1.1', meta)
+
+    def test_download_url(self):
+        attrs = {'name': 'Boa', 'version': '3.0',
+                 'download_url': 'http://example.org/boa'}
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        self.assertIn('Metadata-Version: 1.1', meta)
+
+    def test_long_description(self):
+        long_desc = textwrap.dedent("""\
+        example::
+              We start here
+            and continue here
+          and end here.""")
+        attrs = {"name": "package",
+                 "version": "1.0",
+                 "long_description": long_desc}
+
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        meta = meta.replace('\n' + 8 * ' ', '\n')
+        self.assertIn(long_desc, meta)
 
     def test_custom_pydistutils(self):
         # fixes #2166
@@ -272,14 +296,14 @@
             if sys.platform in ('linux', 'darwin'):
                 os.environ['HOME'] = temp_dir
                 files = dist.find_config_files()
-                self.assertTrue(user_filename in files)
+                self.assertIn(user_filename, files)
 
             # win32-style
             if sys.platform == 'win32':
                 # home drive should be found
                 os.environ['HOME'] = temp_dir
                 files = dist.find_config_files()
-                self.assertTrue(user_filename in files,
+                self.assertIn(user_filename, files,
                              '%r not found in %r' % (user_filename, files))
         finally:
             os.remove(user_filename)
@@ -301,22 +325,8 @@
 
         output = [line for line in s.getvalue().split('\n')
                   if line.strip() != '']
-        self.assertTrue(len(output) > 0)
+        self.assertTrue(output)
 
-    def test_long_description(self):
-        long_desc = textwrap.dedent("""\
-        example::
-              We start here
-            and continue here
-          and end here.""")
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "long_description": long_desc}
-
-        dist = Distribution(attrs)
-        meta = self.format_metadata(dist)
-        meta = meta.replace('\n' + 8 * ' ', '\n')
-        self.assertTrue(long_desc in meta)
 
 def test_suite():
     suite = unittest.TestSuite()
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -274,6 +274,10 @@
 Library
 -------
 
+- Issue #8933: distutils' PKG-INFO files will now correctly report
+  Metadata-Version: 1.1 instead of 1.0 if a Classifier or Download-URL field is
+  present.
+
 - Issue #12567: Add curses.unget_wch() function. Push a character so the next
   get_wch() will return it.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:42:17 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:42:17 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Consolidate_tests_for_packa?=
	=?utf8?q?ging=2Emetadata=2E?=
Message-ID: <E1R38eH-0005yT-S4@dinsdale.python.org>

http://hg.python.org/cpython/rev/f0581d9ad978
changeset:   72349:f0581d9ad978
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 10 05:18:20 2011 +0200
summary:
  Consolidate tests for packaging.metadata.

New tests were added in test_metadata and old tests inherited from
distutils were still in test_dist, so I moved them into test_metadata
(except for one which was more at home in test_run) and merged
duplicates.

I also added some skips to lure contributors <wink>, optimized the
Metadata.update method a trifle, and added notes about a number of
issues.

A note: The tests in test_dist used to dump the Metadata objects to a
file in the METADATA format and look for strings in its contents; I
updated them to use the mapping API of Metadata instead.  For some
fields with special writing rules, I have added tests to ensure my
conversion did not lose anything.

files:
  Lib/packaging/metadata.py            |   14 +-
  Lib/packaging/tests/test_dist.py     |  252 +--------
  Lib/packaging/tests/test_metadata.py |  377 ++++++++++----
  Lib/packaging/tests/test_run.py      |   23 +-
  4 files changed, 324 insertions(+), 342 deletions(-)


diff --git a/Lib/packaging/metadata.py b/Lib/packaging/metadata.py
--- a/Lib/packaging/metadata.py
+++ b/Lib/packaging/metadata.py
@@ -354,11 +354,20 @@
         Keys that don't match a metadata field or that have an empty value are
         dropped.
         """
+        # XXX the code should just use self.set, which does tbe same checks and
+        # conversions already, but that would break packaging.pypi: it uses the
+        # update method, which does not call _set_best_version (which set
+        # does), and thus allows having a Metadata object (as long as you don't
+        # modify or write it) with extra fields from PyPI that are not fields
+        # defined in Metadata PEPs.  to solve it, the best_version system
+        # should be reworked so that it's called only for writing, or in a new
+        # strict mode, or with a new, more lax Metadata subclass in p7g.pypi
         def _set(key, value):
             if key in _ATTR2FIELD and value:
                 self.set(self._convert_name(key), value)
 
-        if other is None:
+        if not other:
+            # other is None or empty container
             pass
         elif hasattr(other, 'keys'):
             for k in other.keys():
@@ -368,7 +377,8 @@
                 _set(k, v)
 
         if kwargs:
-            self.update(kwargs)
+            for k, v in kwargs.items():
+                _set(k, v)
 
     def set(self, name, value):
         """Control then set a metadata field."""
diff --git a/Lib/packaging/tests/test_dist.py b/Lib/packaging/tests/test_dist.py
--- a/Lib/packaging/tests/test_dist.py
+++ b/Lib/packaging/tests/test_dist.py
@@ -1,16 +1,13 @@
 """Tests for packaging.dist."""
 import os
-import io
 import sys
 import logging
 import textwrap
-import sysconfig
 import packaging.dist
 
 from packaging.dist import Distribution
 from packaging.command import set_command
 from packaging.command.cmd import Command
-from packaging.metadata import Metadata
 from packaging.errors import PackagingModuleError, PackagingOptionError
 from packaging.tests import TESTFN, captured_stdout
 from packaging.tests import support, unittest
@@ -49,6 +46,7 @@
         sys.argv[:] = self.argv[1]
         super(DistributionTestCase, self).tearDown()
 
+    @unittest.skip('needs to be updated')
     def test_debug_mode(self):
         self.addCleanup(os.unlink, TESTFN)
         with open(TESTFN, "w") as f:
@@ -59,6 +57,8 @@
         sys.argv.append("build")
         __, stdout = captured_stdout(create_distribution, files)
         self.assertEqual(stdout, '')
+        # XXX debug mode does not exist anymore, test logging levels in this
+        # test instead
         packaging.dist.DEBUG = True
         try:
             __, stdout = captured_stdout(create_distribution, files)
@@ -66,34 +66,6 @@
         finally:
             packaging.dist.DEBUG = False
 
-    def test_write_pkg_file(self):
-        # Check Metadata handling of Unicode fields
-        tmp_dir = self.mkdtemp()
-        my_file = os.path.join(tmp_dir, 'f')
-        cls = Distribution
-
-        dist = cls(attrs={'author': 'Mister Caf?',
-                          'name': 'my.package',
-                          'maintainer': 'Caf? Junior',
-                          'summary': 'Caf? torr?fi?',
-                          'description': 'H?h?h?'})
-
-        # let's make sure the file can be written
-        # with Unicode fields. they are encoded with
-        # PKG_INFO_ENCODING
-        with open(my_file, 'w', encoding='utf-8') as fp:
-            dist.metadata.write_file(fp)
-
-        # regular ascii is of course always usable
-        dist = cls(attrs={'author': 'Mister Cafe',
-                          'name': 'my.package',
-                          'maintainer': 'Cafe Junior',
-                          'summary': 'Cafe torrefie',
-                          'description': 'Hehehe'})
-
-        with open(my_file, 'w') as fp:
-            dist.metadata.write_file(fp)
-
     def test_bad_attr(self):
         Distribution(attrs={'author': 'xxx',
                             'name': 'xxx',
@@ -101,28 +73,18 @@
                             'home-page': 'xxxx',
                             'badoptname': 'xxx'})
         logs = self.get_logs(logging.WARNING)
-        self.assertEqual(1, len(logs))
+        self.assertEqual(len(logs), 1)
         self.assertIn('unknown argument', logs[0])
 
-    def test_bad_version(self):
-        Distribution(attrs={'author': 'xxx',
-                            'name': 'xxx',
-                            'version': 'xxx',
-                            'home-page': 'xxxx'})
-        logs = self.get_logs(logging.WARNING)
-        self.assertEqual(1, len(logs))
-        self.assertIn('not a valid version', logs[0])
-
     def test_empty_options(self):
         # an empty options dictionary should not stay in the
         # list of attributes
-        Distribution(attrs={'author': 'xxx',
-                            'name': 'xxx',
-                            'version': '1.2',
-                            'home-page': 'xxxx',
-                            'options': {}})
+        dist = Distribution(attrs={'author': 'xxx', 'name': 'xxx',
+                                   'version': '1.2', 'home-page': 'xxxx',
+                                   'options': {}})
 
         self.assertEqual([], self.get_logs(logging.WARNING))
+        self.assertNotIn('options', dir(dist))
 
     def test_non_empty_options(self):
         # TODO: how to actually use options is not documented except
@@ -141,7 +103,6 @@
         self.assertIn('owner', dist.get_option_dict('sdist'))
 
     def test_finalize_options(self):
-
         attrs = {'keywords': 'one,two',
                  'platform': 'one,two'}
 
@@ -152,6 +113,24 @@
         self.assertEqual(dist.metadata['platform'], ['one', 'two'])
         self.assertEqual(dist.metadata['keywords'], ['one', 'two'])
 
+    def test_custom_pydistutils(self):
+        # Bug #2166:  make sure pydistutils.cfg is found
+        if os.name == 'posix':
+            user_filename = ".pydistutils.cfg"
+        else:
+            user_filename = "pydistutils.cfg"
+
+        temp_dir = self.mkdtemp()
+        user_filename = os.path.join(temp_dir, user_filename)
+        with open(user_filename, 'w') as f:
+            f.write('.')
+
+        dist = Distribution()
+
+        os.environ['HOME'] = temp_dir
+        files = dist.find_config_files()
+        self.assertIn(user_filename, files)
+
     def test_find_config_files_disable(self):
         # Bug #1180: Allow users to disable their own config file.
         temp_home = self.mkdtemp()
@@ -270,185 +249,8 @@
         self.assertRaises(PackagingOptionError, d.run_command, 'test_dist')
 
 
-class MetadataTestCase(support.TempdirManager,
-                       support.LoggingCatcher,
-                       support.EnvironRestorer,
-                       unittest.TestCase):
-
-    restore_environ = ['HOME']
-
-    def setUp(self):
-        super(MetadataTestCase, self).setUp()
-        self.argv = sys.argv, sys.argv[:]
-
-    def tearDown(self):
-        sys.argv = self.argv[0]
-        sys.argv[:] = self.argv[1]
-        super(MetadataTestCase, self).tearDown()
-
-    def test_simple_metadata(self):
-        attrs = {"name": "package",
-                 "version": "1.0"}
-        dist = Distribution(attrs)
-        meta = self.format_metadata(dist)
-        self.assertIn("Metadata-Version: 1.0", meta)
-        self.assertNotIn("provides:", meta.lower())
-        self.assertNotIn("requires:", meta.lower())
-        self.assertNotIn("obsoletes:", meta.lower())
-
-    def test_provides_dist(self):
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "provides_dist": ["package", "package.sub"]}
-        dist = Distribution(attrs)
-        self.assertEqual(dist.metadata['Provides-Dist'],
-                         ["package", "package.sub"])
-        meta = self.format_metadata(dist)
-        self.assertIn("Metadata-Version: 1.2", meta)
-        self.assertNotIn("requires:", meta.lower())
-        self.assertNotIn("obsoletes:", meta.lower())
-
-    def _test_provides_illegal(self):
-        # XXX to do: check the versions
-        self.assertRaises(ValueError, Distribution,
-                          {"name": "package",
-                           "version": "1.0",
-                           "provides_dist": ["my.pkg (splat)"]})
-
-    def test_requires_dist(self):
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "requires_dist": ["other", "another (==1.0)"]}
-        dist = Distribution(attrs)
-        self.assertEqual(dist.metadata['Requires-Dist'],
-                         ["other", "another (==1.0)"])
-        meta = self.format_metadata(dist)
-        self.assertIn("Metadata-Version: 1.2", meta)
-        self.assertNotIn("provides:", meta.lower())
-        self.assertIn("Requires-Dist: other", meta)
-        self.assertIn("Requires-Dist: another (==1.0)", meta)
-        self.assertNotIn("obsoletes:", meta.lower())
-
-    def _test_requires_illegal(self):
-        # XXX
-        self.assertRaises(ValueError, Distribution,
-                          {"name": "package",
-                           "version": "1.0",
-                           "requires": ["my.pkg (splat)"]})
-
-    def test_obsoletes_dist(self):
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "obsoletes_dist": ["other", "another (<1.0)"]}
-        dist = Distribution(attrs)
-        self.assertEqual(dist.metadata['Obsoletes-Dist'],
-                         ["other", "another (<1.0)"])
-        meta = self.format_metadata(dist)
-        self.assertIn("Metadata-Version: 1.2", meta)
-        self.assertNotIn("provides:", meta.lower())
-        self.assertNotIn("requires:", meta.lower())
-        self.assertIn("Obsoletes-Dist: other", meta)
-        self.assertIn("Obsoletes-Dist: another (<1.0)", meta)
-
-    def _test_obsoletes_illegal(self):
-        # XXX
-        self.assertRaises(ValueError, Distribution,
-                          {"name": "package",
-                           "version": "1.0",
-                           "obsoletes": ["my.pkg (splat)"]})
-
-    def format_metadata(self, dist):
-        sio = io.StringIO()
-        dist.metadata.write_file(sio)
-        return sio.getvalue()
-
-    def test_custom_pydistutils(self):
-        # fixes #2166
-        # make sure pydistutils.cfg is found
-        if os.name == 'posix':
-            user_filename = ".pydistutils.cfg"
-        else:
-            user_filename = "pydistutils.cfg"
-
-        temp_dir = self.mkdtemp()
-        user_filename = os.path.join(temp_dir, user_filename)
-        with open(user_filename, 'w') as f:
-            f.write('.')
-
-        dist = Distribution()
-
-        # linux-style
-        if sys.platform in ('linux', 'darwin'):
-            os.environ['HOME'] = temp_dir
-            files = dist.find_config_files()
-            self.assertIn(user_filename, files)
-
-        # win32-style
-        if sys.platform == 'win32':
-            # home drive should be found
-            os.environ['HOME'] = temp_dir
-            files = dist.find_config_files()
-            self.assertIn(user_filename, files)
-
-    def test_show_help(self):
-        # smoke test, just makes sure some help is displayed
-        dist = Distribution()
-        sys.argv = []
-        dist.help = True
-        dist.script_name = os.path.join(sysconfig.get_path('scripts'),
-                                        'pysetup')
-        __, stdout = captured_stdout(dist.parse_command_line)
-        output = [line for line in stdout.split('\n')
-                  if line.strip() != '']
-        self.assertGreater(len(output), 0)
-
-    def test_description(self):
-        desc = textwrap.dedent("""\
-        example::
-              We start here
-            and continue here
-          and end here.""")
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "description": desc}
-
-        dist = packaging.dist.Distribution(attrs)
-        meta = self.format_metadata(dist)
-        meta = meta.replace('\n' + 7 * ' ' + '|', '\n')
-        self.assertIn(desc, meta)
-
-    def test_read_metadata(self):
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "description": "desc",
-                 "summary": "xxx",
-                 "download_url": "http://example.com",
-                 "keywords": ['one', 'two'],
-                 "requires_dist": ['foo']}
-
-        dist = Distribution(attrs)
-        PKG_INFO = io.StringIO()
-        dist.metadata.write_file(PKG_INFO)
-        PKG_INFO.seek(0)
-
-        metadata = Metadata()
-        metadata.read_file(PKG_INFO)
-
-        self.assertEqual(metadata['name'], "package")
-        self.assertEqual(metadata['version'], "1.0")
-        self.assertEqual(metadata['summary'], "xxx")
-        self.assertEqual(metadata['download_url'], 'http://example.com')
-        self.assertEqual(metadata['keywords'], ['one', 'two'])
-        self.assertEqual(metadata['platform'], [])
-        self.assertEqual(metadata['obsoletes'], [])
-        self.assertEqual(metadata['requires-dist'], ['foo'])
-
-
 def test_suite():
-    suite = unittest.TestSuite()
-    suite.addTest(unittest.makeSuite(DistributionTestCase))
-    suite.addTest(unittest.makeSuite(MetadataTestCase))
-    return suite
+    return unittest.makeSuite(DistributionTestCase)
 
 if __name__ == "__main__":
     unittest.main(defaultTest="test_suite")
diff --git a/Lib/packaging/tests/test_metadata.py b/Lib/packaging/tests/test_metadata.py
--- a/Lib/packaging/tests/test_metadata.py
+++ b/Lib/packaging/tests/test_metadata.py
@@ -2,6 +2,7 @@
 import os
 import sys
 import logging
+from textwrap import dedent
 from io import StringIO
 
 from packaging.errors import (MetadataConflictError, MetadataMissingError,
@@ -9,12 +10,29 @@
 from packaging.metadata import Metadata, PKG_INFO_PREFERRED_VERSION
 
 from packaging.tests import unittest
-from packaging.tests.support import LoggingCatcher
+from packaging.tests.support import (LoggingCatcher, TempdirManager,
+                                     EnvironRestorer)
 
 
 class MetadataTestCase(LoggingCatcher,
+                       TempdirManager,
+                       EnvironRestorer,
                        unittest.TestCase):
 
+    maxDiff = None
+    restore_environ = ['HOME']
+
+    def setUp(self):
+        super(MetadataTestCase, self).setUp()
+        self.argv = sys.argv, sys.argv[:]
+
+    def tearDown(self):
+        sys.argv = self.argv[0]
+        sys.argv[:] = self.argv[1]
+        super(MetadataTestCase, self).tearDown()
+
+    ####  Test various methods of the Metadata class
+
     def test_instantiation(self):
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
         with open(PKG_INFO, 'r', encoding='utf-8') as f:
@@ -43,17 +61,6 @@
         self.assertRaises(TypeError, Metadata,
                           PKG_INFO, mapping=m, fileobj=fp)
 
-    def test_metadata_read_write(self):
-        PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
-        metadata = Metadata(PKG_INFO)
-        out = StringIO()
-        metadata.write_file(out)
-        out.seek(0)
-        res = Metadata()
-        res.read_file(out)
-        for k in metadata:
-            self.assertEqual(metadata[k], res[k])
-
     def test_metadata_markers(self):
         # see if we can be platform-aware
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
@@ -70,31 +77,10 @@
 
         # test with context
         context = {'sys.platform': 'okook'}
-        metadata = Metadata(platform_dependent=True,
-                                        execution_context=context)
+        metadata = Metadata(platform_dependent=True, execution_context=context)
         metadata.read_file(StringIO(content))
         self.assertEqual(metadata['Requires-Dist'], ['foo'])
 
-    def test_description(self):
-        PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
-        with open(PKG_INFO, 'r', encoding='utf-8') as f:
-            content = f.read() % sys.platform
-        metadata = Metadata()
-        metadata.read_file(StringIO(content))
-
-        # see if we can read the description now
-        DESC = os.path.join(os.path.dirname(__file__), 'LONG_DESC.txt')
-        with open(DESC) as f:
-            wanted = f.read()
-        self.assertEqual(wanted, metadata['Description'])
-
-        # save the file somewhere and make sure we can read it back
-        out = StringIO()
-        metadata.write_file(out)
-        out.seek(0)
-        metadata.read_file(out)
-        self.assertEqual(wanted, metadata['Description'])
-
     def test_mapping_api(self):
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
         with open(PKG_INFO, 'r', encoding='utf-8') as f:
@@ -109,73 +95,86 @@
         metadata.update([('version', '0.7')])
         self.assertEqual(metadata['Version'], '0.7')
 
+        # make sure update method checks values like the set method does
+        metadata.update({'version': '1--2'})
+        self.assertEqual(len(self.get_logs()), 1)
+
+        # XXX caveat: the keys method and friends are not 3.x-style views
+        # should be changed or documented
         self.assertEqual(list(metadata), list(metadata.keys()))
 
-    def test_versions(self):
-        metadata = Metadata()
-        metadata['Obsoletes'] = 'ok'
-        self.assertEqual(metadata['Metadata-Version'], '1.1')
+    def test_read_metadata(self):
+        fields = {'name': 'project',
+                  'version': '1.0',
+                  'description': 'desc',
+                  'summary': 'xxx',
+                  'download_url': 'http://example.com',
+                  'keywords': ['one', 'two'],
+                  'requires_dist': ['foo']}
 
-        del metadata['Obsoletes']
-        metadata['Obsoletes-Dist'] = 'ok'
-        self.assertEqual(metadata['Metadata-Version'], '1.2')
+        metadata = Metadata(mapping=fields)
+        PKG_INFO = StringIO()
+        metadata.write_file(PKG_INFO)
+        PKG_INFO.seek(0)
 
-        self.assertRaises(MetadataConflictError, metadata.set,
-                          'Obsoletes', 'ok')
+        metadata = Metadata(fileobj=PKG_INFO)
 
-        del metadata['Obsoletes']
-        del metadata['Obsoletes-Dist']
-        metadata['Version'] = '1'
-        self.assertEqual(metadata['Metadata-Version'], '1.0')
+        self.assertEqual(metadata['name'], 'project')
+        self.assertEqual(metadata['version'], '1.0')
+        self.assertEqual(metadata['summary'], 'xxx')
+        self.assertEqual(metadata['download_url'], 'http://example.com')
+        self.assertEqual(metadata['keywords'], ['one', 'two'])
+        self.assertEqual(metadata['platform'], [])
+        self.assertEqual(metadata['obsoletes'], [])
+        self.assertEqual(metadata['requires-dist'], ['foo'])
 
-        PKG_INFO = os.path.join(os.path.dirname(__file__),
-                                'SETUPTOOLS-PKG-INFO')
-        with open(PKG_INFO, 'r', encoding='utf-8') as f:
-            content = f.read()
-        metadata.read_file(StringIO(content))
-        self.assertEqual(metadata['Metadata-Version'], '1.0')
+    def test_write_metadata(self):
+        # check support of non-ASCII values
+        tmp_dir = self.mkdtemp()
+        my_file = os.path.join(tmp_dir, 'f')
 
-        PKG_INFO = os.path.join(os.path.dirname(__file__),
-                                'SETUPTOOLS-PKG-INFO2')
-        with open(PKG_INFO, 'r', encoding='utf-8') as f:
-            content = f.read()
-        metadata.read_file(StringIO(content))
-        self.assertEqual(metadata['Metadata-Version'], '1.1')
+        metadata = Metadata(mapping={'author': 'Mister Caf?',
+                                     'name': 'my.project',
+                                     'author': 'Caf? Junior',
+                                     'summary': 'Caf? torr?fi?',
+                                     'description': 'H?h?h?',
+                                     'keywords': ['caf?', 'coffee']})
+        metadata.write(my_file)
 
-        # Update the _fields dict directly to prevent 'Metadata-Version'
-        # from being updated by the _set_best_version() method.
-        metadata._fields['Metadata-Version'] = '1.618'
-        self.assertRaises(MetadataUnrecognizedVersionError, metadata.keys)
+        # the file should use UTF-8
+        metadata2 = Metadata()
+        with open(my_file, encoding='utf-8') as fp:
+            metadata2.read_file(fp)
 
-    def test_warnings(self):
-        metadata = Metadata()
+        # XXX when keywords are not defined, metadata will have
+        # 'Keywords': [] but metadata2 will have 'Keywords': ['']
+        # because of a value.split(',') in Metadata.get
+        self.assertEqual(metadata.items(), metadata2.items())
 
-        # these should raise a warning
-        values = (('Requires-Dist', 'Funky (Groovie)'),
-                  ('Requires-Python', '1-4'))
+        # ASCII also works, it's a subset of UTF-8
+        metadata = Metadata(mapping={'author': 'Mister Cafe',
+                                     'name': 'my.project',
+                                     'author': 'Cafe Junior',
+                                     'summary': 'Cafe torrefie',
+                                     'description': 'Hehehe'})
+        metadata.write(my_file)
 
-        for name, value in values:
-            metadata.set(name, value)
+        metadata2 = Metadata()
+        with open(my_file, encoding='utf-8') as fp:
+            metadata2.read_file(fp)
 
-        # we should have a certain amount of warnings
-        self.assertEqual(len(self.get_logs()), 2)
+    def test_metadata_read_write(self):
+        PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
+        metadata = Metadata(PKG_INFO)
+        out = StringIO()
+        metadata.write_file(out)
 
-    def test_multiple_predicates(self):
-        metadata = Metadata()
+        out.seek(0)
+        res = Metadata()
+        res.read_file(out)
+        self.assertEqual(metadata.values(), res.values())
 
-        # see for "3" instead of "3.0"  ???
-        # its seems like the MINOR VERSION can be omitted
-        metadata['Requires-Python'] = '>=2.6, <3.0'
-        metadata['Requires-Dist'] = ['Foo (>=2.6, <3.0)']
-
-        self.assertEqual([], self.get_logs(logging.WARNING))
-
-    def test_project_url(self):
-        metadata = Metadata()
-        metadata['Project-URL'] = [('one', 'http://ok')]
-        self.assertEqual(metadata['Project-URL'],
-                          [('one', 'http://ok')])
-        self.assertEqual(metadata['Metadata-Version'], '1.2')
+    ####  Test checks
 
     def test_check_version(self):
         metadata = Metadata()
@@ -238,38 +237,202 @@
         metadata['Requires-dist'] = ['Foo (a)']
         metadata['Obsoletes-dist'] = ['Foo (a)']
         metadata['Provides-dist'] = ['Foo (a)']
-        if metadata.docutils_support:
-            missing, warnings = metadata.check()
-            self.assertEqual(len(warnings), 4)
-            metadata.docutils_support = False
         missing, warnings = metadata.check()
         self.assertEqual(len(warnings), 4)
 
-    def test_best_choice(self):
-        metadata = Metadata()
-        metadata['Version'] = '1.0'
+    ####  Test fields and metadata versions
+
+    def test_metadata_versions(self):
+        metadata = Metadata(mapping={'name': 'project', 'version': '1.0'})
         self.assertEqual(metadata['Metadata-Version'],
                          PKG_INFO_PREFERRED_VERSION)
+        self.assertNotIn('Provides', metadata)
+        self.assertNotIn('Requires', metadata)
+        self.assertNotIn('Obsoletes', metadata)
+
         metadata['Classifier'] = ['ok']
         self.assertEqual(metadata['Metadata-Version'], '1.2')
 
-    def test_project_urls(self):
-        # project-url is a bit specific, make sure we write it
-        # properly in PKG-INFO
         metadata = Metadata()
-        metadata['Version'] = '1.0'
-        metadata['Project-Url'] = [('one', 'http://ok')]
+        metadata['Obsoletes'] = 'ok'
+        self.assertEqual(metadata['Metadata-Version'], '1.1')
+
+        del metadata['Obsoletes']
+        metadata['Obsoletes-Dist'] = 'ok'
+        self.assertEqual(metadata['Metadata-Version'], '1.2')
+
+        self.assertRaises(MetadataConflictError, metadata.set,
+                          'Obsoletes', 'ok')
+
+        del metadata['Obsoletes']
+        del metadata['Obsoletes-Dist']
+        metadata['Version'] = '1'
+        self.assertEqual(metadata['Metadata-Version'], '1.0')
+
+        PKG_INFO = os.path.join(os.path.dirname(__file__),
+                                'SETUPTOOLS-PKG-INFO')
+        metadata = Metadata(PKG_INFO)
+        self.assertEqual(metadata['Metadata-Version'], '1.0')
+
+        PKG_INFO = os.path.join(os.path.dirname(__file__),
+                                'SETUPTOOLS-PKG-INFO2')
+        metadata = Metadata(PKG_INFO)
+        self.assertEqual(metadata['Metadata-Version'], '1.1')
+
+        # Update the _fields dict directly to prevent 'Metadata-Version'
+        # from being updated by the _set_best_version() method.
+        metadata._fields['Metadata-Version'] = '1.618'
+        self.assertRaises(MetadataUnrecognizedVersionError, metadata.keys)
+
+    def test_version(self):
+        Metadata(mapping={'author': 'xxx',
+                          'name': 'xxx',
+                          'version': 'xxx',
+                          'home-page': 'xxxx'})
+        logs = self.get_logs(logging.WARNING)
+        self.assertEqual(1, len(logs))
+        self.assertIn('not a valid version', logs[0])
+
+    def test_description(self):
+        PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
+        with open(PKG_INFO, 'r', encoding='utf-8') as f:
+            content = f.read() % sys.platform
+        metadata = Metadata()
+        metadata.read_file(StringIO(content))
+
+        # see if we can read the description now
+        DESC = os.path.join(os.path.dirname(__file__), 'LONG_DESC.txt')
+        with open(DESC) as f:
+            wanted = f.read()
+        self.assertEqual(wanted, metadata['Description'])
+
+        # save the file somewhere and make sure we can read it back
+        out = StringIO()
+        metadata.write_file(out)
+        out.seek(0)
+
+        out.seek(0)
+        metadata = Metadata()
+        metadata.read_file(out)
+        self.assertEqual(wanted, metadata['Description'])
+
+    def test_description_folding(self):
+        # make sure the indentation is preserved
+        out = StringIO()
+        desc = dedent("""\
+        example::
+              We start here
+            and continue here
+          and end here.
+        """)
+
+        metadata = Metadata()
+        metadata['description'] = desc
+        metadata.write_file(out)
+
+        folded_desc = desc.replace('\n', '\n' + (7 * ' ') + '|')
+        self.assertIn(folded_desc, out.getvalue())
+
+    def test_project_url(self):
+        metadata = Metadata()
+        metadata['Project-URL'] = [('one', 'http://ok')]
+        self.assertEqual(metadata['Project-URL'], [('one', 'http://ok')])
+        self.assertEqual(metadata['Metadata-Version'], '1.2')
+
+        # make sure this particular field is handled properly when written
+        fp = StringIO()
+        metadata.write_file(fp)
+        self.assertIn('Project-URL: one,http://ok', fp.getvalue().split('\n'))
+
+        fp.seek(0)
+        metadata = Metadata()
+        metadata.read_file(fp)
         self.assertEqual(metadata['Project-Url'], [('one', 'http://ok')])
-        file_ = StringIO()
-        metadata.write_file(file_)
-        file_.seek(0)
-        res = file_.read().split('\n')
-        self.assertIn('Project-URL: one,http://ok', res)
 
-        file_.seek(0)
+    # TODO copy tests for v1.1 requires, obsoletes and provides from distutils
+    # (they're useless but we support them so we should test them anyway)
+
+    def test_provides_dist(self):
+        fields = {'name': 'project',
+                  'version': '1.0',
+                  'provides_dist': ['project', 'my.project']}
+        metadata = Metadata(mapping=fields)
+        self.assertEqual(metadata['Provides-Dist'],
+                         ['project', 'my.project'])
+        self.assertEqual(metadata['Metadata-Version'], '1.2', metadata)
+        self.assertNotIn('Requires', metadata)
+        self.assertNotIn('Obsoletes', metadata)
+
+    @unittest.skip('needs to be implemented')
+    def test_provides_illegal(self):
+        # TODO check the versions (like distutils does for old provides field)
+        self.assertRaises(ValueError, Metadata,
+                          mapping={'name': 'project',
+                                   'version': '1.0',
+                                   'provides_dist': ['my.pkg (splat)']})
+
+    def test_requires_dist(self):
+        fields = {'name': 'project',
+                  'version': '1.0',
+                  'requires_dist': ['other', 'another (==1.0)']}
+        metadata = Metadata(mapping=fields)
+        self.assertEqual(metadata['Requires-Dist'],
+                         ['other', 'another (==1.0)'])
+        self.assertEqual(metadata['Metadata-Version'], '1.2')
+        self.assertNotIn('Provides', metadata)
+        self.assertEqual(metadata['Requires-Dist'],
+                         ['other', 'another (==1.0)'])
+        self.assertNotIn('Obsoletes', metadata)
+
+        # make sure write_file uses one RFC 822 header per item
+        fp = StringIO()
+        metadata.write_file(fp)
+        lines = fp.getvalue().split('\n')
+        self.assertIn('Requires-Dist: other', lines)
+        self.assertIn('Requires-Dist: another (==1.0)', lines)
+
+        # test warnings for invalid version predicates
+        # XXX this would cause no warnings if we used update (or the mapping
+        # argument of the constructor), see comment in Metadata.update
         metadata = Metadata()
-        metadata.read_file(file_)
-        self.assertEqual(metadata['Project-Url'], [('one', 'http://ok')])
+        metadata['Requires-Dist'] = 'Funky (Groovie)'
+        metadata['Requires-Python'] = '1-4'
+        self.assertEqual(len(self.get_logs()), 2)
+
+        # test multiple version predicates
+        metadata = Metadata()
+
+        # XXX check PEP and see if 3 == 3.0
+        metadata['Requires-Python'] = '>=2.6, <3.0'
+        metadata['Requires-Dist'] = ['Foo (>=2.6, <3.0)']
+        self.assertEqual([], self.get_logs(logging.WARNING))
+
+    @unittest.skip('needs to be implemented')
+    def test_requires_illegal(self):
+        self.assertRaises(ValueError, Metadata,
+                          mapping={'name': 'project',
+                                   'version': '1.0',
+                                   'requires': ['my.pkg (splat)']})
+
+    def test_obsoletes_dist(self):
+        fields = {'name': 'project',
+                  'version': '1.0',
+                  'obsoletes_dist': ['other', 'another (<1.0)']}
+        metadata = Metadata(mapping=fields)
+        self.assertEqual(metadata['Obsoletes-Dist'],
+                         ['other', 'another (<1.0)'])
+        self.assertEqual(metadata['Metadata-Version'], '1.2')
+        self.assertNotIn('Provides', metadata)
+        self.assertNotIn('Requires', metadata)
+        self.assertEqual(metadata['Obsoletes-Dist'],
+                         ['other', 'another (<1.0)'])
+
+    @unittest.skip('needs to be implemented')
+    def test_obsoletes_illegal(self):
+        self.assertRaises(ValueError, Metadata,
+                          mapping={'name': 'project',
+                                   'version': '1.0',
+                                   'obsoletes': ['my.pkg (splat)']})
 
 
 def test_suite():
diff --git a/Lib/packaging/tests/test_run.py b/Lib/packaging/tests/test_run.py
--- a/Lib/packaging/tests/test_run.py
+++ b/Lib/packaging/tests/test_run.py
@@ -3,16 +3,16 @@
 import os
 import sys
 import shutil
-from tempfile import mkstemp
 from io import StringIO
 
 from packaging import install
 from packaging.tests import unittest, support, TESTFN
 from packaging.run import main
 
+from test.script_helper import assert_python_ok
+
 # setup script that uses __file__
 setup_using___file__ = """\
-
 __file__
 
 from packaging.run import setup
@@ -20,7 +20,6 @@
 """
 
 setup_prints_cwd = """\
-
 import os
 print os.getcwd()
 
@@ -29,11 +28,12 @@
 """
 
 
-class CoreTestCase(support.TempdirManager, support.LoggingCatcher,
-                   unittest.TestCase):
+class RunTestCase(support.TempdirManager,
+                  support.LoggingCatcher,
+                  unittest.TestCase):
 
     def setUp(self):
-        super(CoreTestCase, self).setUp()
+        super(RunTestCase, self).setUp()
         self.old_stdout = sys.stdout
         self.cleanup_testfn()
         self.old_argv = sys.argv, sys.argv[:]
@@ -43,7 +43,7 @@
         self.cleanup_testfn()
         sys.argv = self.old_argv[0]
         sys.argv[:] = self.old_argv[1]
-        super(CoreTestCase, self).tearDown()
+        super(RunTestCase, self).tearDown()
 
     def cleanup_testfn(self):
         path = TESTFN
@@ -77,9 +77,16 @@
             os.chmod(install_path, old_mod)
             install.get_path = old_get_path
 
+    def test_show_help(self):
+        # smoke test, just makes sure some help is displayed
+        status, out, err = assert_python_ok('-m', 'packaging.run', '--help')
+        self.assertEqual(status, 0)
+        self.assertGreater(out, b'')
+        self.assertEqual(err, b'')
+
 
 def test_suite():
-    return unittest.makeSuite(CoreTestCase)
+    return unittest.makeSuite(RunTestCase)
 
 if __name__ == "__main__":
     unittest.main(defaultTest="test_suite")

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:42:18 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:42:18 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_determination_of_Metada?=
 =?utf8?q?ta_version_in_packaging_=28=238933=29=2E?=
Message-ID: <E1R38eI-0005yX-Jo@dinsdale.python.org>

http://hg.python.org/cpython/rev/ca21a47b1ec2
changeset:   72350:ca21a47b1ec2
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 10 05:22:48 2011 +0200
summary:
  Fix determination of Metadata version in packaging (#8933).

Original patch by Filip Gruszczy?ski.

files:
  Lib/packaging/metadata.py            |   3 ++-
  Lib/packaging/tests/test_metadata.py |  15 ++++++++++++++-
  Misc/NEWS                            |   6 +++---
  3 files changed, 19 insertions(+), 5 deletions(-)


diff --git a/Lib/packaging/metadata.py b/Lib/packaging/metadata.py
--- a/Lib/packaging/metadata.py
+++ b/Lib/packaging/metadata.py
@@ -61,7 +61,8 @@
                'License', 'Classifier', 'Download-URL', 'Obsoletes',
                'Provides', 'Requires')
 
-_314_MARKERS = ('Obsoletes', 'Provides', 'Requires')
+_314_MARKERS = ('Obsoletes', 'Provides', 'Requires', 'Classifier',
+                'Download-URL')
 
 _345_FIELDS = ('Metadata-Version', 'Name', 'Version', 'Platform',
                'Supported-Platform', 'Summary', 'Description',
diff --git a/Lib/packaging/tests/test_metadata.py b/Lib/packaging/tests/test_metadata.py
--- a/Lib/packaging/tests/test_metadata.py
+++ b/Lib/packaging/tests/test_metadata.py
@@ -251,7 +251,11 @@
         self.assertNotIn('Obsoletes', metadata)
 
         metadata['Classifier'] = ['ok']
-        self.assertEqual(metadata['Metadata-Version'], '1.2')
+        self.assertEqual(metadata['Metadata-Version'], '1.1')
+
+        metadata = Metadata()
+        metadata['Download-URL'] = 'ok'
+        self.assertEqual(metadata['Metadata-Version'], '1.1')
 
         metadata = Metadata()
         metadata['Obsoletes'] = 'ok'
@@ -269,6 +273,15 @@
         metadata['Version'] = '1'
         self.assertEqual(metadata['Metadata-Version'], '1.0')
 
+        # make sure the _best_version function works okay with
+        # non-conflicting fields from 1.1 and 1.2 (i.e. we want only the
+        # requires/requires-dist and co. pairs to cause a conflict, not all
+        # fields in _314_MARKERS)
+        metadata = Metadata()
+        metadata['Requires-Python'] = '3'
+        metadata['Classifier'] = ['Programming language :: Python :: 3']
+        self.assertEqual(metadata['Metadata-Version'], '1.2')
+
         PKG_INFO = os.path.join(os.path.dirname(__file__),
                                 'SETUPTOOLS-PKG-INFO')
         metadata = Metadata(PKG_INFO)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -274,9 +274,9 @@
 Library
 -------
 
-- Issue #8933: distutils' PKG-INFO files will now correctly report
-  Metadata-Version: 1.1 instead of 1.0 if a Classifier or Download-URL field is
-  present.
+- Issue #8933: distutils' PKG-INFO files and packaging's METADATA files will
+  now correctly report Metadata-Version: 1.1 instead of 1.0 if a Classifier or
+  Download-URL field is present.
 
 - Issue #12567: Add curses.unget_wch() function. Push a character so the next
   get_wch() will return it.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:42:19 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:42:19 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_usage_of_dry-run_in_pac?=
 =?utf8?q?kaging_bdist=5Fwininst_and_install=5Fdistinfo=2E?=
Message-ID: <E1R38eJ-0005z3-A0@dinsdale.python.org>

http://hg.python.org/cpython/rev/545eb37dc3f7
changeset:   72351:545eb37dc3f7
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 10 18:10:23 2011 +0200
summary:
  Fix usage of dry-run in packaging bdist_wininst and install_distinfo.

In dry-run mode, packaging commands should log the same info as in real
operation and should collect the same files in self.outputs, so that
users can run a command in verbose and dry-run mode to see exactly what
operations will be done in the real run.

files:
  Lib/packaging/command/bdist_wininst.py    |   5 +-
  Lib/packaging/command/install_distinfo.py |  80 +++++-----
  2 files changed, 42 insertions(+), 43 deletions(-)


diff --git a/Lib/packaging/command/bdist_wininst.py b/Lib/packaging/command/bdist_wininst.py
--- a/Lib/packaging/command/bdist_wininst.py
+++ b/Lib/packaging/command/bdist_wininst.py
@@ -186,9 +186,8 @@
         os.remove(arcname)
 
         if not self.keep_temp:
-            if self.dry_run:
-                logger.info('removing %s', self.bdist_dir)
-            else:
+            logger.info('removing %s', self.bdist_dir)
+            if not self.dry_run:
                 rmtree(self.bdist_dir)
 
     def get_inidata(self):
diff --git a/Lib/packaging/command/install_distinfo.py b/Lib/packaging/command/install_distinfo.py
--- a/Lib/packaging/command/install_distinfo.py
+++ b/Lib/packaging/command/install_distinfo.py
@@ -28,7 +28,7 @@
         ('no-record', None,
          "do not generate a RECORD file"),
         ('no-resources', None,
-         "do not generate a RESSOURCES list installed file")
+         "do not generate a RESSOURCES list installed file"),
     ]
 
     boolean_options = ['requested', 'no-record', 'no-resources']
@@ -70,56 +70,56 @@
         self.distinfo_dir = os.path.join(self.distinfo_dir, basename)
 
     def run(self):
-        # FIXME dry-run should be used at a finer level, so that people get
-        # useful logging output and can have an idea of what the command would
-        # have done
+        target = self.distinfo_dir
+
+        if os.path.isdir(target) and not os.path.islink(target):
+            if not self.dry_run:
+                rmtree(target)
+        elif os.path.exists(target):
+            self.execute(os.unlink, (self.distinfo_dir,),
+                         "removing " + target)
+
+        self.execute(os.makedirs, (target,), "creating " + target)
+
+        metadata_path = os.path.join(self.distinfo_dir, 'METADATA')
+        self.execute(self.distribution.metadata.write, (metadata_path,),
+                     "creating " + metadata_path)
+        self.outfiles.append(metadata_path)
+
+        installer_path = os.path.join(self.distinfo_dir, 'INSTALLER')
+        logger.info('creating %s', installer_path)
         if not self.dry_run:
-            target = self.distinfo_dir
-
-            if os.path.isdir(target) and not os.path.islink(target):
-                rmtree(target)
-            elif os.path.exists(target):
-                self.execute(os.unlink, (self.distinfo_dir,),
-                             "removing " + target)
-
-            self.execute(os.makedirs, (target,), "creating " + target)
-
-            metadata_path = os.path.join(self.distinfo_dir, 'METADATA')
-            logger.info('creating %s', metadata_path)
-            self.distribution.metadata.write(metadata_path)
-            self.outfiles.append(metadata_path)
-
-            installer_path = os.path.join(self.distinfo_dir, 'INSTALLER')
-            logger.info('creating %s', installer_path)
             with open(installer_path, 'w') as f:
                 f.write(self.installer)
-            self.outfiles.append(installer_path)
+        self.outfiles.append(installer_path)
 
-            if self.requested:
-                requested_path = os.path.join(self.distinfo_dir, 'REQUESTED')
-                logger.info('creating %s', requested_path)
+        if self.requested:
+            requested_path = os.path.join(self.distinfo_dir, 'REQUESTED')
+            logger.info('creating %s', requested_path)
+            if not self.dry_run:
                 open(requested_path, 'wb').close()
-                self.outfiles.append(requested_path)
+            self.outfiles.append(requested_path)
 
-
-            if not self.no_resources:
-                install_data = self.get_finalized_command('install_data')
-                if install_data.get_resources_out() != []:
-                    resources_path = os.path.join(self.distinfo_dir,
-                                                  'RESOURCES')
-                    logger.info('creating %s', resources_path)
+        if not self.no_resources:
+            install_data = self.get_finalized_command('install_data')
+            if install_data.get_resources_out() != []:
+                resources_path = os.path.join(self.distinfo_dir,
+                                              'RESOURCES')
+                logger.info('creating %s', resources_path)
+                if not self.dry_run:
                     with open(resources_path, 'wb') as f:
                         writer = csv.writer(f, delimiter=',',
                                             lineterminator='\n',
                                             quotechar='"')
-                        for tuple in install_data.get_resources_out():
-                            writer.writerow(tuple)
+                        for row in install_data.get_resources_out():
+                            writer.writerow(row)
 
-                        self.outfiles.append(resources_path)
+                self.outfiles.append(resources_path)
 
-            if not self.no_record:
-                record_path = os.path.join(self.distinfo_dir, 'RECORD')
-                logger.info('creating %s', record_path)
+        if not self.no_record:
+            record_path = os.path.join(self.distinfo_dir, 'RECORD')
+            logger.info('creating %s', record_path)
+            if not self.dry_run:
                 with open(record_path, 'w', encoding='utf-8') as f:
                     writer = csv.writer(f, delimiter=',',
                                         lineterminator='\n',
@@ -141,7 +141,7 @@
 
                     # add the RECORD file itself
                     writer.writerow((record_path, '', ''))
-                    self.outfiles.append(record_path)
+            self.outfiles.append(record_path)
 
     def get_outputs(self):
         return self.outfiles

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:42:20 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:42:20 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Use_bytes_regex_instead_of_?=
	=?utf8?q?decoding_whole_pages?=
Message-ID: <E1R38eK-0005z7-0U@dinsdale.python.org>

http://hg.python.org/cpython/rev/77df8ab7914b
changeset:   72352:77df8ab7914b
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 10 18:10:58 2011 +0200
summary:
  Use bytes regex instead of decoding whole pages

files:
  Lib/packaging/pypi/simple.py |  22 ++++++++++------------
  1 files changed, 10 insertions(+), 12 deletions(-)


diff --git a/Lib/packaging/pypi/simple.py b/Lib/packaging/pypi/simple.py
--- a/Lib/packaging/pypi/simple.py
+++ b/Lib/packaging/pypi/simple.py
@@ -159,22 +159,20 @@
 
         Return a list of names.
         """
+        if '*' in name:
+            name.replace('*', '.*')
+        else:
+            name = "%s%s%s" % ('*.?', name, '*.?')
+        name = name.replace('*', '[^<]*')  # avoid matching end tag
+        pattern = ('<a[^>]*>(%s)</a>' % name).encode('utf-8')
+        projectname = re.compile(pattern, re.I)
+        matching_projects = []
+
         with self._open_url(self.index_url) as index:
-            if '*' in name:
-                name.replace('*', '.*')
-            else:
-                name = "%s%s%s" % ('*.?', name, '*.?')
-            name = name.replace('*', '[^<]*')  # avoid matching end tag
-            projectname = re.compile('<a[^>]*>(%s)</a>' % name, re.I)
-            matching_projects = []
-
             index_content = index.read()
 
-        # FIXME should use bytes I/O and regexes instead of decoding
-        index_content = index_content.decode()
-
         for match in projectname.finditer(index_content):
-            project_name = match.group(1)
+            project_name = match.group(1).decode('utf-8')
             matching_projects.append(self._get_project(project_name))
         return matching_projects
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:42:20 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:42:20 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_usage_of_bytes_in_packa?=
 =?utf8?q?ging=27s_bdist=5Fwininst=2E?=
Message-ID: <E1R38eK-0005zB-NT@dinsdale.python.org>

http://hg.python.org/cpython/rev/770d61fa5bab
changeset:   72353:770d61fa5bab
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 10 18:14:08 2011 +0200
summary:
  Fix usage of bytes in packaging's bdist_wininst.

This is copied from the namesake distutils command; there is no
automated test, so buildbots won?t call for my head this time, but it
should be okay as Python 3 users have tested the distutils command.

files:
  Lib/packaging/command/bdist_wininst.py |  15 +++++++------
  1 files changed, 8 insertions(+), 7 deletions(-)


diff --git a/Lib/packaging/command/bdist_wininst.py b/Lib/packaging/command/bdist_wininst.py
--- a/Lib/packaging/command/bdist_wininst.py
+++ b/Lib/packaging/command/bdist_wininst.py
@@ -1,7 +1,5 @@
 """Create an executable installer for Windows."""
 
-# FIXME synchronize bytes/str use with same file in distutils
-
 import sys
 import os
 
@@ -264,14 +262,17 @@
                 cfgdata = cfgdata.encode("mbcs")
 
             # Append the pre-install script
-            cfgdata = cfgdata + "\0"
+            cfgdata = cfgdata + b"\0"
             if self.pre_install_script:
-                with open(self.pre_install_script) as fp:
-                    script_data = fp.read()
-                cfgdata = cfgdata + script_data + "\n\0"
+                # We need to normalize newlines, so we open in text mode and
+                # convert back to bytes. "latin-1" simply avoids any possible
+                # failures.
+                with open(self.pre_install_script, encoding="latin-1") as fp:
+                    script_data = fp.read().encode("latin-1")
+                cfgdata = cfgdata + script_data + b"\n\0"
             else:
                 # empty pre-install script
-                cfgdata = cfgdata + "\0"
+                cfgdata = cfgdata + b"\0"
             file.write(cfgdata)
 
             # The 'magic number' 0x1234567B is used to make sure that the

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:42:21 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:42:21 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Don=E2=80=99t_let_invalid_l?=
 =?utf8?q?ine_in_setup=2Ecfg_pass_silently?=
Message-ID: <E1R38eL-0005zF-Fr@dinsdale.python.org>

http://hg.python.org/cpython/rev/0f69eb7d5bed
changeset:   72354:0f69eb7d5bed
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 10 18:22:04 2011 +0200
summary:
  Don?t let invalid line in setup.cfg pass silently

files:
  Lib/packaging/config.py |  7 ++++---
  1 files changed, 4 insertions(+), 3 deletions(-)


diff --git a/Lib/packaging/config.py b/Lib/packaging/config.py
--- a/Lib/packaging/config.py
+++ b/Lib/packaging/config.py
@@ -227,10 +227,11 @@
                 self.dist.scripts = [self.dist.scripts]
 
             self.dist.package_data = {}
-            for data in files.get('package_data', []):
-                data = data.split('=')
+            for line in files.get('package_data', []):
+                data = line.split('=')
                 if len(data) != 2:
-                    continue  # FIXME errors should never pass silently
+                    raise ValueError('invalid line for package_data: %s '
+                                     '(misses "=")' % line)
                 key, value = data
                 self.dist.package_data[key.strip()] = value.strip()
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:42:22 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:42:22 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_obsolete_comment_=28?=
 =?utf8?q?yes=2C_build=5Fext_supports_C++=29?=
Message-ID: <E1R38eM-0005zK-3l@dinsdale.python.org>

http://hg.python.org/cpython/rev/0d1f310100ce
changeset:   72355:0d1f310100ce
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 10 18:22:31 2011 +0200
summary:
  Remove obsolete comment (yes, build_ext supports C++)

files:
  Lib/packaging/command/build_ext.py |  4 ----
  1 files changed, 0 insertions(+), 4 deletions(-)


diff --git a/Lib/packaging/command/build_ext.py b/Lib/packaging/command/build_ext.py
--- a/Lib/packaging/command/build_ext.py
+++ b/Lib/packaging/command/build_ext.py
@@ -1,9 +1,5 @@
 """Build extension modules."""
 
-# FIXME Is this module limited to C extensions or do C++ extensions work too?
-# The docstring of this module said that C++ was not supported, but other
-# comments contradict that.
-
 import os
 import re
 import sys

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:42:22 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:42:22 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_unneeded_--all_optio?=
 =?utf8?b?biBvZiDigJxweXNldHVwIGxpc3TigJ0u?=
Message-ID: <E1R38eM-000608-OG@dinsdale.python.org>

http://hg.python.org/cpython/rev/df157cb96a99
changeset:   72356:df157cb96a99
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 12 16:45:38 2011 +0200
summary:
  Remove unneeded --all option of ?pysetup list?.

The command without arguments already prints all installed distributions
found.

In addition, change ?releases? for ?projects? in the description of the
list action.  Strictly speaking, one installed distribution satisfies
the requirement for a release (i.e. version) of a project, but as
currently only one release per project can be installed at a time, the
two are somewhat equivalent, and ?project? is more understandable in
help texts (which call their argument ?dist?, by the way..)

files:
  Doc/install/pysetup.rst                          |  17 ++++----
  Lib/packaging/run.py                             |  20 ++++------
  Lib/packaging/tests/test_command_install_data.py |   1 +
  3 files changed, 18 insertions(+), 20 deletions(-)


diff --git a/Doc/install/pysetup.rst b/Doc/install/pysetup.rst
--- a/Doc/install/pysetup.rst
+++ b/Doc/install/pysetup.rst
@@ -19,13 +19,12 @@
 
 Pysetup makes it easy to find out what Python packages are installed::
 
-   $ pysetup search virtualenv
-   virtualenv 1.6 at /opt/python3.3/lib/python3.3/site-packages/virtualenv-1.6-py3.3.egg-info
+   $ pysetup list virtualenv
+   'virtualenv' 1.6 at '/opt/python3.3/lib/python3.3/site-packages/virtualenv-1.6-py3.3.egg-info'
 
-   $ pysetup search --all
-   pyverify 0.8.1 at /opt/python3.3/lib/python3.3/site-packages/pyverify-0.8.1.dist-info
-   virtualenv 1.6 at /opt/python3.3/lib/python3.3/site-packages/virtualenv-1.6-py3.3.egg-info
-   wsgiref 0.1.2 at /opt/python3.3/lib/python3.3/wsgiref.egg-info
+   $ pysetup list
+   'pyverify' 0.8.1 at '/opt/python3.3/lib/python3.3/site-packages/pyverify-0.8.1.dist-info'
+   'virtualenv' 1.6 at '/opt/python3.3/lib/python3.3/site-packages/virtualenv-1.6-py3.3.egg-info'
    ...
 
 
@@ -146,9 +145,11 @@
        metadata: Display the metadata of a project
        install: Install a project
        remove: Remove a project
-       search: Search for a project
+       search: Search for a project in the indexes
+       list: List installed projects
        graph: Display a graph
-       create: Create a Project
+       create: Create a project
+       generate-setup: Generate a backward-comptatible setup.py
 
    To get more help on an action, use:
 
diff --git a/Lib/packaging/run.py b/Lib/packaging/run.py
--- a/Lib/packaging/run.py
+++ b/Lib/packaging/run.py
@@ -290,27 +290,23 @@
 
 
 @action_help("""\
-Usage: pysetup list dist [dist ...]
+Usage: pysetup list [dist ...]
    or: pysetup list --help
-   or: pysetup list --all
 
 Print name, version and location for the matching installed distributions.
 
 positional arguments:
-   dist  installed distribution name
-
-optional arguments:
-   --all  list all installed distributions
+   dist  installed distribution name; omit to get all distributions
 """)
 def _list(dispatcher, args, **kw):
-    opts = _parse_args(args[1:], '', ['all'])
+    opts = _parse_args(args[1:], '', [])
     dists = get_distributions(use_egg_info=True)
-    if 'all' in opts or opts['args'] == []:
+    if opts['args']:
+        results = (d for d in dists if d.name.lower() in opts['args'])
+        listall = False
+    else:
         results = dists
         listall = True
-    else:
-        results = (d for d in dists if d.name.lower() in opts['args'])
-        listall = False
 
     number = 0
     for dist in results:
@@ -368,7 +364,7 @@
     ('install', 'Install a project', _install),
     ('remove', 'Remove a project', _remove),
     ('search', 'Search for a project in the indexes', _search),
-    ('list', 'List installed releases', _list),
+    ('list', 'List installed projects', _list),
     ('graph', 'Display a graph', _graph),
     ('create', 'Create a project', _create),
     ('generate-setup', 'Generate a backward-comptatible setup.py', _generate),
diff --git a/Lib/packaging/tests/test_command_install_data.py b/Lib/packaging/tests/test_command_install_data.py
--- a/Lib/packaging/tests/test_command_install_data.py
+++ b/Lib/packaging/tests/test_command_install_data.py
@@ -35,6 +35,7 @@
         two = os.path.join(pkg_dir, 'two')
         self.write_file(two, 'xxx')
 
+        # FIXME this creates a literal \{inst2\} directory!
         cmd.data_files = {one: '{inst}/one', two: '{inst2}/two'}
         self.assertCountEqual(cmd.get_inputs(), [one, two])
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:42:23 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:42:23 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Wrap_pydoc_outp?=
	=?utf8?q?ut_under_80_characters?=
Message-ID: <E1R38eN-00060C-IM@dinsdale.python.org>

http://hg.python.org/cpython/rev/28ad3ef7301b
changeset:   72357:28ad3ef7301b
branch:      3.2
parent:      72347:e5c1de856828
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Sep 11 00:43:20 2011 +0200
summary:
  Wrap pydoc output under 80 characters

files:
  Lib/pydoc.py |  9 +++++----
  1 files changed, 5 insertions(+), 4 deletions(-)


diff --git a/Lib/pydoc.py b/Lib/pydoc.py
--- a/Lib/pydoc.py
+++ b/Lib/pydoc.py
@@ -1044,10 +1044,11 @@
         if docloc is not None:
             result = result + self.section('MODULE REFERENCE', docloc + """
 
-The following documentation is automatically generated from the Python source
-files.  It may be incomplete, incorrect or include features that are considered
-implementation detail and may vary between Python implementations.  When in
-doubt, consult the module reference at the location listed above.
+The following documentation is automatically generated from the Python
+source files.  It may be incomplete, incorrect or include features that
+are considered implementation detail and may vary between Python
+implementations.  When in doubt, consult the module reference at the
+location listed above.
 """)
 
         if desc:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:42:25 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:42:25 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMik6?=
	=?utf8?q?_Branch_merge?=
Message-ID: <E1R38eP-00060l-1l@dinsdale.python.org>

http://hg.python.org/cpython/rev/882d3b78b1cc
changeset:   72358:882d3b78b1cc
branch:      3.2
parent:      72344:d14f717b5e3d
parent:      72357:28ad3ef7301b
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 12 17:15:26 2011 +0200
summary:
  Branch merge

files:
  Lib/distutils/dist.py            |    3 +-
  Lib/distutils/tests/test_dist.py |  112 ++++++++++--------
  Lib/pydoc.py                     |    9 +-
  Misc/NEWS                        |    4 +
  4 files changed, 72 insertions(+), 56 deletions(-)


diff --git a/Lib/distutils/dist.py b/Lib/distutils/dist.py
--- a/Lib/distutils/dist.py
+++ b/Lib/distutils/dist.py
@@ -1018,7 +1018,8 @@
         """Write the PKG-INFO format data to a file object.
         """
         version = '1.0'
-        if self.provides or self.requires or self.obsoletes:
+        if (self.provides or self.requires or self.obsoletes or
+            self.classifiers or self.download_url):
             version = '1.1'
 
         file.write('Metadata-Version: %s\n' % version)
diff --git a/Lib/distutils/tests/test_dist.py b/Lib/distutils/tests/test_dist.py
--- a/Lib/distutils/tests/test_dist.py
+++ b/Lib/distutils/tests/test_dist.py
@@ -74,7 +74,7 @@
         self.assertEqual(d.get_command_packages(),
                          ["distutils.command", "foo.bar", "distutils.tests"])
         cmd = d.get_command_obj("test_dist")
-        self.assertTrue(isinstance(cmd, test_dist))
+        self.assertIsInstance(cmd, test_dist)
         self.assertEqual(cmd.sample_option, "sometext")
 
     def test_command_packages_configfile(self):
@@ -106,28 +106,23 @@
     def test_empty_options(self):
         # an empty options dictionary should not stay in the
         # list of attributes
-        klass = Distribution
 
         # catching warnings
         warns = []
+
         def _warn(msg):
             warns.append(msg)
 
-        old_warn = warnings.warn
+        self.addCleanup(setattr, warnings, 'warn', warnings.warn)
         warnings.warn = _warn
-        try:
-            dist = klass(attrs={'author': 'xxx',
-                                'name': 'xxx',
-                                'version': 'xxx',
-                                'url': 'xxxx',
-                                'options': {}})
-        finally:
-            warnings.warn = old_warn
+        dist = Distribution(attrs={'author': 'xxx', 'name': 'xxx',
+                                   'version': 'xxx', 'url': 'xxxx',
+                                   'options': {}})
 
         self.assertEqual(len(warns), 0)
+        self.assertNotIn('options', dir(dist))
 
     def test_finalize_options(self):
-
         attrs = {'keywords': 'one,two',
                  'platforms': 'one,two'}
 
@@ -150,7 +145,6 @@
         cmds = dist.get_command_packages()
         self.assertEqual(cmds, ['distutils.command', 'one', 'two'])
 
-
     def test_announce(self):
         # make sure the level is known
         dist = Distribution()
@@ -158,6 +152,7 @@
         kwargs = {'level': 'ok2'}
         self.assertRaises(ValueError, dist.announce, args, kwargs)
 
+
 class MetadataTestCase(support.TempdirManager, support.EnvironGuard,
                        unittest.TestCase):
 
@@ -170,15 +165,20 @@
         sys.argv[:] = self.argv[1]
         super(MetadataTestCase, self).tearDown()
 
+    def format_metadata(self, dist):
+        sio = io.StringIO()
+        dist.metadata.write_pkg_file(sio)
+        return sio.getvalue()
+
     def test_simple_metadata(self):
         attrs = {"name": "package",
                  "version": "1.0"}
         dist = Distribution(attrs)
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.0" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.0", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertNotIn("requires:", meta.lower())
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_provides(self):
         attrs = {"name": "package",
@@ -190,9 +190,9 @@
         self.assertEqual(dist.get_provides(),
                          ["package", "package.sub"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("requires:", meta.lower())
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_provides_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -210,11 +210,11 @@
         self.assertEqual(dist.get_requires(),
                          ["other", "another (==1.0)"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("Requires: other" in meta)
-        self.assertTrue("Requires: another (==1.0)" in meta)
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertIn("Requires: other", meta)
+        self.assertIn("Requires: another (==1.0)", meta)
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_requires_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -232,11 +232,11 @@
         self.assertEqual(dist.get_obsoletes(),
                          ["other", "another (<1.0)"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("Obsoletes: other" in meta)
-        self.assertTrue("Obsoletes: another (<1.0)" in meta)
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertNotIn("requires:", meta.lower())
+        self.assertIn("Obsoletes: other", meta)
+        self.assertIn("Obsoletes: another (<1.0)", meta)
 
     def test_obsoletes_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -244,10 +244,34 @@
                            "version": "1.0",
                            "obsoletes": ["my.pkg (splat)"]})
 
-    def format_metadata(self, dist):
-        sio = io.StringIO()
-        dist.metadata.write_pkg_file(sio)
-        return sio.getvalue()
+    def test_classifier(self):
+        attrs = {'name': 'Boa', 'version': '3.0',
+                 'classifiers': ['Programming Language :: Python :: 3']}
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        self.assertIn('Metadata-Version: 1.1', meta)
+
+    def test_download_url(self):
+        attrs = {'name': 'Boa', 'version': '3.0',
+                 'download_url': 'http://example.org/boa'}
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        self.assertIn('Metadata-Version: 1.1', meta)
+
+    def test_long_description(self):
+        long_desc = textwrap.dedent("""\
+        example::
+              We start here
+            and continue here
+          and end here.""")
+        attrs = {"name": "package",
+                 "version": "1.0",
+                 "long_description": long_desc}
+
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        meta = meta.replace('\n' + 8 * ' ', '\n')
+        self.assertIn(long_desc, meta)
 
     def test_custom_pydistutils(self):
         # fixes #2166
@@ -272,14 +296,14 @@
             if sys.platform in ('linux', 'darwin'):
                 os.environ['HOME'] = temp_dir
                 files = dist.find_config_files()
-                self.assertTrue(user_filename in files)
+                self.assertIn(user_filename, files)
 
             # win32-style
             if sys.platform == 'win32':
                 # home drive should be found
                 os.environ['HOME'] = temp_dir
                 files = dist.find_config_files()
-                self.assertTrue(user_filename in files,
+                self.assertIn(user_filename, files,
                              '%r not found in %r' % (user_filename, files))
         finally:
             os.remove(user_filename)
@@ -301,22 +325,8 @@
 
         output = [line for line in s.getvalue().split('\n')
                   if line.strip() != '']
-        self.assertTrue(len(output) > 0)
+        self.assertTrue(output)
 
-    def test_long_description(self):
-        long_desc = textwrap.dedent("""\
-        example::
-              We start here
-            and continue here
-          and end here.""")
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "long_description": long_desc}
-
-        dist = Distribution(attrs)
-        meta = self.format_metadata(dist)
-        meta = meta.replace('\n' + 8 * ' ', '\n')
-        self.assertTrue(long_desc in meta)
 
 def test_suite():
     suite = unittest.TestSuite()
diff --git a/Lib/pydoc.py b/Lib/pydoc.py
--- a/Lib/pydoc.py
+++ b/Lib/pydoc.py
@@ -1044,10 +1044,11 @@
         if docloc is not None:
             result = result + self.section('MODULE REFERENCE', docloc + """
 
-The following documentation is automatically generated from the Python source
-files.  It may be incomplete, incorrect or include features that are considered
-implementation detail and may vary between Python implementations.  When in
-doubt, consult the module reference at the location listed above.
+The following documentation is automatically generated from the Python
+source files.  It may be incomplete, incorrect or include features that
+are considered implementation detail and may vary between Python
+implementations.  When in doubt, consult the module reference at the
+location listed above.
 """)
 
         if desc:
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -25,6 +25,10 @@
 Library
 -------
 
+- Issue #8933: distutils' PKG-INFO files will now correctly report
+  Metadata-Version: 1.1 instead of 1.0 if a Classifier or Download-URL field is
+  present.
+
 - Issue #9561: distutils now reads and writes egg-info files using UTF-8,
   instead of the locale encoding.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:42:25 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:42:25 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Branch_merge?=
Message-ID: <E1R38eP-00060p-Lu@dinsdale.python.org>

http://hg.python.org/cpython/rev/ad6b879eab02
changeset:   72359:ad6b879eab02
parent:      72345:c91900e4e805
parent:      72356:df157cb96a99
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 12 17:34:40 2011 +0200
summary:
  Branch merge

files:
  Doc/install/pysetup.rst                          |   17 +-
  Lib/distutils/dist.py                            |    3 +-
  Lib/distutils/tests/test_dist.py                 |  112 +-
  Lib/packaging/command/bdist_wininst.py           |   20 +-
  Lib/packaging/command/build_ext.py               |    4 -
  Lib/packaging/command/install_distinfo.py        |   80 +-
  Lib/packaging/config.py                          |    7 +-
  Lib/packaging/metadata.py                        |   17 +-
  Lib/packaging/pypi/simple.py                     |   22 +-
  Lib/packaging/run.py                             |   20 +-
  Lib/packaging/tests/test_command_install_data.py |    1 +
  Lib/packaging/tests/test_dist.py                 |  252 +-----
  Lib/packaging/tests/test_metadata.py             |  390 +++++++--
  Lib/packaging/tests/test_run.py                  |   23 +-
  Misc/NEWS                                        |    4 +
  15 files changed, 488 insertions(+), 484 deletions(-)


diff --git a/Doc/install/pysetup.rst b/Doc/install/pysetup.rst
--- a/Doc/install/pysetup.rst
+++ b/Doc/install/pysetup.rst
@@ -19,13 +19,12 @@
 
 Pysetup makes it easy to find out what Python packages are installed::
 
-   $ pysetup search virtualenv
-   virtualenv 1.6 at /opt/python3.3/lib/python3.3/site-packages/virtualenv-1.6-py3.3.egg-info
+   $ pysetup list virtualenv
+   'virtualenv' 1.6 at '/opt/python3.3/lib/python3.3/site-packages/virtualenv-1.6-py3.3.egg-info'
 
-   $ pysetup search --all
-   pyverify 0.8.1 at /opt/python3.3/lib/python3.3/site-packages/pyverify-0.8.1.dist-info
-   virtualenv 1.6 at /opt/python3.3/lib/python3.3/site-packages/virtualenv-1.6-py3.3.egg-info
-   wsgiref 0.1.2 at /opt/python3.3/lib/python3.3/wsgiref.egg-info
+   $ pysetup list
+   'pyverify' 0.8.1 at '/opt/python3.3/lib/python3.3/site-packages/pyverify-0.8.1.dist-info'
+   'virtualenv' 1.6 at '/opt/python3.3/lib/python3.3/site-packages/virtualenv-1.6-py3.3.egg-info'
    ...
 
 
@@ -146,9 +145,11 @@
        metadata: Display the metadata of a project
        install: Install a project
        remove: Remove a project
-       search: Search for a project
+       search: Search for a project in the indexes
+       list: List installed projects
        graph: Display a graph
-       create: Create a Project
+       create: Create a project
+       generate-setup: Generate a backward-comptatible setup.py
 
    To get more help on an action, use:
 
diff --git a/Lib/distutils/dist.py b/Lib/distutils/dist.py
--- a/Lib/distutils/dist.py
+++ b/Lib/distutils/dist.py
@@ -1018,7 +1018,8 @@
         """Write the PKG-INFO format data to a file object.
         """
         version = '1.0'
-        if self.provides or self.requires or self.obsoletes:
+        if (self.provides or self.requires or self.obsoletes or
+            self.classifiers or self.download_url):
             version = '1.1'
 
         file.write('Metadata-Version: %s\n' % version)
diff --git a/Lib/distutils/tests/test_dist.py b/Lib/distutils/tests/test_dist.py
--- a/Lib/distutils/tests/test_dist.py
+++ b/Lib/distutils/tests/test_dist.py
@@ -74,7 +74,7 @@
         self.assertEqual(d.get_command_packages(),
                          ["distutils.command", "foo.bar", "distutils.tests"])
         cmd = d.get_command_obj("test_dist")
-        self.assertTrue(isinstance(cmd, test_dist))
+        self.assertIsInstance(cmd, test_dist)
         self.assertEqual(cmd.sample_option, "sometext")
 
     def test_command_packages_configfile(self):
@@ -106,28 +106,23 @@
     def test_empty_options(self):
         # an empty options dictionary should not stay in the
         # list of attributes
-        klass = Distribution
 
         # catching warnings
         warns = []
+
         def _warn(msg):
             warns.append(msg)
 
-        old_warn = warnings.warn
+        self.addCleanup(setattr, warnings, 'warn', warnings.warn)
         warnings.warn = _warn
-        try:
-            dist = klass(attrs={'author': 'xxx',
-                                'name': 'xxx',
-                                'version': 'xxx',
-                                'url': 'xxxx',
-                                'options': {}})
-        finally:
-            warnings.warn = old_warn
+        dist = Distribution(attrs={'author': 'xxx', 'name': 'xxx',
+                                   'version': 'xxx', 'url': 'xxxx',
+                                   'options': {}})
 
         self.assertEqual(len(warns), 0)
+        self.assertNotIn('options', dir(dist))
 
     def test_finalize_options(self):
-
         attrs = {'keywords': 'one,two',
                  'platforms': 'one,two'}
 
@@ -150,7 +145,6 @@
         cmds = dist.get_command_packages()
         self.assertEqual(cmds, ['distutils.command', 'one', 'two'])
 
-
     def test_announce(self):
         # make sure the level is known
         dist = Distribution()
@@ -158,6 +152,7 @@
         kwargs = {'level': 'ok2'}
         self.assertRaises(ValueError, dist.announce, args, kwargs)
 
+
 class MetadataTestCase(support.TempdirManager, support.EnvironGuard,
                        unittest.TestCase):
 
@@ -170,15 +165,20 @@
         sys.argv[:] = self.argv[1]
         super(MetadataTestCase, self).tearDown()
 
+    def format_metadata(self, dist):
+        sio = io.StringIO()
+        dist.metadata.write_pkg_file(sio)
+        return sio.getvalue()
+
     def test_simple_metadata(self):
         attrs = {"name": "package",
                  "version": "1.0"}
         dist = Distribution(attrs)
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.0" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.0", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertNotIn("requires:", meta.lower())
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_provides(self):
         attrs = {"name": "package",
@@ -190,9 +190,9 @@
         self.assertEqual(dist.get_provides(),
                          ["package", "package.sub"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("requires:", meta.lower())
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_provides_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -210,11 +210,11 @@
         self.assertEqual(dist.get_requires(),
                          ["other", "another (==1.0)"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("Requires: other" in meta)
-        self.assertTrue("Requires: another (==1.0)" in meta)
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertIn("Requires: other", meta)
+        self.assertIn("Requires: another (==1.0)", meta)
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_requires_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -232,11 +232,11 @@
         self.assertEqual(dist.get_obsoletes(),
                          ["other", "another (<1.0)"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("Obsoletes: other" in meta)
-        self.assertTrue("Obsoletes: another (<1.0)" in meta)
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertNotIn("requires:", meta.lower())
+        self.assertIn("Obsoletes: other", meta)
+        self.assertIn("Obsoletes: another (<1.0)", meta)
 
     def test_obsoletes_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -244,10 +244,34 @@
                            "version": "1.0",
                            "obsoletes": ["my.pkg (splat)"]})
 
-    def format_metadata(self, dist):
-        sio = io.StringIO()
-        dist.metadata.write_pkg_file(sio)
-        return sio.getvalue()
+    def test_classifier(self):
+        attrs = {'name': 'Boa', 'version': '3.0',
+                 'classifiers': ['Programming Language :: Python :: 3']}
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        self.assertIn('Metadata-Version: 1.1', meta)
+
+    def test_download_url(self):
+        attrs = {'name': 'Boa', 'version': '3.0',
+                 'download_url': 'http://example.org/boa'}
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        self.assertIn('Metadata-Version: 1.1', meta)
+
+    def test_long_description(self):
+        long_desc = textwrap.dedent("""\
+        example::
+              We start here
+            and continue here
+          and end here.""")
+        attrs = {"name": "package",
+                 "version": "1.0",
+                 "long_description": long_desc}
+
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        meta = meta.replace('\n' + 8 * ' ', '\n')
+        self.assertIn(long_desc, meta)
 
     def test_custom_pydistutils(self):
         # fixes #2166
@@ -272,14 +296,14 @@
             if sys.platform in ('linux', 'darwin'):
                 os.environ['HOME'] = temp_dir
                 files = dist.find_config_files()
-                self.assertTrue(user_filename in files)
+                self.assertIn(user_filename, files)
 
             # win32-style
             if sys.platform == 'win32':
                 # home drive should be found
                 os.environ['HOME'] = temp_dir
                 files = dist.find_config_files()
-                self.assertTrue(user_filename in files,
+                self.assertIn(user_filename, files,
                              '%r not found in %r' % (user_filename, files))
         finally:
             os.remove(user_filename)
@@ -301,22 +325,8 @@
 
         output = [line for line in s.getvalue().split('\n')
                   if line.strip() != '']
-        self.assertTrue(len(output) > 0)
+        self.assertTrue(output)
 
-    def test_long_description(self):
-        long_desc = textwrap.dedent("""\
-        example::
-              We start here
-            and continue here
-          and end here.""")
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "long_description": long_desc}
-
-        dist = Distribution(attrs)
-        meta = self.format_metadata(dist)
-        meta = meta.replace('\n' + 8 * ' ', '\n')
-        self.assertTrue(long_desc in meta)
 
 def test_suite():
     suite = unittest.TestSuite()
diff --git a/Lib/packaging/command/bdist_wininst.py b/Lib/packaging/command/bdist_wininst.py
--- a/Lib/packaging/command/bdist_wininst.py
+++ b/Lib/packaging/command/bdist_wininst.py
@@ -1,7 +1,5 @@
 """Create an executable installer for Windows."""
 
-# FIXME synchronize bytes/str use with same file in distutils
-
 import sys
 import os
 
@@ -186,9 +184,8 @@
         os.remove(arcname)
 
         if not self.keep_temp:
-            if self.dry_run:
-                logger.info('removing %s', self.bdist_dir)
-            else:
+            logger.info('removing %s', self.bdist_dir)
+            if not self.dry_run:
                 rmtree(self.bdist_dir)
 
     def get_inidata(self):
@@ -265,14 +262,17 @@
                 cfgdata = cfgdata.encode("mbcs")
 
             # Append the pre-install script
-            cfgdata = cfgdata + "\0"
+            cfgdata = cfgdata + b"\0"
             if self.pre_install_script:
-                with open(self.pre_install_script) as fp:
-                    script_data = fp.read()
-                cfgdata = cfgdata + script_data + "\n\0"
+                # We need to normalize newlines, so we open in text mode and
+                # convert back to bytes. "latin-1" simply avoids any possible
+                # failures.
+                with open(self.pre_install_script, encoding="latin-1") as fp:
+                    script_data = fp.read().encode("latin-1")
+                cfgdata = cfgdata + script_data + b"\n\0"
             else:
                 # empty pre-install script
-                cfgdata = cfgdata + "\0"
+                cfgdata = cfgdata + b"\0"
             file.write(cfgdata)
 
             # The 'magic number' 0x1234567B is used to make sure that the
diff --git a/Lib/packaging/command/build_ext.py b/Lib/packaging/command/build_ext.py
--- a/Lib/packaging/command/build_ext.py
+++ b/Lib/packaging/command/build_ext.py
@@ -1,9 +1,5 @@
 """Build extension modules."""
 
-# FIXME Is this module limited to C extensions or do C++ extensions work too?
-# The docstring of this module said that C++ was not supported, but other
-# comments contradict that.
-
 import os
 import re
 import sys
diff --git a/Lib/packaging/command/install_distinfo.py b/Lib/packaging/command/install_distinfo.py
--- a/Lib/packaging/command/install_distinfo.py
+++ b/Lib/packaging/command/install_distinfo.py
@@ -28,7 +28,7 @@
         ('no-record', None,
          "do not generate a RECORD file"),
         ('no-resources', None,
-         "do not generate a RESSOURCES list installed file")
+         "do not generate a RESSOURCES list installed file"),
     ]
 
     boolean_options = ['requested', 'no-record', 'no-resources']
@@ -70,56 +70,56 @@
         self.distinfo_dir = os.path.join(self.distinfo_dir, basename)
 
     def run(self):
-        # FIXME dry-run should be used at a finer level, so that people get
-        # useful logging output and can have an idea of what the command would
-        # have done
+        target = self.distinfo_dir
+
+        if os.path.isdir(target) and not os.path.islink(target):
+            if not self.dry_run:
+                rmtree(target)
+        elif os.path.exists(target):
+            self.execute(os.unlink, (self.distinfo_dir,),
+                         "removing " + target)
+
+        self.execute(os.makedirs, (target,), "creating " + target)
+
+        metadata_path = os.path.join(self.distinfo_dir, 'METADATA')
+        self.execute(self.distribution.metadata.write, (metadata_path,),
+                     "creating " + metadata_path)
+        self.outfiles.append(metadata_path)
+
+        installer_path = os.path.join(self.distinfo_dir, 'INSTALLER')
+        logger.info('creating %s', installer_path)
         if not self.dry_run:
-            target = self.distinfo_dir
-
-            if os.path.isdir(target) and not os.path.islink(target):
-                rmtree(target)
-            elif os.path.exists(target):
-                self.execute(os.unlink, (self.distinfo_dir,),
-                             "removing " + target)
-
-            self.execute(os.makedirs, (target,), "creating " + target)
-
-            metadata_path = os.path.join(self.distinfo_dir, 'METADATA')
-            logger.info('creating %s', metadata_path)
-            self.distribution.metadata.write(metadata_path)
-            self.outfiles.append(metadata_path)
-
-            installer_path = os.path.join(self.distinfo_dir, 'INSTALLER')
-            logger.info('creating %s', installer_path)
             with open(installer_path, 'w') as f:
                 f.write(self.installer)
-            self.outfiles.append(installer_path)
+        self.outfiles.append(installer_path)
 
-            if self.requested:
-                requested_path = os.path.join(self.distinfo_dir, 'REQUESTED')
-                logger.info('creating %s', requested_path)
+        if self.requested:
+            requested_path = os.path.join(self.distinfo_dir, 'REQUESTED')
+            logger.info('creating %s', requested_path)
+            if not self.dry_run:
                 open(requested_path, 'wb').close()
-                self.outfiles.append(requested_path)
+            self.outfiles.append(requested_path)
 
-
-            if not self.no_resources:
-                install_data = self.get_finalized_command('install_data')
-                if install_data.get_resources_out() != []:
-                    resources_path = os.path.join(self.distinfo_dir,
-                                                  'RESOURCES')
-                    logger.info('creating %s', resources_path)
+        if not self.no_resources:
+            install_data = self.get_finalized_command('install_data')
+            if install_data.get_resources_out() != []:
+                resources_path = os.path.join(self.distinfo_dir,
+                                              'RESOURCES')
+                logger.info('creating %s', resources_path)
+                if not self.dry_run:
                     with open(resources_path, 'wb') as f:
                         writer = csv.writer(f, delimiter=',',
                                             lineterminator='\n',
                                             quotechar='"')
-                        for tuple in install_data.get_resources_out():
-                            writer.writerow(tuple)
+                        for row in install_data.get_resources_out():
+                            writer.writerow(row)
 
-                        self.outfiles.append(resources_path)
+                self.outfiles.append(resources_path)
 
-            if not self.no_record:
-                record_path = os.path.join(self.distinfo_dir, 'RECORD')
-                logger.info('creating %s', record_path)
+        if not self.no_record:
+            record_path = os.path.join(self.distinfo_dir, 'RECORD')
+            logger.info('creating %s', record_path)
+            if not self.dry_run:
                 with open(record_path, 'w', encoding='utf-8') as f:
                     writer = csv.writer(f, delimiter=',',
                                         lineterminator='\n',
@@ -141,7 +141,7 @@
 
                     # add the RECORD file itself
                     writer.writerow((record_path, '', ''))
-                    self.outfiles.append(record_path)
+            self.outfiles.append(record_path)
 
     def get_outputs(self):
         return self.outfiles
diff --git a/Lib/packaging/config.py b/Lib/packaging/config.py
--- a/Lib/packaging/config.py
+++ b/Lib/packaging/config.py
@@ -227,10 +227,11 @@
                 self.dist.scripts = [self.dist.scripts]
 
             self.dist.package_data = {}
-            for data in files.get('package_data', []):
-                data = data.split('=')
+            for line in files.get('package_data', []):
+                data = line.split('=')
                 if len(data) != 2:
-                    continue  # FIXME errors should never pass silently
+                    raise ValueError('invalid line for package_data: %s '
+                                     '(misses "=")' % line)
                 key, value = data
                 self.dist.package_data[key.strip()] = value.strip()
 
diff --git a/Lib/packaging/metadata.py b/Lib/packaging/metadata.py
--- a/Lib/packaging/metadata.py
+++ b/Lib/packaging/metadata.py
@@ -61,7 +61,8 @@
                'License', 'Classifier', 'Download-URL', 'Obsoletes',
                'Provides', 'Requires')
 
-_314_MARKERS = ('Obsoletes', 'Provides', 'Requires')
+_314_MARKERS = ('Obsoletes', 'Provides', 'Requires', 'Classifier',
+                'Download-URL')
 
 _345_FIELDS = ('Metadata-Version', 'Name', 'Version', 'Platform',
                'Supported-Platform', 'Summary', 'Description',
@@ -354,11 +355,20 @@
         Keys that don't match a metadata field or that have an empty value are
         dropped.
         """
+        # XXX the code should just use self.set, which does tbe same checks and
+        # conversions already, but that would break packaging.pypi: it uses the
+        # update method, which does not call _set_best_version (which set
+        # does), and thus allows having a Metadata object (as long as you don't
+        # modify or write it) with extra fields from PyPI that are not fields
+        # defined in Metadata PEPs.  to solve it, the best_version system
+        # should be reworked so that it's called only for writing, or in a new
+        # strict mode, or with a new, more lax Metadata subclass in p7g.pypi
         def _set(key, value):
             if key in _ATTR2FIELD and value:
                 self.set(self._convert_name(key), value)
 
-        if other is None:
+        if not other:
+            # other is None or empty container
             pass
         elif hasattr(other, 'keys'):
             for k in other.keys():
@@ -368,7 +378,8 @@
                 _set(k, v)
 
         if kwargs:
-            self.update(kwargs)
+            for k, v in kwargs.items():
+                _set(k, v)
 
     def set(self, name, value):
         """Control then set a metadata field."""
diff --git a/Lib/packaging/pypi/simple.py b/Lib/packaging/pypi/simple.py
--- a/Lib/packaging/pypi/simple.py
+++ b/Lib/packaging/pypi/simple.py
@@ -159,22 +159,20 @@
 
         Return a list of names.
         """
+        if '*' in name:
+            name.replace('*', '.*')
+        else:
+            name = "%s%s%s" % ('*.?', name, '*.?')
+        name = name.replace('*', '[^<]*')  # avoid matching end tag
+        pattern = ('<a[^>]*>(%s)</a>' % name).encode('utf-8')
+        projectname = re.compile(pattern, re.I)
+        matching_projects = []
+
         with self._open_url(self.index_url) as index:
-            if '*' in name:
-                name.replace('*', '.*')
-            else:
-                name = "%s%s%s" % ('*.?', name, '*.?')
-            name = name.replace('*', '[^<]*')  # avoid matching end tag
-            projectname = re.compile('<a[^>]*>(%s)</a>' % name, re.I)
-            matching_projects = []
-
             index_content = index.read()
 
-        # FIXME should use bytes I/O and regexes instead of decoding
-        index_content = index_content.decode()
-
         for match in projectname.finditer(index_content):
-            project_name = match.group(1)
+            project_name = match.group(1).decode('utf-8')
             matching_projects.append(self._get_project(project_name))
         return matching_projects
 
diff --git a/Lib/packaging/run.py b/Lib/packaging/run.py
--- a/Lib/packaging/run.py
+++ b/Lib/packaging/run.py
@@ -290,27 +290,23 @@
 
 
 @action_help("""\
-Usage: pysetup list dist [dist ...]
+Usage: pysetup list [dist ...]
    or: pysetup list --help
-   or: pysetup list --all
 
 Print name, version and location for the matching installed distributions.
 
 positional arguments:
-   dist  installed distribution name
-
-optional arguments:
-   --all  list all installed distributions
+   dist  installed distribution name; omit to get all distributions
 """)
 def _list(dispatcher, args, **kw):
-    opts = _parse_args(args[1:], '', ['all'])
+    opts = _parse_args(args[1:], '', [])
     dists = get_distributions(use_egg_info=True)
-    if 'all' in opts or opts['args'] == []:
+    if opts['args']:
+        results = (d for d in dists if d.name.lower() in opts['args'])
+        listall = False
+    else:
         results = dists
         listall = True
-    else:
-        results = (d for d in dists if d.name.lower() in opts['args'])
-        listall = False
 
     number = 0
     for dist in results:
@@ -368,7 +364,7 @@
     ('install', 'Install a project', _install),
     ('remove', 'Remove a project', _remove),
     ('search', 'Search for a project in the indexes', _search),
-    ('list', 'List installed releases', _list),
+    ('list', 'List installed projects', _list),
     ('graph', 'Display a graph', _graph),
     ('create', 'Create a project', _create),
     ('generate-setup', 'Generate a backward-comptatible setup.py', _generate),
diff --git a/Lib/packaging/tests/test_command_install_data.py b/Lib/packaging/tests/test_command_install_data.py
--- a/Lib/packaging/tests/test_command_install_data.py
+++ b/Lib/packaging/tests/test_command_install_data.py
@@ -35,6 +35,7 @@
         two = os.path.join(pkg_dir, 'two')
         self.write_file(two, 'xxx')
 
+        # FIXME this creates a literal \{inst2\} directory!
         cmd.data_files = {one: '{inst}/one', two: '{inst2}/two'}
         self.assertCountEqual(cmd.get_inputs(), [one, two])
 
diff --git a/Lib/packaging/tests/test_dist.py b/Lib/packaging/tests/test_dist.py
--- a/Lib/packaging/tests/test_dist.py
+++ b/Lib/packaging/tests/test_dist.py
@@ -1,16 +1,13 @@
 """Tests for packaging.dist."""
 import os
-import io
 import sys
 import logging
 import textwrap
-import sysconfig
 import packaging.dist
 
 from packaging.dist import Distribution
 from packaging.command import set_command
 from packaging.command.cmd import Command
-from packaging.metadata import Metadata
 from packaging.errors import PackagingModuleError, PackagingOptionError
 from packaging.tests import TESTFN, captured_stdout
 from packaging.tests import support, unittest
@@ -49,6 +46,7 @@
         sys.argv[:] = self.argv[1]
         super(DistributionTestCase, self).tearDown()
 
+    @unittest.skip('needs to be updated')
     def test_debug_mode(self):
         self.addCleanup(os.unlink, TESTFN)
         with open(TESTFN, "w") as f:
@@ -59,6 +57,8 @@
         sys.argv.append("build")
         __, stdout = captured_stdout(create_distribution, files)
         self.assertEqual(stdout, '')
+        # XXX debug mode does not exist anymore, test logging levels in this
+        # test instead
         packaging.dist.DEBUG = True
         try:
             __, stdout = captured_stdout(create_distribution, files)
@@ -66,34 +66,6 @@
         finally:
             packaging.dist.DEBUG = False
 
-    def test_write_pkg_file(self):
-        # Check Metadata handling of Unicode fields
-        tmp_dir = self.mkdtemp()
-        my_file = os.path.join(tmp_dir, 'f')
-        cls = Distribution
-
-        dist = cls(attrs={'author': 'Mister Caf?',
-                          'name': 'my.package',
-                          'maintainer': 'Caf? Junior',
-                          'summary': 'Caf? torr?fi?',
-                          'description': 'H?h?h?'})
-
-        # let's make sure the file can be written
-        # with Unicode fields. they are encoded with
-        # PKG_INFO_ENCODING
-        with open(my_file, 'w', encoding='utf-8') as fp:
-            dist.metadata.write_file(fp)
-
-        # regular ascii is of course always usable
-        dist = cls(attrs={'author': 'Mister Cafe',
-                          'name': 'my.package',
-                          'maintainer': 'Cafe Junior',
-                          'summary': 'Cafe torrefie',
-                          'description': 'Hehehe'})
-
-        with open(my_file, 'w') as fp:
-            dist.metadata.write_file(fp)
-
     def test_bad_attr(self):
         Distribution(attrs={'author': 'xxx',
                             'name': 'xxx',
@@ -101,28 +73,18 @@
                             'home-page': 'xxxx',
                             'badoptname': 'xxx'})
         logs = self.get_logs(logging.WARNING)
-        self.assertEqual(1, len(logs))
+        self.assertEqual(len(logs), 1)
         self.assertIn('unknown argument', logs[0])
 
-    def test_bad_version(self):
-        Distribution(attrs={'author': 'xxx',
-                            'name': 'xxx',
-                            'version': 'xxx',
-                            'home-page': 'xxxx'})
-        logs = self.get_logs(logging.WARNING)
-        self.assertEqual(1, len(logs))
-        self.assertIn('not a valid version', logs[0])
-
     def test_empty_options(self):
         # an empty options dictionary should not stay in the
         # list of attributes
-        Distribution(attrs={'author': 'xxx',
-                            'name': 'xxx',
-                            'version': '1.2',
-                            'home-page': 'xxxx',
-                            'options': {}})
+        dist = Distribution(attrs={'author': 'xxx', 'name': 'xxx',
+                                   'version': '1.2', 'home-page': 'xxxx',
+                                   'options': {}})
 
         self.assertEqual([], self.get_logs(logging.WARNING))
+        self.assertNotIn('options', dir(dist))
 
     def test_non_empty_options(self):
         # TODO: how to actually use options is not documented except
@@ -141,7 +103,6 @@
         self.assertIn('owner', dist.get_option_dict('sdist'))
 
     def test_finalize_options(self):
-
         attrs = {'keywords': 'one,two',
                  'platform': 'one,two'}
 
@@ -152,6 +113,24 @@
         self.assertEqual(dist.metadata['platform'], ['one', 'two'])
         self.assertEqual(dist.metadata['keywords'], ['one', 'two'])
 
+    def test_custom_pydistutils(self):
+        # Bug #2166:  make sure pydistutils.cfg is found
+        if os.name == 'posix':
+            user_filename = ".pydistutils.cfg"
+        else:
+            user_filename = "pydistutils.cfg"
+
+        temp_dir = self.mkdtemp()
+        user_filename = os.path.join(temp_dir, user_filename)
+        with open(user_filename, 'w') as f:
+            f.write('.')
+
+        dist = Distribution()
+
+        os.environ['HOME'] = temp_dir
+        files = dist.find_config_files()
+        self.assertIn(user_filename, files)
+
     def test_find_config_files_disable(self):
         # Bug #1180: Allow users to disable their own config file.
         temp_home = self.mkdtemp()
@@ -270,185 +249,8 @@
         self.assertRaises(PackagingOptionError, d.run_command, 'test_dist')
 
 
-class MetadataTestCase(support.TempdirManager,
-                       support.LoggingCatcher,
-                       support.EnvironRestorer,
-                       unittest.TestCase):
-
-    restore_environ = ['HOME']
-
-    def setUp(self):
-        super(MetadataTestCase, self).setUp()
-        self.argv = sys.argv, sys.argv[:]
-
-    def tearDown(self):
-        sys.argv = self.argv[0]
-        sys.argv[:] = self.argv[1]
-        super(MetadataTestCase, self).tearDown()
-
-    def test_simple_metadata(self):
-        attrs = {"name": "package",
-                 "version": "1.0"}
-        dist = Distribution(attrs)
-        meta = self.format_metadata(dist)
-        self.assertIn("Metadata-Version: 1.0", meta)
-        self.assertNotIn("provides:", meta.lower())
-        self.assertNotIn("requires:", meta.lower())
-        self.assertNotIn("obsoletes:", meta.lower())
-
-    def test_provides_dist(self):
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "provides_dist": ["package", "package.sub"]}
-        dist = Distribution(attrs)
-        self.assertEqual(dist.metadata['Provides-Dist'],
-                         ["package", "package.sub"])
-        meta = self.format_metadata(dist)
-        self.assertIn("Metadata-Version: 1.2", meta)
-        self.assertNotIn("requires:", meta.lower())
-        self.assertNotIn("obsoletes:", meta.lower())
-
-    def _test_provides_illegal(self):
-        # XXX to do: check the versions
-        self.assertRaises(ValueError, Distribution,
-                          {"name": "package",
-                           "version": "1.0",
-                           "provides_dist": ["my.pkg (splat)"]})
-
-    def test_requires_dist(self):
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "requires_dist": ["other", "another (==1.0)"]}
-        dist = Distribution(attrs)
-        self.assertEqual(dist.metadata['Requires-Dist'],
-                         ["other", "another (==1.0)"])
-        meta = self.format_metadata(dist)
-        self.assertIn("Metadata-Version: 1.2", meta)
-        self.assertNotIn("provides:", meta.lower())
-        self.assertIn("Requires-Dist: other", meta)
-        self.assertIn("Requires-Dist: another (==1.0)", meta)
-        self.assertNotIn("obsoletes:", meta.lower())
-
-    def _test_requires_illegal(self):
-        # XXX
-        self.assertRaises(ValueError, Distribution,
-                          {"name": "package",
-                           "version": "1.0",
-                           "requires": ["my.pkg (splat)"]})
-
-    def test_obsoletes_dist(self):
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "obsoletes_dist": ["other", "another (<1.0)"]}
-        dist = Distribution(attrs)
-        self.assertEqual(dist.metadata['Obsoletes-Dist'],
-                         ["other", "another (<1.0)"])
-        meta = self.format_metadata(dist)
-        self.assertIn("Metadata-Version: 1.2", meta)
-        self.assertNotIn("provides:", meta.lower())
-        self.assertNotIn("requires:", meta.lower())
-        self.assertIn("Obsoletes-Dist: other", meta)
-        self.assertIn("Obsoletes-Dist: another (<1.0)", meta)
-
-    def _test_obsoletes_illegal(self):
-        # XXX
-        self.assertRaises(ValueError, Distribution,
-                          {"name": "package",
-                           "version": "1.0",
-                           "obsoletes": ["my.pkg (splat)"]})
-
-    def format_metadata(self, dist):
-        sio = io.StringIO()
-        dist.metadata.write_file(sio)
-        return sio.getvalue()
-
-    def test_custom_pydistutils(self):
-        # fixes #2166
-        # make sure pydistutils.cfg is found
-        if os.name == 'posix':
-            user_filename = ".pydistutils.cfg"
-        else:
-            user_filename = "pydistutils.cfg"
-
-        temp_dir = self.mkdtemp()
-        user_filename = os.path.join(temp_dir, user_filename)
-        with open(user_filename, 'w') as f:
-            f.write('.')
-
-        dist = Distribution()
-
-        # linux-style
-        if sys.platform in ('linux', 'darwin'):
-            os.environ['HOME'] = temp_dir
-            files = dist.find_config_files()
-            self.assertIn(user_filename, files)
-
-        # win32-style
-        if sys.platform == 'win32':
-            # home drive should be found
-            os.environ['HOME'] = temp_dir
-            files = dist.find_config_files()
-            self.assertIn(user_filename, files)
-
-    def test_show_help(self):
-        # smoke test, just makes sure some help is displayed
-        dist = Distribution()
-        sys.argv = []
-        dist.help = True
-        dist.script_name = os.path.join(sysconfig.get_path('scripts'),
-                                        'pysetup')
-        __, stdout = captured_stdout(dist.parse_command_line)
-        output = [line for line in stdout.split('\n')
-                  if line.strip() != '']
-        self.assertGreater(len(output), 0)
-
-    def test_description(self):
-        desc = textwrap.dedent("""\
-        example::
-              We start here
-            and continue here
-          and end here.""")
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "description": desc}
-
-        dist = packaging.dist.Distribution(attrs)
-        meta = self.format_metadata(dist)
-        meta = meta.replace('\n' + 7 * ' ' + '|', '\n')
-        self.assertIn(desc, meta)
-
-    def test_read_metadata(self):
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "description": "desc",
-                 "summary": "xxx",
-                 "download_url": "http://example.com",
-                 "keywords": ['one', 'two'],
-                 "requires_dist": ['foo']}
-
-        dist = Distribution(attrs)
-        PKG_INFO = io.StringIO()
-        dist.metadata.write_file(PKG_INFO)
-        PKG_INFO.seek(0)
-
-        metadata = Metadata()
-        metadata.read_file(PKG_INFO)
-
-        self.assertEqual(metadata['name'], "package")
-        self.assertEqual(metadata['version'], "1.0")
-        self.assertEqual(metadata['summary'], "xxx")
-        self.assertEqual(metadata['download_url'], 'http://example.com')
-        self.assertEqual(metadata['keywords'], ['one', 'two'])
-        self.assertEqual(metadata['platform'], [])
-        self.assertEqual(metadata['obsoletes'], [])
-        self.assertEqual(metadata['requires-dist'], ['foo'])
-
-
 def test_suite():
-    suite = unittest.TestSuite()
-    suite.addTest(unittest.makeSuite(DistributionTestCase))
-    suite.addTest(unittest.makeSuite(MetadataTestCase))
-    return suite
+    return unittest.makeSuite(DistributionTestCase)
 
 if __name__ == "__main__":
     unittest.main(defaultTest="test_suite")
diff --git a/Lib/packaging/tests/test_metadata.py b/Lib/packaging/tests/test_metadata.py
--- a/Lib/packaging/tests/test_metadata.py
+++ b/Lib/packaging/tests/test_metadata.py
@@ -2,6 +2,7 @@
 import os
 import sys
 import logging
+from textwrap import dedent
 from io import StringIO
 
 from packaging.errors import (MetadataConflictError, MetadataMissingError,
@@ -9,12 +10,29 @@
 from packaging.metadata import Metadata, PKG_INFO_PREFERRED_VERSION
 
 from packaging.tests import unittest
-from packaging.tests.support import LoggingCatcher
+from packaging.tests.support import (LoggingCatcher, TempdirManager,
+                                     EnvironRestorer)
 
 
 class MetadataTestCase(LoggingCatcher,
+                       TempdirManager,
+                       EnvironRestorer,
                        unittest.TestCase):
 
+    maxDiff = None
+    restore_environ = ['HOME']
+
+    def setUp(self):
+        super(MetadataTestCase, self).setUp()
+        self.argv = sys.argv, sys.argv[:]
+
+    def tearDown(self):
+        sys.argv = self.argv[0]
+        sys.argv[:] = self.argv[1]
+        super(MetadataTestCase, self).tearDown()
+
+    ####  Test various methods of the Metadata class
+
     def test_instantiation(self):
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
         with open(PKG_INFO, 'r', encoding='utf-8') as f:
@@ -43,17 +61,6 @@
         self.assertRaises(TypeError, Metadata,
                           PKG_INFO, mapping=m, fileobj=fp)
 
-    def test_metadata_read_write(self):
-        PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
-        metadata = Metadata(PKG_INFO)
-        out = StringIO()
-        metadata.write_file(out)
-        out.seek(0)
-        res = Metadata()
-        res.read_file(out)
-        for k in metadata:
-            self.assertEqual(metadata[k], res[k])
-
     def test_metadata_markers(self):
         # see if we can be platform-aware
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
@@ -70,31 +77,10 @@
 
         # test with context
         context = {'sys.platform': 'okook'}
-        metadata = Metadata(platform_dependent=True,
-                                        execution_context=context)
+        metadata = Metadata(platform_dependent=True, execution_context=context)
         metadata.read_file(StringIO(content))
         self.assertEqual(metadata['Requires-Dist'], ['foo'])
 
-    def test_description(self):
-        PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
-        with open(PKG_INFO, 'r', encoding='utf-8') as f:
-            content = f.read() % sys.platform
-        metadata = Metadata()
-        metadata.read_file(StringIO(content))
-
-        # see if we can read the description now
-        DESC = os.path.join(os.path.dirname(__file__), 'LONG_DESC.txt')
-        with open(DESC) as f:
-            wanted = f.read()
-        self.assertEqual(wanted, metadata['Description'])
-
-        # save the file somewhere and make sure we can read it back
-        out = StringIO()
-        metadata.write_file(out)
-        out.seek(0)
-        metadata.read_file(out)
-        self.assertEqual(wanted, metadata['Description'])
-
     def test_mapping_api(self):
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
         with open(PKG_INFO, 'r', encoding='utf-8') as f:
@@ -109,73 +95,86 @@
         metadata.update([('version', '0.7')])
         self.assertEqual(metadata['Version'], '0.7')
 
+        # make sure update method checks values like the set method does
+        metadata.update({'version': '1--2'})
+        self.assertEqual(len(self.get_logs()), 1)
+
+        # XXX caveat: the keys method and friends are not 3.x-style views
+        # should be changed or documented
         self.assertEqual(list(metadata), list(metadata.keys()))
 
-    def test_versions(self):
-        metadata = Metadata()
-        metadata['Obsoletes'] = 'ok'
-        self.assertEqual(metadata['Metadata-Version'], '1.1')
+    def test_read_metadata(self):
+        fields = {'name': 'project',
+                  'version': '1.0',
+                  'description': 'desc',
+                  'summary': 'xxx',
+                  'download_url': 'http://example.com',
+                  'keywords': ['one', 'two'],
+                  'requires_dist': ['foo']}
 
-        del metadata['Obsoletes']
-        metadata['Obsoletes-Dist'] = 'ok'
-        self.assertEqual(metadata['Metadata-Version'], '1.2')
+        metadata = Metadata(mapping=fields)
+        PKG_INFO = StringIO()
+        metadata.write_file(PKG_INFO)
+        PKG_INFO.seek(0)
 
-        self.assertRaises(MetadataConflictError, metadata.set,
-                          'Obsoletes', 'ok')
+        metadata = Metadata(fileobj=PKG_INFO)
 
-        del metadata['Obsoletes']
-        del metadata['Obsoletes-Dist']
-        metadata['Version'] = '1'
-        self.assertEqual(metadata['Metadata-Version'], '1.0')
+        self.assertEqual(metadata['name'], 'project')
+        self.assertEqual(metadata['version'], '1.0')
+        self.assertEqual(metadata['summary'], 'xxx')
+        self.assertEqual(metadata['download_url'], 'http://example.com')
+        self.assertEqual(metadata['keywords'], ['one', 'two'])
+        self.assertEqual(metadata['platform'], [])
+        self.assertEqual(metadata['obsoletes'], [])
+        self.assertEqual(metadata['requires-dist'], ['foo'])
 
-        PKG_INFO = os.path.join(os.path.dirname(__file__),
-                                'SETUPTOOLS-PKG-INFO')
-        with open(PKG_INFO, 'r', encoding='utf-8') as f:
-            content = f.read()
-        metadata.read_file(StringIO(content))
-        self.assertEqual(metadata['Metadata-Version'], '1.0')
+    def test_write_metadata(self):
+        # check support of non-ASCII values
+        tmp_dir = self.mkdtemp()
+        my_file = os.path.join(tmp_dir, 'f')
 
-        PKG_INFO = os.path.join(os.path.dirname(__file__),
-                                'SETUPTOOLS-PKG-INFO2')
-        with open(PKG_INFO, 'r', encoding='utf-8') as f:
-            content = f.read()
-        metadata.read_file(StringIO(content))
-        self.assertEqual(metadata['Metadata-Version'], '1.1')
+        metadata = Metadata(mapping={'author': 'Mister Caf?',
+                                     'name': 'my.project',
+                                     'author': 'Caf? Junior',
+                                     'summary': 'Caf? torr?fi?',
+                                     'description': 'H?h?h?',
+                                     'keywords': ['caf?', 'coffee']})
+        metadata.write(my_file)
 
-        # Update the _fields dict directly to prevent 'Metadata-Version'
-        # from being updated by the _set_best_version() method.
-        metadata._fields['Metadata-Version'] = '1.618'
-        self.assertRaises(MetadataUnrecognizedVersionError, metadata.keys)
+        # the file should use UTF-8
+        metadata2 = Metadata()
+        with open(my_file, encoding='utf-8') as fp:
+            metadata2.read_file(fp)
 
-    def test_warnings(self):
-        metadata = Metadata()
+        # XXX when keywords are not defined, metadata will have
+        # 'Keywords': [] but metadata2 will have 'Keywords': ['']
+        # because of a value.split(',') in Metadata.get
+        self.assertEqual(metadata.items(), metadata2.items())
 
-        # these should raise a warning
-        values = (('Requires-Dist', 'Funky (Groovie)'),
-                  ('Requires-Python', '1-4'))
+        # ASCII also works, it's a subset of UTF-8
+        metadata = Metadata(mapping={'author': 'Mister Cafe',
+                                     'name': 'my.project',
+                                     'author': 'Cafe Junior',
+                                     'summary': 'Cafe torrefie',
+                                     'description': 'Hehehe'})
+        metadata.write(my_file)
 
-        for name, value in values:
-            metadata.set(name, value)
+        metadata2 = Metadata()
+        with open(my_file, encoding='utf-8') as fp:
+            metadata2.read_file(fp)
 
-        # we should have a certain amount of warnings
-        self.assertEqual(len(self.get_logs()), 2)
+    def test_metadata_read_write(self):
+        PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
+        metadata = Metadata(PKG_INFO)
+        out = StringIO()
+        metadata.write_file(out)
 
-    def test_multiple_predicates(self):
-        metadata = Metadata()
+        out.seek(0)
+        res = Metadata()
+        res.read_file(out)
+        self.assertEqual(metadata.values(), res.values())
 
-        # see for "3" instead of "3.0"  ???
-        # its seems like the MINOR VERSION can be omitted
-        metadata['Requires-Python'] = '>=2.6, <3.0'
-        metadata['Requires-Dist'] = ['Foo (>=2.6, <3.0)']
-
-        self.assertEqual([], self.get_logs(logging.WARNING))
-
-    def test_project_url(self):
-        metadata = Metadata()
-        metadata['Project-URL'] = [('one', 'http://ok')]
-        self.assertEqual(metadata['Project-URL'],
-                          [('one', 'http://ok')])
-        self.assertEqual(metadata['Metadata-Version'], '1.2')
+    ####  Test checks
 
     def test_check_version(self):
         metadata = Metadata()
@@ -238,38 +237,215 @@
         metadata['Requires-dist'] = ['Foo (a)']
         metadata['Obsoletes-dist'] = ['Foo (a)']
         metadata['Provides-dist'] = ['Foo (a)']
-        if metadata.docutils_support:
-            missing, warnings = metadata.check()
-            self.assertEqual(len(warnings), 4)
-            metadata.docutils_support = False
         missing, warnings = metadata.check()
         self.assertEqual(len(warnings), 4)
 
-    def test_best_choice(self):
-        metadata = Metadata()
-        metadata['Version'] = '1.0'
+    ####  Test fields and metadata versions
+
+    def test_metadata_versions(self):
+        metadata = Metadata(mapping={'name': 'project', 'version': '1.0'})
         self.assertEqual(metadata['Metadata-Version'],
                          PKG_INFO_PREFERRED_VERSION)
+        self.assertNotIn('Provides', metadata)
+        self.assertNotIn('Requires', metadata)
+        self.assertNotIn('Obsoletes', metadata)
+
         metadata['Classifier'] = ['ok']
+        self.assertEqual(metadata['Metadata-Version'], '1.1')
+
+        metadata = Metadata()
+        metadata['Download-URL'] = 'ok'
+        self.assertEqual(metadata['Metadata-Version'], '1.1')
+
+        metadata = Metadata()
+        metadata['Obsoletes'] = 'ok'
+        self.assertEqual(metadata['Metadata-Version'], '1.1')
+
+        del metadata['Obsoletes']
+        metadata['Obsoletes-Dist'] = 'ok'
         self.assertEqual(metadata['Metadata-Version'], '1.2')
 
-    def test_project_urls(self):
-        # project-url is a bit specific, make sure we write it
-        # properly in PKG-INFO
+        self.assertRaises(MetadataConflictError, metadata.set,
+                          'Obsoletes', 'ok')
+
+        del metadata['Obsoletes']
+        del metadata['Obsoletes-Dist']
+        metadata['Version'] = '1'
+        self.assertEqual(metadata['Metadata-Version'], '1.0')
+
+        # make sure the _best_version function works okay with
+        # non-conflicting fields from 1.1 and 1.2 (i.e. we want only the
+        # requires/requires-dist and co. pairs to cause a conflict, not all
+        # fields in _314_MARKERS)
         metadata = Metadata()
-        metadata['Version'] = '1.0'
-        metadata['Project-Url'] = [('one', 'http://ok')]
+        metadata['Requires-Python'] = '3'
+        metadata['Classifier'] = ['Programming language :: Python :: 3']
+        self.assertEqual(metadata['Metadata-Version'], '1.2')
+
+        PKG_INFO = os.path.join(os.path.dirname(__file__),
+                                'SETUPTOOLS-PKG-INFO')
+        metadata = Metadata(PKG_INFO)
+        self.assertEqual(metadata['Metadata-Version'], '1.0')
+
+        PKG_INFO = os.path.join(os.path.dirname(__file__),
+                                'SETUPTOOLS-PKG-INFO2')
+        metadata = Metadata(PKG_INFO)
+        self.assertEqual(metadata['Metadata-Version'], '1.1')
+
+        # Update the _fields dict directly to prevent 'Metadata-Version'
+        # from being updated by the _set_best_version() method.
+        metadata._fields['Metadata-Version'] = '1.618'
+        self.assertRaises(MetadataUnrecognizedVersionError, metadata.keys)
+
+    def test_version(self):
+        Metadata(mapping={'author': 'xxx',
+                          'name': 'xxx',
+                          'version': 'xxx',
+                          'home-page': 'xxxx'})
+        logs = self.get_logs(logging.WARNING)
+        self.assertEqual(1, len(logs))
+        self.assertIn('not a valid version', logs[0])
+
+    def test_description(self):
+        PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
+        with open(PKG_INFO, 'r', encoding='utf-8') as f:
+            content = f.read() % sys.platform
+        metadata = Metadata()
+        metadata.read_file(StringIO(content))
+
+        # see if we can read the description now
+        DESC = os.path.join(os.path.dirname(__file__), 'LONG_DESC.txt')
+        with open(DESC) as f:
+            wanted = f.read()
+        self.assertEqual(wanted, metadata['Description'])
+
+        # save the file somewhere and make sure we can read it back
+        out = StringIO()
+        metadata.write_file(out)
+        out.seek(0)
+
+        out.seek(0)
+        metadata = Metadata()
+        metadata.read_file(out)
+        self.assertEqual(wanted, metadata['Description'])
+
+    def test_description_folding(self):
+        # make sure the indentation is preserved
+        out = StringIO()
+        desc = dedent("""\
+        example::
+              We start here
+            and continue here
+          and end here.
+        """)
+
+        metadata = Metadata()
+        metadata['description'] = desc
+        metadata.write_file(out)
+
+        folded_desc = desc.replace('\n', '\n' + (7 * ' ') + '|')
+        self.assertIn(folded_desc, out.getvalue())
+
+    def test_project_url(self):
+        metadata = Metadata()
+        metadata['Project-URL'] = [('one', 'http://ok')]
+        self.assertEqual(metadata['Project-URL'], [('one', 'http://ok')])
+        self.assertEqual(metadata['Metadata-Version'], '1.2')
+
+        # make sure this particular field is handled properly when written
+        fp = StringIO()
+        metadata.write_file(fp)
+        self.assertIn('Project-URL: one,http://ok', fp.getvalue().split('\n'))
+
+        fp.seek(0)
+        metadata = Metadata()
+        metadata.read_file(fp)
         self.assertEqual(metadata['Project-Url'], [('one', 'http://ok')])
-        file_ = StringIO()
-        metadata.write_file(file_)
-        file_.seek(0)
-        res = file_.read().split('\n')
-        self.assertIn('Project-URL: one,http://ok', res)
 
-        file_.seek(0)
+    # TODO copy tests for v1.1 requires, obsoletes and provides from distutils
+    # (they're useless but we support them so we should test them anyway)
+
+    def test_provides_dist(self):
+        fields = {'name': 'project',
+                  'version': '1.0',
+                  'provides_dist': ['project', 'my.project']}
+        metadata = Metadata(mapping=fields)
+        self.assertEqual(metadata['Provides-Dist'],
+                         ['project', 'my.project'])
+        self.assertEqual(metadata['Metadata-Version'], '1.2', metadata)
+        self.assertNotIn('Requires', metadata)
+        self.assertNotIn('Obsoletes', metadata)
+
+    @unittest.skip('needs to be implemented')
+    def test_provides_illegal(self):
+        # TODO check the versions (like distutils does for old provides field)
+        self.assertRaises(ValueError, Metadata,
+                          mapping={'name': 'project',
+                                   'version': '1.0',
+                                   'provides_dist': ['my.pkg (splat)']})
+
+    def test_requires_dist(self):
+        fields = {'name': 'project',
+                  'version': '1.0',
+                  'requires_dist': ['other', 'another (==1.0)']}
+        metadata = Metadata(mapping=fields)
+        self.assertEqual(metadata['Requires-Dist'],
+                         ['other', 'another (==1.0)'])
+        self.assertEqual(metadata['Metadata-Version'], '1.2')
+        self.assertNotIn('Provides', metadata)
+        self.assertEqual(metadata['Requires-Dist'],
+                         ['other', 'another (==1.0)'])
+        self.assertNotIn('Obsoletes', metadata)
+
+        # make sure write_file uses one RFC 822 header per item
+        fp = StringIO()
+        metadata.write_file(fp)
+        lines = fp.getvalue().split('\n')
+        self.assertIn('Requires-Dist: other', lines)
+        self.assertIn('Requires-Dist: another (==1.0)', lines)
+
+        # test warnings for invalid version predicates
+        # XXX this would cause no warnings if we used update (or the mapping
+        # argument of the constructor), see comment in Metadata.update
         metadata = Metadata()
-        metadata.read_file(file_)
-        self.assertEqual(metadata['Project-Url'], [('one', 'http://ok')])
+        metadata['Requires-Dist'] = 'Funky (Groovie)'
+        metadata['Requires-Python'] = '1-4'
+        self.assertEqual(len(self.get_logs()), 2)
+
+        # test multiple version predicates
+        metadata = Metadata()
+
+        # XXX check PEP and see if 3 == 3.0
+        metadata['Requires-Python'] = '>=2.6, <3.0'
+        metadata['Requires-Dist'] = ['Foo (>=2.6, <3.0)']
+        self.assertEqual([], self.get_logs(logging.WARNING))
+
+    @unittest.skip('needs to be implemented')
+    def test_requires_illegal(self):
+        self.assertRaises(ValueError, Metadata,
+                          mapping={'name': 'project',
+                                   'version': '1.0',
+                                   'requires': ['my.pkg (splat)']})
+
+    def test_obsoletes_dist(self):
+        fields = {'name': 'project',
+                  'version': '1.0',
+                  'obsoletes_dist': ['other', 'another (<1.0)']}
+        metadata = Metadata(mapping=fields)
+        self.assertEqual(metadata['Obsoletes-Dist'],
+                         ['other', 'another (<1.0)'])
+        self.assertEqual(metadata['Metadata-Version'], '1.2')
+        self.assertNotIn('Provides', metadata)
+        self.assertNotIn('Requires', metadata)
+        self.assertEqual(metadata['Obsoletes-Dist'],
+                         ['other', 'another (<1.0)'])
+
+    @unittest.skip('needs to be implemented')
+    def test_obsoletes_illegal(self):
+        self.assertRaises(ValueError, Metadata,
+                          mapping={'name': 'project',
+                                   'version': '1.0',
+                                   'obsoletes': ['my.pkg (splat)']})
 
 
 def test_suite():
diff --git a/Lib/packaging/tests/test_run.py b/Lib/packaging/tests/test_run.py
--- a/Lib/packaging/tests/test_run.py
+++ b/Lib/packaging/tests/test_run.py
@@ -3,16 +3,16 @@
 import os
 import sys
 import shutil
-from tempfile import mkstemp
 from io import StringIO
 
 from packaging import install
 from packaging.tests import unittest, support, TESTFN
 from packaging.run import main
 
+from test.script_helper import assert_python_ok
+
 # setup script that uses __file__
 setup_using___file__ = """\
-
 __file__
 
 from packaging.run import setup
@@ -20,7 +20,6 @@
 """
 
 setup_prints_cwd = """\
-
 import os
 print os.getcwd()
 
@@ -29,11 +28,12 @@
 """
 
 
-class CoreTestCase(support.TempdirManager, support.LoggingCatcher,
-                   unittest.TestCase):
+class RunTestCase(support.TempdirManager,
+                  support.LoggingCatcher,
+                  unittest.TestCase):
 
     def setUp(self):
-        super(CoreTestCase, self).setUp()
+        super(RunTestCase, self).setUp()
         self.old_stdout = sys.stdout
         self.cleanup_testfn()
         self.old_argv = sys.argv, sys.argv[:]
@@ -43,7 +43,7 @@
         self.cleanup_testfn()
         sys.argv = self.old_argv[0]
         sys.argv[:] = self.old_argv[1]
-        super(CoreTestCase, self).tearDown()
+        super(RunTestCase, self).tearDown()
 
     def cleanup_testfn(self):
         path = TESTFN
@@ -77,9 +77,16 @@
             os.chmod(install_path, old_mod)
             install.get_path = old_get_path
 
+    def test_show_help(self):
+        # smoke test, just makes sure some help is displayed
+        status, out, err = assert_python_ok('-m', 'packaging.run', '--help')
+        self.assertEqual(status, 0)
+        self.assertGreater(out, b'')
+        self.assertEqual(err, b'')
+
 
 def test_suite():
-    return unittest.makeSuite(CoreTestCase)
+    return unittest.makeSuite(RunTestCase)
 
 if __name__ == "__main__":
     unittest.main(defaultTest="test_suite")
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -278,6 +278,10 @@
   ZLIB_RUNTIME_VERSION, in the zlib module. Patch by Torsten Landschoff.
 
 - Issue #12959: Add collections.ChainMap to collections.__all__.
+ 
+- Issue #8933: distutils' PKG-INFO files and packaging's METADATA files will
+  now correctly report Metadata-Version: 1.1 instead of 1.0 if a Classifier or
+  Download-URL field is present.
 
 - Issue #12567: Add curses.unget_wch() function. Push a character so the next
   get_wch() will return it.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:42:26 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:42:26 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge_3=2E2?=
Message-ID: <E1R38eQ-00060t-Dx@dinsdale.python.org>

http://hg.python.org/cpython/rev/76edde9b63ae
changeset:   72360:76edde9b63ae
parent:      72359:ad6b879eab02
parent:      72358:882d3b78b1cc
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 12 17:41:24 2011 +0200
summary:
  Merge 3.2

files:
  Lib/pydoc.py |  9 +++++----
  1 files changed, 5 insertions(+), 4 deletions(-)


diff --git a/Lib/pydoc.py b/Lib/pydoc.py
--- a/Lib/pydoc.py
+++ b/Lib/pydoc.py
@@ -1041,10 +1041,11 @@
         if docloc is not None:
             result = result + self.section('MODULE REFERENCE', docloc + """
 
-The following documentation is automatically generated from the Python source
-files.  It may be incomplete, incorrect or include features that are considered
-implementation detail and may vary between Python implementations.  When in
-doubt, consult the module reference at the location listed above.
+The following documentation is automatically generated from the Python
+source files.  It may be incomplete, incorrect or include features that
+are considered implementation detail and may vary between Python
+implementations.  When in doubt, consult the module reference at the
+location listed above.
 """)
 
         if desc:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:49:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:49:39 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Slight_cleanup_?=
 =?utf8?q?in_distutils_test=5Fdist=2E?=
Message-ID: <E1R38lP-0006fG-DY@dinsdale.python.org>

http://hg.python.org/cpython/rev/76399f965013
changeset:   72361:76399f965013
branch:      2.7
parent:      72319:f3bd49d07a53
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 10 05:37:33 2011 +0200
summary:
  Slight cleanup in distutils test_dist.

I have tests to add in this file and it?s always nice to start from a
clean base.  I?ve also changed a test that used to write an invalid
config file ('[global]command_packages = etc.' on one line), but the
test passes before and after this change, so either it magically works
or the test is poorly written.  Sigh.

files:
  Lib/distutils/tests/test_dist.py |  103 +++++++++---------
  1 files changed, 49 insertions(+), 54 deletions(-)


diff --git a/Lib/distutils/tests/test_dist.py b/Lib/distutils/tests/test_dist.py
--- a/Lib/distutils/tests/test_dist.py
+++ b/Lib/distutils/tests/test_dist.py
@@ -8,12 +8,13 @@
 import warnings
 import textwrap
 
-from distutils.dist import Distribution, fix_help_options, DistributionMetadata
+from distutils.dist import Distribution, fix_help_options
 from distutils.cmd import Command
 import distutils.dist
 from test.test_support import TESTFN, captured_stdout, run_unittest
 from distutils.tests import support
 
+
 class test_dist(Command):
     """Sample distutils extension command."""
 
@@ -61,7 +62,7 @@
 
     def test_debug_mode(self):
         with open(TESTFN, "w") as f:
-            f.write("[global]")
+            f.write("[global]\n")
             f.write("command_packages = foo.bar, splat")
 
         files = [TESTFN]
@@ -97,7 +98,7 @@
         self.assertEqual(d.get_command_packages(),
                          ["distutils.command", "foo.bar", "distutils.tests"])
         cmd = d.get_command_obj("test_dist")
-        self.assertTrue(isinstance(cmd, test_dist))
+        self.assertIsInstance(cmd, test_dist)
         self.assertEqual(cmd.sample_option, "sometext")
 
     def test_command_packages_configfile(self):
@@ -105,8 +106,8 @@
         self.addCleanup(os.unlink, TESTFN)
         f = open(TESTFN, "w")
         try:
-            print >>f, "[global]"
-            print >>f, "command_packages = foo.bar, splat"
+            print >> f, "[global]"
+            print >> f, "command_packages = foo.bar, splat"
         finally:
             f.close()
 
@@ -138,7 +139,6 @@
                             'description': u'Caf? torr?fi?',
                             'long_description': u'H?h?h?'})
 
-
         # let's make sure the file can be written
         # with Unicode fields. they are encoded with
         # PKG_INFO_ENCODING
@@ -152,33 +152,28 @@
                             'long_description': 'Hehehe'})
 
         my_file2 = os.path.join(tmp_dir, 'f2')
-        dist.metadata.write_pkg_file(open(my_file, 'w'))
+        dist.metadata.write_pkg_file(open(my_file2, 'w'))
 
     def test_empty_options(self):
         # an empty options dictionary should not stay in the
         # list of attributes
-        klass = Distribution
 
         # catching warnings
         warns = []
+
         def _warn(msg):
             warns.append(msg)
 
-        old_warn = warnings.warn
+        self.addCleanup(setattr, warnings, 'warn', warnings.warn)
         warnings.warn = _warn
-        try:
-            dist = klass(attrs={'author': 'xxx',
-                                'name': 'xxx',
-                                'version': 'xxx',
-                                'url': 'xxxx',
-                                'options': {}})
-        finally:
-            warnings.warn = old_warn
+        dist = Distribution(attrs={'author': 'xxx', 'name': 'xxx',
+                                   'version': 'xxx', 'url': 'xxxx',
+                                   'options': {}})
 
         self.assertEqual(len(warns), 0)
+        self.assertNotIn('options', dir(dist))
 
     def test_finalize_options(self):
-
         attrs = {'keywords': 'one,two',
                  'platforms': 'one,two'}
 
@@ -201,7 +196,6 @@
         cmds = dist.get_command_packages()
         self.assertEqual(cmds, ['distutils.command', 'one', 'two'])
 
-
     def test_announce(self):
         # make sure the level is known
         dist = Distribution()
@@ -251,15 +245,30 @@
         sys.argv[:] = self.argv[1]
         super(MetadataTestCase, self).tearDown()
 
+    def test_long_description(self):
+        long_desc = textwrap.dedent("""\
+        example::
+              We start here
+            and continue here
+          and end here.""")
+        attrs = {"name": "package",
+                 "version": "1.0",
+                 "long_description": long_desc}
+
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        meta = meta.replace('\n' + 8 * ' ', '\n')
+        self.assertIn(long_desc, meta)
+
     def test_simple_metadata(self):
         attrs = {"name": "package",
                  "version": "1.0"}
         dist = Distribution(attrs)
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.0" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.0", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertNotIn("requires:", meta.lower())
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_provides(self):
         attrs = {"name": "package",
@@ -271,9 +280,9 @@
         self.assertEqual(dist.get_provides(),
                          ["package", "package.sub"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("requires:", meta.lower())
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_provides_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -291,11 +300,11 @@
         self.assertEqual(dist.get_requires(),
                          ["other", "another (==1.0)"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("Requires: other" in meta)
-        self.assertTrue("Requires: another (==1.0)" in meta)
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertIn("Requires: other", meta)
+        self.assertIn("Requires: another (==1.0)", meta)
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_requires_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -313,11 +322,11 @@
         self.assertEqual(dist.get_obsoletes(),
                          ["other", "another (<1.0)"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("Obsoletes: other" in meta)
-        self.assertTrue("Obsoletes: another (<1.0)" in meta)
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertNotIn("requires:", meta.lower())
+        self.assertIn("Obsoletes: other", meta)
+        self.assertIn("Obsoletes: another (<1.0)", meta)
 
     def test_obsoletes_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -353,14 +362,14 @@
             if sys.platform in ('linux', 'darwin'):
                 os.environ['HOME'] = temp_dir
                 files = dist.find_config_files()
-                self.assertTrue(user_filename in files)
+                self.assertIn(user_filename, files)
 
             # win32-style
             if sys.platform == 'win32':
                 # home drive should be found
                 os.environ['HOME'] = temp_dir
                 files = dist.find_config_files()
-                self.assertTrue(user_filename in files,
+                self.assertIn(user_filename, files,
                              '%r not found in %r' % (user_filename, files))
         finally:
             os.remove(user_filename)
@@ -382,22 +391,7 @@
 
         output = [line for line in s.getvalue().split('\n')
                   if line.strip() != '']
-        self.assertTrue(len(output) > 0)
-
-    def test_long_description(self):
-        long_desc = textwrap.dedent("""\
-        example::
-              We start here
-            and continue here
-          and end here.""")
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "long_description": long_desc}
-
-        dist = distutils.dist.Distribution(attrs)
-        meta = self.format_metadata(dist)
-        meta = meta.replace('\n' + 8 * ' ', '\n')
-        self.assertTrue(long_desc in meta)
+        self.assertTrue(output)
 
     def test_read_metadata(self):
         attrs = {"name": "package",
@@ -426,6 +420,7 @@
         self.assertEqual(metadata.obsoletes, None)
         self.assertEqual(metadata.requires, ['foo'])
 
+
 def test_suite():
     suite = unittest.TestSuite()
     suite.addTest(unittest.makeSuite(DistributionTestCase))

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:49:40 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:49:40 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_determinati?=
 =?utf8?q?on_of_Metadata_version_=28=238933=29=2E__Patch_by_Filip_Gruszczy?=
 =?utf8?b?xYRza2ku?=
Message-ID: <E1R38lQ-0006fm-5J@dinsdale.python.org>

http://hg.python.org/cpython/rev/4b72d56cf42f
changeset:   72362:4b72d56cf42f
branch:      2.7
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 10 05:39:45 2011 +0200
summary:
  Fix determination of Metadata version (#8933).  Patch by Filip Gruszczy?ski.

files:
  Lib/distutils/dist.py            |   3 ++-
  Lib/distutils/tests/test_dist.py |  14 ++++++++++++++
  Misc/NEWS                        |   6 +++++-
  3 files changed, 21 insertions(+), 2 deletions(-)


diff --git a/Lib/distutils/dist.py b/Lib/distutils/dist.py
--- a/Lib/distutils/dist.py
+++ b/Lib/distutils/dist.py
@@ -1111,7 +1111,8 @@
         """Write the PKG-INFO format data to a file object.
         """
         version = '1.0'
-        if self.provides or self.requires or self.obsoletes:
+        if (self.provides or self.requires or self.obsoletes or
+            self.classifiers or self.download_url):
             version = '1.1'
 
         self._write_field(file, 'Metadata-Version', version)
diff --git a/Lib/distutils/tests/test_dist.py b/Lib/distutils/tests/test_dist.py
--- a/Lib/distutils/tests/test_dist.py
+++ b/Lib/distutils/tests/test_dist.py
@@ -245,6 +245,20 @@
         sys.argv[:] = self.argv[1]
         super(MetadataTestCase, self).tearDown()
 
+    def test_classifier(self):
+        attrs = {'name': 'Boa', 'version': '3.0',
+                 'classifiers': ['Programming Language :: Python :: 3']}
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        self.assertIn('Metadata-Version: 1.1', meta)
+
+    def test_download_url(self):
+        attrs = {'name': 'Boa', 'version': '3.0',
+                 'download_url': 'http://example.org/boa'}
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        self.assertIn('Metadata-Version: 1.1', meta)
+
     def test_long_description(self):
         long_desc = textwrap.dedent("""\
         example::
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -39,7 +39,11 @@
 
 Library
 -------
- 
+
+- Issue #8933: distutils' PKG-INFO files will now correctly report
+  Metadata-Version: 1.1 instead of 1.0 if a Classifier or Download-URL field is
+  present.
+
 - Issue #8286: The distutils command sdist will print a warning message instead
   of crashing when an invalid path is given in the manifest template.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 17:49:41 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 12 Sep 2011 17:49:41 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?=
	=?utf8?q?_Branch_merge?=
Message-ID: <E1R38lR-0006gk-0W@dinsdale.python.org>

http://hg.python.org/cpython/rev/18676f0118e4
changeset:   72363:18676f0118e4
branch:      2.7
parent:      72334:8812996e4062
parent:      72362:4b72d56cf42f
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 12 17:47:48 2011 +0200
summary:
  Branch merge

files:
  Lib/distutils/dist.py            |    3 +-
  Lib/distutils/tests/test_dist.py |  117 ++++++++++--------
  Misc/NEWS                        |    6 +-
  3 files changed, 70 insertions(+), 56 deletions(-)


diff --git a/Lib/distutils/dist.py b/Lib/distutils/dist.py
--- a/Lib/distutils/dist.py
+++ b/Lib/distutils/dist.py
@@ -1111,7 +1111,8 @@
         """Write the PKG-INFO format data to a file object.
         """
         version = '1.0'
-        if self.provides or self.requires or self.obsoletes:
+        if (self.provides or self.requires or self.obsoletes or
+            self.classifiers or self.download_url):
             version = '1.1'
 
         self._write_field(file, 'Metadata-Version', version)
diff --git a/Lib/distutils/tests/test_dist.py b/Lib/distutils/tests/test_dist.py
--- a/Lib/distutils/tests/test_dist.py
+++ b/Lib/distutils/tests/test_dist.py
@@ -8,12 +8,13 @@
 import warnings
 import textwrap
 
-from distutils.dist import Distribution, fix_help_options, DistributionMetadata
+from distutils.dist import Distribution, fix_help_options
 from distutils.cmd import Command
 import distutils.dist
 from test.test_support import TESTFN, captured_stdout, run_unittest
 from distutils.tests import support
 
+
 class test_dist(Command):
     """Sample distutils extension command."""
 
@@ -61,7 +62,7 @@
 
     def test_debug_mode(self):
         with open(TESTFN, "w") as f:
-            f.write("[global]")
+            f.write("[global]\n")
             f.write("command_packages = foo.bar, splat")
 
         files = [TESTFN]
@@ -97,7 +98,7 @@
         self.assertEqual(d.get_command_packages(),
                          ["distutils.command", "foo.bar", "distutils.tests"])
         cmd = d.get_command_obj("test_dist")
-        self.assertTrue(isinstance(cmd, test_dist))
+        self.assertIsInstance(cmd, test_dist)
         self.assertEqual(cmd.sample_option, "sometext")
 
     def test_command_packages_configfile(self):
@@ -105,8 +106,8 @@
         self.addCleanup(os.unlink, TESTFN)
         f = open(TESTFN, "w")
         try:
-            print >>f, "[global]"
-            print >>f, "command_packages = foo.bar, splat"
+            print >> f, "[global]"
+            print >> f, "command_packages = foo.bar, splat"
         finally:
             f.close()
 
@@ -138,7 +139,6 @@
                             'description': u'Caf? torr?fi?',
                             'long_description': u'H?h?h?'})
 
-
         # let's make sure the file can be written
         # with Unicode fields. they are encoded with
         # PKG_INFO_ENCODING
@@ -152,33 +152,28 @@
                             'long_description': 'Hehehe'})
 
         my_file2 = os.path.join(tmp_dir, 'f2')
-        dist.metadata.write_pkg_file(open(my_file, 'w'))
+        dist.metadata.write_pkg_file(open(my_file2, 'w'))
 
     def test_empty_options(self):
         # an empty options dictionary should not stay in the
         # list of attributes
-        klass = Distribution
 
         # catching warnings
         warns = []
+
         def _warn(msg):
             warns.append(msg)
 
-        old_warn = warnings.warn
+        self.addCleanup(setattr, warnings, 'warn', warnings.warn)
         warnings.warn = _warn
-        try:
-            dist = klass(attrs={'author': 'xxx',
-                                'name': 'xxx',
-                                'version': 'xxx',
-                                'url': 'xxxx',
-                                'options': {}})
-        finally:
-            warnings.warn = old_warn
+        dist = Distribution(attrs={'author': 'xxx', 'name': 'xxx',
+                                   'version': 'xxx', 'url': 'xxxx',
+                                   'options': {}})
 
         self.assertEqual(len(warns), 0)
+        self.assertNotIn('options', dir(dist))
 
     def test_finalize_options(self):
-
         attrs = {'keywords': 'one,two',
                  'platforms': 'one,two'}
 
@@ -201,7 +196,6 @@
         cmds = dist.get_command_packages()
         self.assertEqual(cmds, ['distutils.command', 'one', 'two'])
 
-
     def test_announce(self):
         # make sure the level is known
         dist = Distribution()
@@ -251,15 +245,44 @@
         sys.argv[:] = self.argv[1]
         super(MetadataTestCase, self).tearDown()
 
+    def test_classifier(self):
+        attrs = {'name': 'Boa', 'version': '3.0',
+                 'classifiers': ['Programming Language :: Python :: 3']}
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        self.assertIn('Metadata-Version: 1.1', meta)
+
+    def test_download_url(self):
+        attrs = {'name': 'Boa', 'version': '3.0',
+                 'download_url': 'http://example.org/boa'}
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        self.assertIn('Metadata-Version: 1.1', meta)
+
+    def test_long_description(self):
+        long_desc = textwrap.dedent("""\
+        example::
+              We start here
+            and continue here
+          and end here.""")
+        attrs = {"name": "package",
+                 "version": "1.0",
+                 "long_description": long_desc}
+
+        dist = Distribution(attrs)
+        meta = self.format_metadata(dist)
+        meta = meta.replace('\n' + 8 * ' ', '\n')
+        self.assertIn(long_desc, meta)
+
     def test_simple_metadata(self):
         attrs = {"name": "package",
                  "version": "1.0"}
         dist = Distribution(attrs)
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.0" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.0", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertNotIn("requires:", meta.lower())
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_provides(self):
         attrs = {"name": "package",
@@ -271,9 +294,9 @@
         self.assertEqual(dist.get_provides(),
                          ["package", "package.sub"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("requires:", meta.lower())
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_provides_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -291,11 +314,11 @@
         self.assertEqual(dist.get_requires(),
                          ["other", "another (==1.0)"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("Requires: other" in meta)
-        self.assertTrue("Requires: another (==1.0)" in meta)
-        self.assertTrue("obsoletes:" not in meta.lower())
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertIn("Requires: other", meta)
+        self.assertIn("Requires: another (==1.0)", meta)
+        self.assertNotIn("obsoletes:", meta.lower())
 
     def test_requires_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -313,11 +336,11 @@
         self.assertEqual(dist.get_obsoletes(),
                          ["other", "another (<1.0)"])
         meta = self.format_metadata(dist)
-        self.assertTrue("Metadata-Version: 1.1" in meta)
-        self.assertTrue("provides:" not in meta.lower())
-        self.assertTrue("requires:" not in meta.lower())
-        self.assertTrue("Obsoletes: other" in meta)
-        self.assertTrue("Obsoletes: another (<1.0)" in meta)
+        self.assertIn("Metadata-Version: 1.1", meta)
+        self.assertNotIn("provides:", meta.lower())
+        self.assertNotIn("requires:", meta.lower())
+        self.assertIn("Obsoletes: other", meta)
+        self.assertIn("Obsoletes: another (<1.0)", meta)
 
     def test_obsoletes_illegal(self):
         self.assertRaises(ValueError, Distribution,
@@ -353,14 +376,14 @@
             if sys.platform in ('linux', 'darwin'):
                 os.environ['HOME'] = temp_dir
                 files = dist.find_config_files()
-                self.assertTrue(user_filename in files)
+                self.assertIn(user_filename, files)
 
             # win32-style
             if sys.platform == 'win32':
                 # home drive should be found
                 os.environ['HOME'] = temp_dir
                 files = dist.find_config_files()
-                self.assertTrue(user_filename in files,
+                self.assertIn(user_filename, files,
                              '%r not found in %r' % (user_filename, files))
         finally:
             os.remove(user_filename)
@@ -382,22 +405,7 @@
 
         output = [line for line in s.getvalue().split('\n')
                   if line.strip() != '']
-        self.assertTrue(len(output) > 0)
-
-    def test_long_description(self):
-        long_desc = textwrap.dedent("""\
-        example::
-              We start here
-            and continue here
-          and end here.""")
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "long_description": long_desc}
-
-        dist = distutils.dist.Distribution(attrs)
-        meta = self.format_metadata(dist)
-        meta = meta.replace('\n' + 8 * ' ', '\n')
-        self.assertTrue(long_desc in meta)
+        self.assertTrue(output)
 
     def test_read_metadata(self):
         attrs = {"name": "package",
@@ -426,6 +434,7 @@
         self.assertEqual(metadata.obsoletes, None)
         self.assertEqual(metadata.requires, ['foo'])
 
+
 def test_suite():
     suite = unittest.TestSuite()
     suite.addTest(unittest.makeSuite(DistributionTestCase))
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -39,7 +39,11 @@
 
 Library
 -------
- 
+
+- Issue #8933: distutils' PKG-INFO files will now correctly report
+  Metadata-Version: 1.1 instead of 1.0 if a Classifier or Download-URL field is
+  present.
+
 - Issue #8286: The distutils command sdist will print a warning message instead
   of crashing when an invalid path is given in the manifest template.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 19:13:45 2011
From: python-checkins at python.org (alexis.metaireau)
Date: Mon, 12 Sep 2011 19:13:45 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Factor_out_the_distribut?=
 =?utf8?q?ion_file-system_safe_name_functions_from?=
Message-ID: <E1R3A4n-0006iG-7d@dinsdale.python.org>

http://hg.python.org/distutils2/rev/d2d36e2c95cf
changeset:   1134:d2d36e2c95cf
user:        Jeremy Kloth <jeremy.kloth at gmail.com>
date:        Tue Sep 06 13:10:31 2011 -0600
summary:
  Factor out the distribution file-system safe name functions from install_distinfo to allow all metadata consumers access to them.

files:
  distutils2/command/install_distinfo.py |  32 +-------------
  distutils2/dist.py                     |   4 +-
  distutils2/metadata.py                 |  16 +++++-
  3 files changed, 16 insertions(+), 36 deletions(-)


diff --git a/distutils2/command/install_distinfo.py b/distutils2/command/install_distinfo.py
--- a/distutils2/command/install_distinfo.py
+++ b/distutils2/command/install_distinfo.py
@@ -66,9 +66,7 @@
 
         metadata = self.distribution.metadata
 
-        basename = "%s-%s.dist-info" % (
-            to_filename(safe_name(metadata['Name'])),
-            to_filename(safe_version(metadata['Version'])))
+        basename = metadata.get_fullname(filesafe=True) + ".dist-info"
 
         self.distinfo_dir = os.path.join(self.distinfo_dir, basename)
         self.outputs = []
@@ -152,31 +150,3 @@
 
     def get_outputs(self):
         return self.outputs
-
-
-# The following functions are taken from setuptools' pkg_resources module.
-
-def safe_name(name):
-    """Convert an arbitrary string to a standard distribution name
-
-    Any runs of non-alphanumeric/. characters are replaced with a single '-'.
-    """
-    return re.sub('[^A-Za-z0-9.]+', '-', name)
-
-
-def safe_version(version):
-    """Convert an arbitrary string to a standard version string
-
-    Spaces become dots, and all other non-alphanumeric characters become
-    dashes, with runs of multiple dashes condensed to a single dash.
-    """
-    version = version.replace(' ', '.')
-    return re.sub('[^A-Za-z0-9.]+', '-', version)
-
-
-def to_filename(name):
-    """Convert a project or version name to its filename-escaped form
-
-    Any '-' characters are currently replaced with '_'.
-    """
-    return name.replace('-', '_')
diff --git a/distutils2/dist.py b/distutils2/dist.py
--- a/distutils2/dist.py
+++ b/distutils2/dist.py
@@ -275,8 +275,8 @@
             d = self.command_options[command] = {}
         return d
 
-    def get_fullname(self):
-        return self.metadata.get_fullname()
+    def get_fullname(self, filesafe=False):
+        return self.metadata.get_fullname(filesafe)
 
     def dump_option_dicts(self, header=None, commands=None, indent=""):
         from pprint import pformat
diff --git a/distutils2/metadata.py b/distutils2/metadata.py
--- a/distutils2/metadata.py
+++ b/distutils2/metadata.py
@@ -182,6 +182,7 @@
 
 _MISSING = object()
 
+_FILESAFE = re.compile('[^A-Za-z0-9.]+')
 
 class Metadata(object):
     """The metadata of a release.
@@ -285,9 +286,18 @@
     #
     # Public API
     #
-    def get_fullname(self):
-        """Return the distribution name with version"""
-        return '%s-%s' % (self['Name'], self['Version'])
+    def get_fullname(self, filesafe=False):
+        """Return the distribution name with version.
+
+        If filesafe is true, return a filename-escaped form."""
+        name, version = self['Name'], self['Version']
+        if filesafe:
+            # For both name and version any runs of non-alphanumeric or '.'
+            # characters are replaced with a single '-'.  Additionally any
+            # spaces in the version string become '.'
+            name = _FILESAFE.sub('-', name)
+            version = _FILESAFE.sub('-', version.replace(' ', '.'))
+        return '%s-%s' % (name, version)
 
     def is_metadata_field(self, name):
         """return True if name is a valid metadata key"""

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 12 19:13:45 2011
From: python-checkins at python.org (alexis.metaireau)
Date: Mon, 12 Sep 2011 19:13:45 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fixes_for_Python_2=2E4_c?=
	=?utf8?q?ompatibility?=
Message-ID: <E1R3A4n-0006iX-B5@dinsdale.python.org>

http://hg.python.org/distutils2/rev/b5852bdcb91c
changeset:   1137:b5852bdcb91c
user:        Jeremy Kloth <jeremy.kloth at gmail.com>
date:        Mon Sep 12 05:33:18 2011 -0600
summary:
  Fixes for Python 2.4 compatibility

files:
  distutils2/dist.py |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/distutils2/dist.py b/distutils2/dist.py
--- a/distutils2/dist.py
+++ b/distutils2/dist.py
@@ -61,7 +61,7 @@
 
     # 'common_usage' is a short (2-3 line) string describing the common
     # usage of the setup script.
-    common_usage = u"""\
+    common_usage = """\
 Common commands: (see '--help-commands' for more)
 
   pysetup run build      will build the package underneath 'build/'
@@ -546,7 +546,7 @@
         # we ignore "foo bar").
         if self.help_commands:
             self.print_commands()
-            print()
+            print(u'')
             print(gen_usage(self.script_name))
             return 1
 
@@ -614,7 +614,7 @@
                                 "Standard commands",
                                 max_length)
         if extra_commands:
-            print()
+            print(u'')
             self.print_command_list(extra_commands,
                                     "Extra commands",
                                     max_length)

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 12 19:13:45 2011
From: python-checkins at python.org (alexis.metaireau)
Date: Mon, 12 Sep 2011 19:13:45 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fixes_to_actually_use_th?=
	=?utf8?q?e_backports=2E?=
Message-ID: <E1R3A4n-0006iU-AT@dinsdale.python.org>

http://hg.python.org/distutils2/rev/7811ada4da3a
changeset:   1136:7811ada4da3a
user:        Jeremy Kloth <jeremy.kloth at gmail.com>
date:        Mon Sep 12 05:10:57 2011 -0600
summary:
  Fixes to actually use the backports.

files:
  distutils2/command/build_ext.py                |  2 +-
  distutils2/command/build_scripts.py            |  2 +-
  distutils2/command/install_data.py             |  2 +-
  distutils2/command/install_dist.py             |  6 +++---
  distutils2/compiler/__init__.py                |  2 +-
  distutils2/compiler/cygwinccompiler.py         |  2 +-
  distutils2/compiler/unixccompiler.py           |  2 +-
  distutils2/create.py                           |  6 +++---
  distutils2/database.py                         |  4 ++--
  distutils2/tests/test_command_build_ext.py     |  2 +-
  distutils2/tests/test_command_build_scripts.py |  2 +-
  distutils2/tests/test_command_install_data.py  |  4 ++--
  distutils2/tests/test_create.py                |  2 +-
  distutils2/tests/test_cygwinccompiler.py       |  2 +-
  distutils2/tests/test_dist.py                  |  2 +-
  distutils2/tests/test_unixccompiler.py         |  2 +-
  distutils2/util.py                             |  8 ++++----
  17 files changed, 26 insertions(+), 26 deletions(-)


diff --git a/distutils2/command/build_ext.py b/distutils2/command/build_ext.py
--- a/distutils2/command/build_ext.py
+++ b/distutils2/command/build_ext.py
@@ -8,8 +8,8 @@
 import re
 import sys
 import logging
-import sysconfig
 
+from distutils2._backport import sysconfig
 from distutils2.util import get_platform
 from distutils2.command.cmd import Command
 from distutils2.errors import (CCompilerError, CompileError, PackagingError,
diff --git a/distutils2/command/build_scripts.py b/distutils2/command/build_scripts.py
--- a/distutils2/command/build_scripts.py
+++ b/distutils2/command/build_scripts.py
@@ -2,8 +2,8 @@
 
 import os
 import re
-import sysconfig
 
+from distutils2._backport import sysconfig
 from distutils2.command.cmd import Command
 from distutils2.util import convert_path, newer, detect_encoding, fsencode
 from distutils2 import logger
diff --git a/distutils2/command/install_data.py b/distutils2/command/install_data.py
--- a/distutils2/command/install_data.py
+++ b/distutils2/command/install_data.py
@@ -4,7 +4,7 @@
 
 import os, sys
 from shutil import Error
-from sysconfig import get_paths, format_value
+from distutils2._backport.sysconfig import get_paths, format_value
 from distutils2 import logger
 from distutils2.util import convert_path
 from distutils2.command.cmd import Command
diff --git a/distutils2/command/install_dist.py b/distutils2/command/install_dist.py
--- a/distutils2/command/install_dist.py
+++ b/distutils2/command/install_dist.py
@@ -3,9 +3,9 @@
 import sys
 import os
 
-import sysconfig
-from sysconfig import get_config_vars, get_paths, get_path, get_config_var
-
+from distutils2._backport import sysconfig
+from distutils2._backport.sysconfig import (get_config_vars, get_paths,
+                                            get_path, get_config_var)
 from distutils2 import logger
 from distutils2.command.cmd import Command
 from distutils2.errors import PackagingPlatformError
diff --git a/distutils2/compiler/__init__.py b/distutils2/compiler/__init__.py
--- a/distutils2/compiler/__init__.py
+++ b/distutils2/compiler/__init__.py
@@ -16,8 +16,8 @@
 import os
 import sys
 import re
-import sysconfig
 
+from distutils2._backport import sysconfig
 from distutils2.util import resolve_name
 from distutils2.errors import PackagingPlatformError
 from distutils2 import logger
diff --git a/distutils2/compiler/cygwinccompiler.py b/distutils2/compiler/cygwinccompiler.py
--- a/distutils2/compiler/cygwinccompiler.py
+++ b/distutils2/compiler/cygwinccompiler.py
@@ -54,7 +54,7 @@
 from distutils2.util import write_file
 from distutils2.errors import PackagingExecError, CompileError, UnknownFileError
 from distutils2.util import get_compiler_versions
-import sysconfig
+from distutils2._backport import sysconfig
 
 
 def get_msvcr():
diff --git a/distutils2/compiler/unixccompiler.py b/distutils2/compiler/unixccompiler.py
--- a/distutils2/compiler/unixccompiler.py
+++ b/distutils2/compiler/unixccompiler.py
@@ -21,7 +21,7 @@
 from distutils2.errors import (PackagingExecError, CompileError,
                                LibError, LinkError)
 from distutils2 import logger
-import sysconfig
+from distutils2._backport import sysconfig
 
 
 # XXX Things not currently handled:
diff --git a/distutils2/create.py b/distutils2/create.py
--- a/distutils2/create.py
+++ b/distutils2/create.py
@@ -25,13 +25,13 @@
 import sys
 import glob
 import shutil
-import sysconfig
+from distutils2._backport import sysconfig
 if 'any' not in dir(__builtins__):
     from distutils2._backport import any
 try:
     from hashlib import md5
-except ImportError:
-    from distutils2._backport.hashlib import md5
+except ImportError: #<2.5
+    from md5 import md5
 from textwrap import dedent
 from distutils2.util import cmp_to_key, detect_encoding
 from ConfigParser import RawConfigParser
diff --git a/distutils2/database.py b/distutils2/database.py
--- a/distutils2/database.py
+++ b/distutils2/database.py
@@ -8,8 +8,8 @@
 import zipimport
 try:
     from hashlib import md5
-except ImportError:
-    from _backport.hashlib import md5
+except ImportError: #<2.5
+    from md5 import md5
 from distutils2 import logger
 from distutils2.errors import PackagingError
 from distutils2.version import suggest_normalized_version, VersionPredicate
diff --git a/distutils2/tests/test_command_build_ext.py b/distutils2/tests/test_command_build_ext.py
--- a/distutils2/tests/test_command_build_ext.py
+++ b/distutils2/tests/test_command_build_ext.py
@@ -2,9 +2,9 @@
 import sys
 import site
 import shutil
-import sysconfig
 import textwrap
 from StringIO import StringIO
+from distutils2._backport import sysconfig
 from distutils2.dist import Distribution
 from distutils2.errors import (UnknownFileError, CompileError,
                               PackagingPlatformError)
diff --git a/distutils2/tests/test_command_build_scripts.py b/distutils2/tests/test_command_build_scripts.py
--- a/distutils2/tests/test_command_build_scripts.py
+++ b/distutils2/tests/test_command_build_scripts.py
@@ -2,7 +2,7 @@
 
 import os
 import sys
-import sysconfig
+from distutils2._backport import sysconfig
 from distutils2.dist import Distribution
 from distutils2.command.build_scripts import build_scripts
 from distutils2.tests import unittest, support
diff --git a/distutils2/tests/test_command_install_data.py b/distutils2/tests/test_command_install_data.py
--- a/distutils2/tests/test_command_install_data.py
+++ b/distutils2/tests/test_command_install_data.py
@@ -1,7 +1,7 @@
 """Tests for distutils2.command.install_data."""
 import os
-import sysconfig
-from sysconfig import _get_default_scheme
+from distutils2._backport import sysconfig
+from distutils2._backport.sysconfig import _get_default_scheme
 from distutils2.tests import unittest, support
 from distutils2.command.install_data import install_data
 
diff --git a/distutils2/tests/test_create.py b/distutils2/tests/test_create.py
--- a/distutils2/tests/test_create.py
+++ b/distutils2/tests/test_create.py
@@ -3,8 +3,8 @@
 import codecs
 import os
 import sys
-import sysconfig
 from textwrap import dedent
+from distutils2._backport import sysconfig
 from distutils2.create import MainProgram, ask_yn, ask, main
 
 from distutils2.tests import support, unittest
diff --git a/distutils2/tests/test_cygwinccompiler.py b/distutils2/tests/test_cygwinccompiler.py
--- a/distutils2/tests/test_cygwinccompiler.py
+++ b/distutils2/tests/test_cygwinccompiler.py
@@ -1,7 +1,7 @@
 """Tests for distutils2.cygwinccompiler."""
 import os
 import sys
-import sysconfig
+from distutils2._backport import sysconfig
 from distutils2.compiler.cygwinccompiler import (
     check_config_h, get_msvcr,
     CONFIG_H_OK, CONFIG_H_NOTOK, CONFIG_H_UNCERTAIN)
diff --git a/distutils2/tests/test_dist.py b/distutils2/tests/test_dist.py
--- a/distutils2/tests/test_dist.py
+++ b/distutils2/tests/test_dist.py
@@ -4,7 +4,7 @@
 import sys
 import logging
 import textwrap
-import sysconfig
+from distutils2._backport import sysconfig
 import distutils2.dist
 from StringIO import StringIO
 
diff --git a/distutils2/tests/test_unixccompiler.py b/distutils2/tests/test_unixccompiler.py
--- a/distutils2/tests/test_unixccompiler.py
+++ b/distutils2/tests/test_unixccompiler.py
@@ -1,7 +1,7 @@
 """Tests for distutils2.unixccompiler."""
 import sys
 
-import sysconfig
+from distutils2._backport import sysconfig
 from distutils2.compiler.unixccompiler import UnixCCompiler
 from distutils2.tests import unittest
 
diff --git a/distutils2/util.py b/distutils2/util.py
--- a/distutils2/util.py
+++ b/distutils2/util.py
@@ -9,14 +9,13 @@
 import shutil
 import string
 try:
-    import hashlib
+    from hashlib import md5
 except ImportError: #<2.5
-    from _backport import hashlib
+    from md5 import md5
 import tarfile
 import zipfile
 import posixpath
 import subprocess
-import sysconfig
 try:
     from glob import iglob as std_iglob
 except ImportError:#<2.5
@@ -29,6 +28,7 @@
 from distutils2.errors import (PackagingPlatformError, PackagingFileError,
                               PackagingByteCompileError, PackagingExecError,
                               InstallationException, PackagingInternalError)
+from distutils2._backport import sysconfig
 
 _PLATFORM = None
 _DEFAULT_INSTALLER = 'distutils2'
@@ -1212,7 +1212,7 @@
             # do not put size and md5 hash, as in PEP-376
             writer.writerow((fpath, '', ''))
         else:
-            hash = hashlib.md5()
+            hash = md5()
             fp = open(fpath, 'rb')
             hash.update(fp.read())
             fp.close()

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 12 19:13:45 2011
From: python-checkins at python.org (alexis.metaireau)
Date: Mon, 12 Sep 2011 19:13:45 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Isolate_imports_for_each?=
 =?utf8?q?_action_to_allow_=60pysetup=60_to_run_from_source_with?=
Message-ID: <E1R3A4n-0006ib-BZ@dinsdale.python.org>

http://hg.python.org/distutils2/rev/824e4aca701b
changeset:   1138:824e4aca701b
user:        Jeremy Kloth <jeremy.kloth at gmail.com>
date:        Mon Sep 12 05:36:18 2011 -0600
summary:
  Isolate imports for each action to allow `pysetup` to run from source with Python 2.4

files:
  distutils2/run.py |  19 ++++++++++---------
  1 files changed, 10 insertions(+), 9 deletions(-)


diff --git a/distutils2/run.py b/distutils2/run.py
--- a/distutils2/run.py
+++ b/distutils2/run.py
@@ -10,9 +10,7 @@
 from distutils2.dist import Distribution
 from distutils2.util import _is_archive_file, generate_setup_py
 from distutils2.command import get_command_class, STANDARD_COMMANDS
-from distutils2.install import install, install_local_project, remove
 from distutils2.database import get_distribution, get_distributions
-from distutils2.depgraph import generate_graph
 from distutils2.fancy_getopt import FancyGetopt
 from distutils2.errors import (PackagingArgError, PackagingError,
                               PackagingModuleError, PackagingClassError,
@@ -219,6 +217,7 @@
 
 @action_help(graph_usage)
 def _graph(dispatcher, args, **kw):
+    from distutils2.depgraph import generate_graph
     name = args[1]
     dist = get_distribution(name, use_egg_info=True)
     if dist is None:
@@ -232,6 +231,7 @@
 
 @action_help(install_usage)
 def _install(dispatcher, args, **kw):
+    from distutils2.install import install, install_local_project
     # first check if we are in a source directory
     if len(args) < 2:
         # are we inside a project dir?
@@ -290,6 +290,7 @@
 
 @action_help(remove_usage)
 def _remove(distpatcher, args, **kw):
+    from distutils2.install import remove
     opts = _parse_args(args[1:], 'y', [])
     if 'y' in opts:
         auto_confirm = True
@@ -574,17 +575,17 @@
         from distutils2.command.cmd import Command
 
         print('Usage: pysetup [options] action [action_options]')
-        print()
+        print(u'')
         if global_options_:
             self.print_usage(self.parser)
-            print()
+            print(u'')
 
         if display_options_:
             parser.set_option_table(display_options)
             parser.print_help(
                 "Information display options (just display " +
                 "information, ignore any commands)")
-            print()
+            print(u'')
 
         for command in commands:
             if isinstance(command, type) and issubclass(command, Command):
@@ -598,7 +599,7 @@
                 parser.set_option_table(cls.user_options)
 
             parser.print_help("Options for %r command:" % cls.__name__)
-            print()
+            print(u'')
 
     def _show_command_help(self, command):
         if isinstance(command, basestring):
@@ -606,7 +607,7 @@
 
         desc = getattr(command, 'description', '(no description available)')
         print('Description:', desc)
-        print()
+        print(u'')
 
         if (hasattr(command, 'help_options') and
             isinstance(command.help_options, list)):
@@ -616,7 +617,7 @@
             self.parser.set_option_table(command.user_options)
 
         self.parser.print_help("Options:")
-        print()
+        print(u'')
 
     def _get_command_groups(self):
         """Helper function to retrieve all the command class names divided
@@ -643,7 +644,7 @@
 
         self.print_command_list(std_commands, "Standard commands", max_length)
         if extra_commands:
-            print()
+            print(u'')
             self.print_command_list(extra_commands, "Extra commands",
                                     max_length)
 

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 12 19:13:45 2011
From: python-checkins at python.org (alexis.metaireau)
Date: Mon, 12 Sep 2011 19:13:45 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Keep_backported_files_to?=
	=?utf8?q?gether?=
Message-ID: <E1R3A4n-0006iQ-94@dinsdale.python.org>

http://hg.python.org/distutils2/rev/6ccdebb210ca
changeset:   1135:6ccdebb210ca
user:        Jeremy Kloth <jeremy.kloth at gmail.com>
date:        Mon Sep 12 04:58:39 2011 -0600
summary:
  Keep backported files together

files:
  distutils2/_backport/sysconfig.cfg |   18 +-
  distutils2/_backport/sysconfig.py  |  147 +-
  sysconfig.cfg                      |  111 --
  sysconfig.py                       |  764 -----------------
  4 files changed, 97 insertions(+), 943 deletions(-)


diff --git a/distutils2/_backport/sysconfig.cfg b/distutils2/_backport/sysconfig.cfg
--- a/distutils2/_backport/sysconfig.cfg
+++ b/distutils2/_backport/sysconfig.cfg
@@ -1,24 +1,24 @@
 [globals]
-# These are the useful categories that are sometimes referenced at runtime,
-# using pkgutil.open():
+# These are useful categories that can be referenced at run time,
+# using packaging.database.get_file.
 # Configuration files
 config = {confdir}/{distribution.name}
 # Non-writable data that is independent of architecture (images, many xml/text files)
 appdata = {datadir}/{distribution.name}
 # Non-writable data that is architecture-dependent (some binary data formats)
 appdata.arch = {libdir}/{distribution.name}
-# Data, written by the package, that must be preserved (databases)
+# Data, written by the app/lib, that must be preserved (databases)
 appdata.persistent = {statedir}/lib/{distribution.name}
-# Data, written by the package, that can be safely discarded (cache)
+# Data, written by the app/lib, that can be safely discarded (cache)
 appdata.disposable = {statedir}/cache/{distribution.name}
-# Help or documentation files referenced at runtime
+# Help or documentation files
 help = {datadir}/{distribution.name}
 icon = {datadir}/pixmaps
 scripts = {base}/bin
 
 # Non-runtime files.  These are valid categories for marking files for
-# install, but they should not be referenced by the app at runtime:
-# Help or documentation files not referenced by the package at runtime
+# install, but they should not be referenced by the app/lib at run time.
+# Help or documentation files
 doc = {datadir}/doc/{distribution.name}
 # GNU info documentation files
 info = {datadir}/info
@@ -40,8 +40,8 @@
 platstdlib = {platbase}/lib/python{py_version_short}
 purelib = {base}/lib/python{py_version_short}/site-packages
 platlib = {platbase}/lib/python{py_version_short}/site-packages
-include = {base}/include/python{py_version_short}
-platinclude = {platbase}/include/python{py_version_short}
+include = {base}/include/python{py_version_short}{abiflags}
+platinclude = {platbase}/include/python{py_version_short}{abiflags}
 data = {base}
 
 [posix_home]
diff --git a/distutils2/_backport/sysconfig.py b/distutils2/_backport/sysconfig.py
--- a/distutils2/_backport/sysconfig.py
+++ b/distutils2/_backport/sysconfig.py
@@ -1,12 +1,24 @@
-"""Provide access to Python's configuration information."""
+"""Access to Python's configuration information."""
+
 import os
+import re
 import sys
-import re
 from os.path import pardir, realpath
 from ConfigParser import RawConfigParser
 
-_PREFIX = os.path.normpath(sys.prefix)
-_EXEC_PREFIX = os.path.normpath(sys.exec_prefix)
+__all__ = [
+    'get_config_h_filename',
+    'get_config_var',
+    'get_config_vars',
+    'get_makefile_filename',
+    'get_path',
+    'get_path_names',
+    'get_paths',
+    'get_platform',
+    'get_python_version',
+    'get_scheme_names',
+    'parse_config_h',
+]
 
 # let's read the configuration file
 # XXX _CONFIG_DIR will be set by the Makefile later
@@ -49,26 +61,38 @@
 
 _expand_globals(_SCHEMES)
 
+ # FIXME don't rely on sys.version here, its format is an implementatin detail
+ # of CPython, use sys.version_info or sys.hexversion
 _PY_VERSION = sys.version.split()[0]
 _PY_VERSION_SHORT = sys.version[:3]
 _PY_VERSION_SHORT_NO_DOT = _PY_VERSION[0] + _PY_VERSION[2]
+_PREFIX = os.path.normpath(sys.prefix)
+_EXEC_PREFIX = os.path.normpath(sys.exec_prefix)
 _CONFIG_VARS = None
 _USER_BASE = None
+
+
+def _safe_realpath(path):
+    try:
+        return realpath(path)
+    except OSError:
+        return path
+
 if sys.executable:
-    _PROJECT_BASE = os.path.dirname(realpath(sys.executable))
+    _PROJECT_BASE = os.path.dirname(_safe_realpath(sys.executable))
 else:
     # sys.executable can be empty if argv[0] has been changed and Python is
     # unable to retrieve the real program name
-    _PROJECT_BASE = realpath(os.getcwd())
+    _PROJECT_BASE = _safe_realpath(os.getcwd())
 
 if os.name == "nt" and "pcbuild" in _PROJECT_BASE[-8:].lower():
-    _PROJECT_BASE = realpath(os.path.join(_PROJECT_BASE, pardir))
+    _PROJECT_BASE = _safe_realpath(os.path.join(_PROJECT_BASE, pardir))
 # PC/VS7.1
 if os.name == "nt" and "\\pc\\v" in _PROJECT_BASE[-10:].lower():
-    _PROJECT_BASE = realpath(os.path.join(_PROJECT_BASE, pardir, pardir))
+    _PROJECT_BASE = _safe_realpath(os.path.join(_PROJECT_BASE, pardir, pardir))
 # PC/AMD64
 if os.name == "nt" and "\\pcbuild\\amd64" in _PROJECT_BASE[-14:].lower():
-    _PROJECT_BASE = realpath(os.path.join(_PROJECT_BASE, pardir, pardir))
+    _PROJECT_BASE = _safe_realpath(os.path.join(_PROJECT_BASE, pardir, pardir))
 
 
 def is_python_build():
@@ -102,8 +126,9 @@
 
 
 def _extend_dict(target_dict, other_dict):
-    for key, value in other_dict.iteritems():
-        if key in target_dict:
+    target_keys = target_dict.keys()
+    for key, value in other_dict.items():
+        if key in target_keys:
             continue
         target_dict[key] = value
 
@@ -120,14 +145,15 @@
         res[key] = os.path.normpath(_subst_vars(value, vars))
     return res
 
+
 def format_value(value, vars):
     def _replacer(matchobj):
-         name = matchobj.group(1)
-         if name in vars:
-             return vars[name]
-         return matchobj.group(0)
+        name = matchobj.group(1)
+        if name in vars:
+            return vars[name]
+        return matchobj.group(0)
     return _VAR_REPL.sub(_replacer, value)
- 
+
 
 def _get_default_scheme():
     if os.name == 'posix':
@@ -236,7 +262,8 @@
                     item = os.environ[n]
 
                 elif n in renamed_variables:
-                    if name.startswith('PY_') and name[3:] in renamed_variables:
+                    if (name.startswith('PY_') and
+                        name[3:] in renamed_variables):
                         item = ""
 
                     elif 'PY_' + n in notdone:
@@ -270,9 +297,16 @@
                                 done[name] = value
 
             else:
-                # bogus variable reference; just drop it since we can't deal
+                # bogus variable reference (e.g. "prefix=$/opt/python");
+                # just drop it since we can't deal
+                done[name] = value
                 variables.remove(name)
 
+    # strip spurious spaces
+    for k, v in done.items():
+        if isinstance(v, str):
+            done[k] = v.strip()
+
     # save the results in the global dictionary
     vars.update(done)
     return vars
@@ -282,7 +316,11 @@
     """Return the path of the Makefile."""
     if _PYTHON_BUILD:
         return os.path.join(_PROJECT_BASE, "Makefile")
-    return os.path.join(get_path('stdlib'), "config", "Makefile")
+    if hasattr(sys, 'abiflags'):
+        config_dir_name = 'config-%s%s' % (_PY_VERSION_SHORT, sys.abiflags)
+    else:
+        config_dir_name = 'config'
+    return os.path.join(get_path('stdlib'), config_dir_name, 'Makefile')
 
 
 def _init_posix(vars):
@@ -296,32 +334,17 @@
         if hasattr(e, "strerror"):
             msg = msg + " (%s)" % e.strerror
         raise IOError(msg)
-
     # load the installed pyconfig.h:
     config_h = get_config_h_filename()
     try:
-        parse_config_h(open(config_h), vars)
+        f = open(config_h)
+        parse_config_h(f, vars)
+        f.close()
     except IOError, e:
         msg = "invalid Python installation: unable to open %s" % config_h
         if hasattr(e, "strerror"):
             msg = msg + " (%s)" % e.strerror
         raise IOError(msg)
-
-    # On MacOSX we need to check the setting of the environment variable
-    # MACOSX_DEPLOYMENT_TARGET: configure bases some choices on it so
-    # it needs to be compatible.
-    # If it isn't set we set it to the configure-time value
-    if sys.platform == 'darwin' and 'MACOSX_DEPLOYMENT_TARGET' in vars:
-        cfg_target = vars['MACOSX_DEPLOYMENT_TARGET']
-        cur_target = os.getenv('MACOSX_DEPLOYMENT_TARGET', '')
-        if cur_target == '':
-            cur_target = cfg_target
-            os.putenv('MACOSX_DEPLOYMENT_TARGET', cfg_target)
-        elif map(int, cfg_target.split('.')) > map(int, cur_target.split('.')):
-            msg = ('$MACOSX_DEPLOYMENT_TARGET mismatch: now "%s" but "%s" '
-                   'during configure' % (cur_target, cfg_target))
-            raise IOError(msg)
-
     # On AIX, there are wrong paths to the linker scripts in the Makefile
     # -- these paths are relative to the Python source, but when installed
     # the scripts are in another directory.
@@ -338,7 +361,7 @@
     vars['SO'] = '.pyd'
     vars['EXE'] = '.exe'
     vars['VERSION'] = _PY_VERSION_SHORT_NO_DOT
-    vars['BINDIR'] = os.path.dirname(realpath(sys.executable))
+    vars['BINDIR'] = os.path.dirname(_safe_realpath(sys.executable))
 
 #
 # public APIs
@@ -434,7 +457,7 @@
         _CONFIG_VARS = {}
         # Normalized versions of prefix and exec_prefix are handy to have;
         # in fact, these are the standard versions used most places in the
-        # Distutils.
+        # distutils2 module.
         _CONFIG_VARS['prefix'] = _PREFIX
         _CONFIG_VARS['exec_prefix'] = _EXEC_PREFIX
         _CONFIG_VARS['py_version'] = _PY_VERSION
@@ -443,12 +466,16 @@
         _CONFIG_VARS['base'] = _PREFIX
         _CONFIG_VARS['platbase'] = _EXEC_PREFIX
         _CONFIG_VARS['projectbase'] = _PROJECT_BASE
+        try:
+            _CONFIG_VARS['abiflags'] = sys.abiflags
+        except AttributeError:
+            # sys.abiflags may not be defined on all platforms.
+            _CONFIG_VARS['abiflags'] = ''
 
         if os.name in ('nt', 'os2'):
             _init_non_posix(_CONFIG_VARS)
         if os.name == 'posix':
             _init_posix(_CONFIG_VARS)
-
         # Setting 'userbase' is done below the call to the
         # init function to enable using 'get_config_var' in
         # the init-function.
@@ -458,7 +485,7 @@
         if 'srcdir' not in _CONFIG_VARS:
             _CONFIG_VARS['srcdir'] = _PROJECT_BASE
         else:
-            _CONFIG_VARS['srcdir'] = realpath(_CONFIG_VARS['srcdir'])
+            _CONFIG_VARS['srcdir'] = _safe_realpath(_CONFIG_VARS['srcdir'])
 
         # Convert srcdir into an absolute path if it appears necessary.
         # Normally it is relative to the build directory.  However, during
@@ -466,8 +493,12 @@
         # from a different directory.
         if _PYTHON_BUILD and os.name == "posix":
             base = _PROJECT_BASE
+            try:
+                cwd = os.getcwd()
+            except OSError:
+                cwd = None
             if (not os.path.isabs(_CONFIG_VARS['srcdir']) and
-                base != os.getcwd()):
+                base != cwd):
                 # srcdir is relative and we are not in the same directory
                 # as the executable. Assume executable is in the build
                 # directory and make srcdir absolute.
@@ -583,7 +614,7 @@
         if i == -1:
             return sys.platform
         j = sys.version.find(")", i)
-        look = sys.version[i + len(prefix):j].lower()
+        look = sys.version[i+len(prefix):j].lower()
         if look == 'amd64':
             return 'win-amd64'
         if look == 'itanium':
@@ -632,11 +663,9 @@
         # machine is going to compile and link as if it were
         # MACOSX_DEPLOYMENT_TARGET.
         cfgvars = get_config_vars()
-        macver = os.environ.get('MACOSX_DEPLOYMENT_TARGET')
-        if not macver:
-            macver = cfgvars.get('MACOSX_DEPLOYMENT_TARGET')
+        macver = cfgvars.get('MACOSX_DEPLOYMENT_TARGET')
 
-        if 1:
+        if True:
             # Always calculate the release of the running machine,
             # needed to determine if we can build fat binaries or not.
 
@@ -667,8 +696,8 @@
             release = macver
             osname = "macosx"
 
-            if (macrelease + '.') >= '10.4.' and \
-                    '-arch' in get_config_vars().get('CFLAGS', '').strip():
+            if ((macrelease + '.') >= '10.4.' and
+                '-arch' in get_config_vars().get('CFLAGS', '').strip()):
                 # The universal build will build fat binaries, but not on
                 # systems before 10.4
                 #
@@ -701,13 +730,13 @@
                 # On OSX the machine type returned by uname is always the
                 # 32-bit variant, even if the executable architecture is
                 # the 64-bit variant
-                if sys.maxint >= (2 ** 32):
+                if sys.maxsize >= 2**32:
                     machine = 'x86_64'
 
             elif machine in ('PowerPC', 'Power_Macintosh'):
                 # Pick a sane name for the PPC architecture.
                 # See 'i386' case
-                if sys.maxint >= (2 ** 32):
+                if sys.maxsize >= 2**32:
                     machine = 'ppc64'
                 else:
                     machine = 'ppc'
@@ -720,20 +749,20 @@
 
 
 def _print_dict(title, data):
-    for index, (key, value) in enumerate(sorted(data.iteritems())):
+    for index, (key, value) in enumerate(sorted(data.items())):
         if index == 0:
-            print '%s: ' % (title)
-        print '\t%s = "%s"' % (key, value)
+            print('%s: ' % (title))
+        print('\t%s = "%s"' % (key, value))
 
 
 def _main():
     """Display all information sysconfig detains."""
-    print 'Platform: "%s"' % get_platform()
-    print 'Python version: "%s"' % get_python_version()
-    print 'Current installation scheme: "%s"' % _get_default_scheme()
-    print ''
+    print('Platform: "%s"' % get_platform())
+    print('Python version: "%s"' % get_python_version())
+    print('Current installation scheme: "%s"' % _get_default_scheme())
+    print(u'')
     _print_dict('Paths', get_paths())
-    print
+    print(u'')
     _print_dict('Variables', get_config_vars())
 
 
diff --git a/sysconfig.cfg b/sysconfig.cfg
deleted file mode 100644
--- a/sysconfig.cfg
+++ /dev/null
@@ -1,111 +0,0 @@
-[globals]
-# These are useful categories that can be referenced at run time,
-# using packaging.database.get_file.
-# Configuration files
-config = {confdir}/{distribution.name}
-# Non-writable data that is independent of architecture (images, many xml/text files)
-appdata = {datadir}/{distribution.name}
-# Non-writable data that is architecture-dependent (some binary data formats)
-appdata.arch = {libdir}/{distribution.name}
-# Data, written by the app/lib, that must be preserved (databases)
-appdata.persistent = {statedir}/lib/{distribution.name}
-# Data, written by the app/lib, that can be safely discarded (cache)
-appdata.disposable = {statedir}/cache/{distribution.name}
-# Help or documentation files
-help = {datadir}/{distribution.name}
-icon = {datadir}/pixmaps
-scripts = {base}/bin
-
-# Non-runtime files.  These are valid categories for marking files for
-# install, but they should not be referenced by the app/lib at run time.
-# Help or documentation files
-doc = {datadir}/doc/{distribution.name}
-# GNU info documentation files
-info = {datadir}/info
-# man pages
-man = {datadir}/man
-
-[posix_prefix]
-# Configuration directories.  Some of these come straight out of the
-# configure script.  They are for implementing the other variables, not to
-# be used directly in [resource_locations].
-confdir = /etc
-datadir = /usr/share
-libdir = /usr/lib  ; or /usr/lib64 on a multilib system
-statedir = /var
-# User resource directory
-local = ~/.local/{distribution.name}
-
-stdlib = {base}/lib/python{py_version_short}
-platstdlib = {platbase}/lib/python{py_version_short}
-purelib = {base}/lib/python{py_version_short}/site-packages
-platlib = {platbase}/lib/python{py_version_short}/site-packages
-include = {base}/include/python{py_version_short}{abiflags}
-platinclude = {platbase}/include/python{py_version_short}{abiflags}
-data = {base}
-
-[posix_home]
-stdlib = {base}/lib/python
-platstdlib = {base}/lib/python
-purelib = {base}/lib/python
-platlib = {base}/lib/python
-include = {base}/include/python
-platinclude = {base}/include/python
-scripts = {base}/bin
-data = {base}
-
-[nt]
-stdlib = {base}/Lib
-platstdlib = {base}/Lib
-purelib = {base}/Lib/site-packages
-platlib = {base}/Lib/site-packages
-include = {base}/Include
-platinclude = {base}/Include
-scripts = {base}/Scripts
-data = {base}
-
-[os2]
-stdlib = {base}/Lib
-platstdlib = {base}/Lib
-purelib = {base}/Lib/site-packages
-platlib = {base}/Lib/site-packages
-include = {base}/Include
-platinclude = {base}/Include
-scripts = {base}/Scripts
-data = {base}
-
-[os2_home]
-stdlib = {userbase}/lib/python{py_version_short}
-platstdlib = {userbase}/lib/python{py_version_short}
-purelib = {userbase}/lib/python{py_version_short}/site-packages
-platlib = {userbase}/lib/python{py_version_short}/site-packages
-include = {userbase}/include/python{py_version_short}
-scripts = {userbase}/bin
-data = {userbase}
-
-[nt_user]
-stdlib = {userbase}/Python{py_version_nodot}
-platstdlib = {userbase}/Python{py_version_nodot}
-purelib = {userbase}/Python{py_version_nodot}/site-packages
-platlib = {userbase}/Python{py_version_nodot}/site-packages
-include = {userbase}/Python{py_version_nodot}/Include
-scripts = {userbase}/Scripts
-data = {userbase}
-
-[posix_user]
-stdlib = {userbase}/lib/python{py_version_short}
-platstdlib = {userbase}/lib/python{py_version_short}
-purelib = {userbase}/lib/python{py_version_short}/site-packages
-platlib = {userbase}/lib/python{py_version_short}/site-packages
-include = {userbase}/include/python{py_version_short}
-scripts = {userbase}/bin
-data = {userbase}
-
-[osx_framework_user]
-stdlib = {userbase}/lib/python
-platstdlib = {userbase}/lib/python
-purelib = {userbase}/lib/python/site-packages
-platlib = {userbase}/lib/python/site-packages
-include = {userbase}/include
-scripts = {userbase}/bin
-data = {userbase}
diff --git a/sysconfig.py b/sysconfig.py
deleted file mode 100644
--- a/sysconfig.py
+++ /dev/null
@@ -1,764 +0,0 @@
-"""Access to Python's configuration information."""
-
-import os
-import re
-import sys
-from os.path import pardir, realpath
-from ConfigParser import RawConfigParser
-
-__all__ = [
-    'get_config_h_filename',
-    'get_config_var',
-    'get_config_vars',
-    'get_makefile_filename',
-    'get_path',
-    'get_path_names',
-    'get_paths',
-    'get_platform',
-    'get_python_version',
-    'get_scheme_names',
-    'parse_config_h',
-]
-
-# let's read the configuration file
-# XXX _CONFIG_DIR will be set by the Makefile later
-_CONFIG_DIR = os.path.normpath(os.path.dirname(__file__))
-_CONFIG_FILE = os.path.join(_CONFIG_DIR, 'sysconfig.cfg')
-_SCHEMES = RawConfigParser()
-_SCHEMES.read(_CONFIG_FILE)
-_VAR_REPL = re.compile(r'\{([^{]*?)\}')
-
-
-def _expand_globals(config):
-    if config.has_section('globals'):
-        globals = config.items('globals')
-    else:
-        globals = tuple()
-
-    sections = config.sections()
-    for section in sections:
-        if section == 'globals':
-            continue
-        for option, value in globals:
-            if config.has_option(section, option):
-                continue
-            config.set(section, option, value)
-    config.remove_section('globals')
-
-    # now expanding local variables defined in the cfg file
-    #
-    for section in config.sections():
-        variables = dict(config.items(section))
-
-        def _replacer(matchobj):
-            name = matchobj.group(1)
-            if name in variables:
-                return variables[name]
-            return matchobj.group(0)
-
-        for option, value in config.items(section):
-            config.set(section, option, _VAR_REPL.sub(_replacer, value))
-
-_expand_globals(_SCHEMES)
-
- # FIXME don't rely on sys.version here, its format is an implementatin detail
- # of CPython, use sys.version_info or sys.hexversion
-_PY_VERSION = sys.version.split()[0]
-_PY_VERSION_SHORT = sys.version[:3]
-_PY_VERSION_SHORT_NO_DOT = _PY_VERSION[0] + _PY_VERSION[2]
-_PREFIX = os.path.normpath(sys.prefix)
-_EXEC_PREFIX = os.path.normpath(sys.exec_prefix)
-_CONFIG_VARS = None
-_USER_BASE = None
-
-
-def _safe_realpath(path):
-    try:
-        return realpath(path)
-    except OSError:
-        return path
-
-if sys.executable:
-    _PROJECT_BASE = os.path.dirname(_safe_realpath(sys.executable))
-else:
-    # sys.executable can be empty if argv[0] has been changed and Python is
-    # unable to retrieve the real program name
-    _PROJECT_BASE = _safe_realpath(os.getcwd())
-
-if os.name == "nt" and "pcbuild" in _PROJECT_BASE[-8:].lower():
-    _PROJECT_BASE = _safe_realpath(os.path.join(_PROJECT_BASE, pardir))
-# PC/VS7.1
-if os.name == "nt" and "\\pc\\v" in _PROJECT_BASE[-10:].lower():
-    _PROJECT_BASE = _safe_realpath(os.path.join(_PROJECT_BASE, pardir, pardir))
-# PC/AMD64
-if os.name == "nt" and "\\pcbuild\\amd64" in _PROJECT_BASE[-14:].lower():
-    _PROJECT_BASE = _safe_realpath(os.path.join(_PROJECT_BASE, pardir, pardir))
-
-
-def is_python_build():
-    for fn in ("Setup.dist", "Setup.local"):
-        if os.path.isfile(os.path.join(_PROJECT_BASE, "Modules", fn)):
-            return True
-    return False
-
-_PYTHON_BUILD = is_python_build()
-
-if _PYTHON_BUILD:
-    for scheme in ('posix_prefix', 'posix_home'):
-        _SCHEMES.set(scheme, 'include', '{srcdir}/Include')
-        _SCHEMES.set(scheme, 'platinclude', '{projectbase}/.')
-
-
-def _subst_vars(path, local_vars):
-    """In the string `path`, replace tokens like {some.thing} with the
-    corresponding value from the map `local_vars`.
-
-    If there is no corresponding value, leave the token unchanged.
-    """
-    def _replacer(matchobj):
-        name = matchobj.group(1)
-        if name in local_vars:
-            return local_vars[name]
-        elif name in os.environ:
-            return os.environ[name]
-        return matchobj.group(0)
-    return _VAR_REPL.sub(_replacer, path)
-
-
-def _extend_dict(target_dict, other_dict):
-    target_keys = target_dict.keys()
-    for key, value in other_dict.items():
-        if key in target_keys:
-            continue
-        target_dict[key] = value
-
-
-def _expand_vars(scheme, vars):
-    res = {}
-    if vars is None:
-        vars = {}
-    _extend_dict(vars, get_config_vars())
-
-    for key, value in _SCHEMES.items(scheme):
-        if os.name in ('posix', 'nt'):
-            value = os.path.expanduser(value)
-        res[key] = os.path.normpath(_subst_vars(value, vars))
-    return res
-
-
-def format_value(value, vars):
-    def _replacer(matchobj):
-        name = matchobj.group(1)
-        if name in vars:
-            return vars[name]
-        return matchobj.group(0)
-    return _VAR_REPL.sub(_replacer, value)
-
-
-def _get_default_scheme():
-    if os.name == 'posix':
-        # the default scheme for posix is posix_prefix
-        return 'posix_prefix'
-    return os.name
-
-
-def _getuserbase():
-    env_base = os.environ.get("PYTHONUSERBASE", None)
-
-    def joinuser(*args):
-        return os.path.expanduser(os.path.join(*args))
-
-    # what about 'os2emx', 'riscos' ?
-    if os.name == "nt":
-        base = os.environ.get("APPDATA") or "~"
-        if env_base:
-            return env_base
-        else:
-            return joinuser(base, "Python")
-
-    if sys.platform == "darwin":
-        framework = get_config_var("PYTHONFRAMEWORK")
-        if framework:
-            if env_base:
-                return env_base
-            else:
-                return joinuser("~", "Library", framework, "%d.%d" %
-                                sys.version_info[:2])
-
-    if env_base:
-        return env_base
-    else:
-        return joinuser("~", ".local")
-
-
-def _parse_makefile(filename, vars=None):
-    """Parse a Makefile-style file.
-
-    A dictionary containing name/value pairs is returned.  If an
-    optional dictionary is passed in as the second argument, it is
-    used instead of a new dictionary.
-    """
-    # Regexes needed for parsing Makefile (and similar syntaxes,
-    # like old-style Setup files).
-    _variable_rx = re.compile("([a-zA-Z][a-zA-Z0-9_]+)\s*=\s*(.*)")
-    _findvar1_rx = re.compile(r"\$\(([A-Za-z][A-Za-z0-9_]*)\)")
-    _findvar2_rx = re.compile(r"\${([A-Za-z][A-Za-z0-9_]*)}")
-
-    if vars is None:
-        vars = {}
-    done = {}
-    notdone = {}
-
-    for line in open(filename).readlines():
-        if line.startswith('#') or line.strip() == '':
-            continue
-        m = _variable_rx.match(line)
-        if m:
-            n, v = m.group(1, 2)
-            v = v.strip()
-            # `$$' is a literal `$' in make
-            tmpv = v.replace('$$', '')
-
-            if "$" in tmpv:
-                notdone[n] = v
-            else:
-                try:
-                    v = int(v)
-                except ValueError:
-                    # insert literal `$'
-                    done[n] = v.replace('$$', '$')
-                else:
-                    done[n] = v
-
-    # do variable interpolation here
-    variables = list(notdone.keys())
-
-    # Variables with a 'PY_' prefix in the makefile. These need to
-    # be made available without that prefix through sysconfig.
-    # Special care is needed to ensure that variable expansion works, even
-    # if the expansion uses the name without a prefix.
-    renamed_variables = ('CFLAGS', 'LDFLAGS', 'CPPFLAGS')
-
-    while len(variables) > 0:
-        for name in tuple(variables):
-            value = notdone[name]
-            m = _findvar1_rx.search(value) or _findvar2_rx.search(value)
-            if m is not None:
-                n = m.group(1)
-                found = True
-                if n in done:
-                    item = str(done[n])
-                elif n in notdone:
-                    # get it on a subsequent round
-                    found = False
-                elif n in os.environ:
-                    # do it like make: fall back to environment
-                    item = os.environ[n]
-
-                elif n in renamed_variables:
-                    if (name.startswith('PY_') and
-                        name[3:] in renamed_variables):
-                        item = ""
-
-                    elif 'PY_' + n in notdone:
-                        found = False
-
-                    else:
-                        item = str(done['PY_' + n])
-
-                else:
-                    done[n] = item = ""
-
-                if found:
-                    after = value[m.end():]
-                    value = value[:m.start()] + item + after
-                    if "$" in after:
-                        notdone[name] = value
-                    else:
-                        try:
-                            value = int(value)
-                        except ValueError:
-                            done[name] = value.strip()
-                        else:
-                            done[name] = value
-                        variables.remove(name)
-
-                        if name.startswith('PY_') \
-                        and name[3:] in renamed_variables:
-
-                            name = name[3:]
-                            if name not in done:
-                                done[name] = value
-
-            else:
-                # bogus variable reference (e.g. "prefix=$/opt/python");
-                # just drop it since we can't deal
-                done[name] = value
-                variables.remove(name)
-
-    # strip spurious spaces
-    for k, v in done.items():
-        if isinstance(v, str):
-            done[k] = v.strip()
-
-    # save the results in the global dictionary
-    vars.update(done)
-    return vars
-
-
-def get_makefile_filename():
-    """Return the path of the Makefile."""
-    if _PYTHON_BUILD:
-        return os.path.join(_PROJECT_BASE, "Makefile")
-    if hasattr(sys, 'abiflags'):
-        config_dir_name = 'config-%s%s' % (_PY_VERSION_SHORT, sys.abiflags)
-    else:
-        config_dir_name = 'config'
-    return os.path.join(get_path('stdlib'), config_dir_name, 'Makefile')
-
-
-def _init_posix(vars):
-    """Initialize the module as appropriate for POSIX systems."""
-    # load the installed Makefile:
-    makefile = get_makefile_filename()
-    try:
-        _parse_makefile(makefile, vars)
-    except IOError, e:
-        msg = "invalid Python installation: unable to open %s" % makefile
-        if hasattr(e, "strerror"):
-            msg = msg + " (%s)" % e.strerror
-        raise IOError(msg)
-    # load the installed pyconfig.h:
-    config_h = get_config_h_filename()
-    try:
-        f = open(config_h)
-        parse_config_h(f, vars)
-        f.close()
-    except IOError, e:
-        msg = "invalid Python installation: unable to open %s" % config_h
-        if hasattr(e, "strerror"):
-            msg = msg + " (%s)" % e.strerror
-        raise IOError(msg)
-    # On AIX, there are wrong paths to the linker scripts in the Makefile
-    # -- these paths are relative to the Python source, but when installed
-    # the scripts are in another directory.
-    if _PYTHON_BUILD:
-        vars['LDSHARED'] = vars['BLDSHARED']
-
-
-def _init_non_posix(vars):
-    """Initialize the module as appropriate for NT"""
-    # set basic install directories
-    vars['LIBDEST'] = get_path('stdlib')
-    vars['BINLIBDEST'] = get_path('platstdlib')
-    vars['INCLUDEPY'] = get_path('include')
-    vars['SO'] = '.pyd'
-    vars['EXE'] = '.exe'
-    vars['VERSION'] = _PY_VERSION_SHORT_NO_DOT
-    vars['BINDIR'] = os.path.dirname(_safe_realpath(sys.executable))
-
-#
-# public APIs
-#
-
-
-def parse_config_h(fp, vars=None):
-    """Parse a config.h-style file.
-
-    A dictionary containing name/value pairs is returned.  If an
-    optional dictionary is passed in as the second argument, it is
-    used instead of a new dictionary.
-    """
-    if vars is None:
-        vars = {}
-    define_rx = re.compile("#define ([A-Z][A-Za-z0-9_]+) (.*)\n")
-    undef_rx = re.compile("/[*] #undef ([A-Z][A-Za-z0-9_]+) [*]/\n")
-
-    while True:
-        line = fp.readline()
-        if not line:
-            break
-        m = define_rx.match(line)
-        if m:
-            n, v = m.group(1, 2)
-            try:
-                v = int(v)
-            except ValueError:
-                pass
-            vars[n] = v
-        else:
-            m = undef_rx.match(line)
-            if m:
-                vars[m.group(1)] = 0
-    return vars
-
-
-def get_config_h_filename():
-    """Return the path of pyconfig.h."""
-    if _PYTHON_BUILD:
-        if os.name == "nt":
-            inc_dir = os.path.join(_PROJECT_BASE, "PC")
-        else:
-            inc_dir = _PROJECT_BASE
-    else:
-        inc_dir = get_path('platinclude')
-    return os.path.join(inc_dir, 'pyconfig.h')
-
-
-def get_scheme_names():
-    """Return a tuple containing the schemes names."""
-    return tuple(sorted(_SCHEMES.sections()))
-
-
-def get_path_names():
-    """Return a tuple containing the paths names."""
-    # xxx see if we want a static list
-    return _SCHEMES.options('posix_prefix')
-
-
-def get_paths(scheme=_get_default_scheme(), vars=None, expand=True):
-    """Return a mapping containing an install scheme.
-
-    ``scheme`` is the install scheme name. If not provided, it will
-    return the default scheme for the current platform.
-    """
-    if expand:
-        return _expand_vars(scheme, vars)
-    else:
-        return dict(_SCHEMES.items(scheme))
-
-
-def get_path(name, scheme=_get_default_scheme(), vars=None, expand=True):
-    """Return a path corresponding to the scheme.
-
-    ``scheme`` is the install scheme name.
-    """
-    return get_paths(scheme, vars, expand)[name]
-
-
-def get_config_vars(*args):
-    """With no arguments, return a dictionary of all configuration
-    variables relevant for the current platform.
-
-    On Unix, this means every variable defined in Python's installed Makefile;
-    On Windows and Mac OS it's a much smaller set.
-
-    With arguments, return a list of values that result from looking up
-    each argument in the configuration variable dictionary.
-    """
-    global _CONFIG_VARS
-    if _CONFIG_VARS is None:
-        _CONFIG_VARS = {}
-        # Normalized versions of prefix and exec_prefix are handy to have;
-        # in fact, these are the standard versions used most places in the
-        # distutils2 module.
-        _CONFIG_VARS['prefix'] = _PREFIX
-        _CONFIG_VARS['exec_prefix'] = _EXEC_PREFIX
-        _CONFIG_VARS['py_version'] = _PY_VERSION
-        _CONFIG_VARS['py_version_short'] = _PY_VERSION_SHORT
-        _CONFIG_VARS['py_version_nodot'] = _PY_VERSION[0] + _PY_VERSION[2]
-        _CONFIG_VARS['base'] = _PREFIX
-        _CONFIG_VARS['platbase'] = _EXEC_PREFIX
-        _CONFIG_VARS['projectbase'] = _PROJECT_BASE
-        try:
-            _CONFIG_VARS['abiflags'] = sys.abiflags
-        except AttributeError:
-            # sys.abiflags may not be defined on all platforms.
-            _CONFIG_VARS['abiflags'] = ''
-
-        if os.name in ('nt', 'os2'):
-            _init_non_posix(_CONFIG_VARS)
-        if os.name == 'posix':
-            _init_posix(_CONFIG_VARS)
-        # Setting 'userbase' is done below the call to the
-        # init function to enable using 'get_config_var' in
-        # the init-function.
-        if sys.version >= '2.6':
-            _CONFIG_VARS['userbase'] = _getuserbase()
-
-        if 'srcdir' not in _CONFIG_VARS:
-            _CONFIG_VARS['srcdir'] = _PROJECT_BASE
-        else:
-            _CONFIG_VARS['srcdir'] = _safe_realpath(_CONFIG_VARS['srcdir'])
-
-        # Convert srcdir into an absolute path if it appears necessary.
-        # Normally it is relative to the build directory.  However, during
-        # testing, for example, we might be running a non-installed python
-        # from a different directory.
-        if _PYTHON_BUILD and os.name == "posix":
-            base = _PROJECT_BASE
-            try:
-                cwd = os.getcwd()
-            except OSError:
-                cwd = None
-            if (not os.path.isabs(_CONFIG_VARS['srcdir']) and
-                base != cwd):
-                # srcdir is relative and we are not in the same directory
-                # as the executable. Assume executable is in the build
-                # directory and make srcdir absolute.
-                srcdir = os.path.join(base, _CONFIG_VARS['srcdir'])
-                _CONFIG_VARS['srcdir'] = os.path.normpath(srcdir)
-
-        if sys.platform == 'darwin':
-            kernel_version = os.uname()[2]  # Kernel version (8.4.3)
-            major_version = int(kernel_version.split('.')[0])
-
-            if major_version < 8:
-                # On Mac OS X before 10.4, check if -arch and -isysroot
-                # are in CFLAGS or LDFLAGS and remove them if they are.
-                # This is needed when building extensions on a 10.3 system
-                # using a universal build of python.
-                for key in ('LDFLAGS', 'BASECFLAGS',
-                        # a number of derived variables. These need to be
-                        # patched up as well.
-                        'CFLAGS', 'PY_CFLAGS', 'BLDSHARED'):
-                    flags = _CONFIG_VARS[key]
-                    flags = re.sub('-arch\s+\w+\s', ' ', flags)
-                    flags = re.sub('-isysroot [^ \t]*', ' ', flags)
-                    _CONFIG_VARS[key] = flags
-            else:
-                # Allow the user to override the architecture flags using
-                # an environment variable.
-                # NOTE: This name was introduced by Apple in OSX 10.5 and
-                # is used by several scripting languages distributed with
-                # that OS release.
-                if 'ARCHFLAGS' in os.environ:
-                    arch = os.environ['ARCHFLAGS']
-                    for key in ('LDFLAGS', 'BASECFLAGS',
-                        # a number of derived variables. These need to be
-                        # patched up as well.
-                        'CFLAGS', 'PY_CFLAGS', 'BLDSHARED'):
-
-                        flags = _CONFIG_VARS[key]
-                        flags = re.sub('-arch\s+\w+\s', ' ', flags)
-                        flags = flags + ' ' + arch
-                        _CONFIG_VARS[key] = flags
-
-                # If we're on OSX 10.5 or later and the user tries to
-                # compiles an extension using an SDK that is not present
-                # on the current machine it is better to not use an SDK
-                # than to fail.
-                #
-                # The major usecase for this is users using a Python.org
-                # binary installer  on OSX 10.6: that installer uses
-                # the 10.4u SDK, but that SDK is not installed by default
-                # when you install Xcode.
-                #
-                CFLAGS = _CONFIG_VARS.get('CFLAGS', '')
-                m = re.search('-isysroot\s+(\S+)', CFLAGS)
-                if m is not None:
-                    sdk = m.group(1)
-                    if not os.path.exists(sdk):
-                        for key in ('LDFLAGS', 'BASECFLAGS',
-                             # a number of derived variables. These need to be
-                             # patched up as well.
-                            'CFLAGS', 'PY_CFLAGS', 'BLDSHARED'):
-
-                            flags = _CONFIG_VARS[key]
-                            flags = re.sub('-isysroot\s+\S+(\s|$)', ' ', flags)
-                            _CONFIG_VARS[key] = flags
-
-    if args:
-        vals = []
-        for name in args:
-            vals.append(_CONFIG_VARS.get(name))
-        return vals
-    else:
-        return _CONFIG_VARS
-
-
-def get_config_var(name):
-    """Return the value of a single variable using the dictionary returned by
-    'get_config_vars()'.
-
-    Equivalent to get_config_vars().get(name)
-    """
-    return get_config_vars().get(name)
-
-
-def get_platform():
-    """Return a string that identifies the current platform.
-
-    This is used mainly to distinguish platform-specific build directories and
-    platform-specific built distributions.  Typically includes the OS name
-    and version and the architecture (as supplied by 'os.uname()'),
-    although the exact information included depends on the OS; eg. for IRIX
-    the architecture isn't particularly important (IRIX only runs on SGI
-    hardware), but for Linux the kernel version isn't particularly
-    important.
-
-    Examples of returned values:
-       linux-i586
-       linux-alpha (?)
-       solaris-2.6-sun4u
-       irix-5.3
-       irix64-6.2
-
-    Windows will return one of:
-       win-amd64 (64bit Windows on AMD64 (aka x86_64, Intel64, EM64T, etc)
-       win-ia64 (64bit Windows on Itanium)
-       win32 (all others - specifically, sys.platform is returned)
-
-    For other non-POSIX platforms, currently just returns 'sys.platform'.
-    """
-    if os.name == 'nt':
-        # sniff sys.version for architecture.
-        prefix = " bit ("
-        i = sys.version.find(prefix)
-        if i == -1:
-            return sys.platform
-        j = sys.version.find(")", i)
-        look = sys.version[i+len(prefix):j].lower()
-        if look == 'amd64':
-            return 'win-amd64'
-        if look == 'itanium':
-            return 'win-ia64'
-        return sys.platform
-
-    if os.name != "posix" or not hasattr(os, 'uname'):
-        # XXX what about the architecture? NT is Intel or Alpha,
-        # Mac OS is M68k or PPC, etc.
-        return sys.platform
-
-    # Try to distinguish various flavours of Unix
-    osname, host, release, version, machine = os.uname()
-
-    # Convert the OS name to lowercase, remove '/' characters
-    # (to accommodate BSD/OS), and translate spaces (for "Power Macintosh")
-    osname = osname.lower().replace('/', '')
-    machine = machine.replace(' ', '_')
-    machine = machine.replace('/', '-')
-
-    if osname[:5] == "linux":
-        # At least on Linux/Intel, 'machine' is the processor --
-        # i386, etc.
-        # XXX what about Alpha, SPARC, etc?
-        return  "%s-%s" % (osname, machine)
-    elif osname[:5] == "sunos":
-        if release[0] >= "5":           # SunOS 5 == Solaris 2
-            osname = "solaris"
-            release = "%d.%s" % (int(release[0]) - 3, release[2:])
-        # fall through to standard osname-release-machine representation
-    elif osname[:4] == "irix":              # could be "irix64"!
-        return "%s-%s" % (osname, release)
-    elif osname[:3] == "aix":
-        return "%s-%s.%s" % (osname, version, release)
-    elif osname[:6] == "cygwin":
-        osname = "cygwin"
-        rel_re = re.compile(r'[\d.]+')
-        m = rel_re.match(release)
-        if m:
-            release = m.group()
-    elif osname[:6] == "darwin":
-        #
-        # For our purposes, we'll assume that the system version from
-        # distutils' perspective is what MACOSX_DEPLOYMENT_TARGET is set
-        # to. This makes the compatibility story a bit more sane because the
-        # machine is going to compile and link as if it were
-        # MACOSX_DEPLOYMENT_TARGET.
-        cfgvars = get_config_vars()
-        macver = cfgvars.get('MACOSX_DEPLOYMENT_TARGET')
-
-        if True:
-            # Always calculate the release of the running machine,
-            # needed to determine if we can build fat binaries or not.
-
-            macrelease = macver
-            # Get the system version. Reading this plist is a documented
-            # way to get the system version (see the documentation for
-            # the Gestalt Manager)
-            try:
-                f = open('/System/Library/CoreServices/SystemVersion.plist')
-            except IOError:
-                # We're on a plain darwin box, fall back to the default
-                # behaviour.
-                pass
-            else:
-                try:
-                    m = re.search(r'<key>ProductUserVisibleVersion</key>\s*'
-                                  r'<string>(.*?)</string>', f.read())
-                finally:
-                    f.close()
-                if m is not None:
-                    macrelease = '.'.join(m.group(1).split('.')[:2])
-                # else: fall back to the default behaviour
-
-        if not macver:
-            macver = macrelease
-
-        if macver:
-            release = macver
-            osname = "macosx"
-
-            if ((macrelease + '.') >= '10.4.' and
-                '-arch' in get_config_vars().get('CFLAGS', '').strip()):
-                # The universal build will build fat binaries, but not on
-                # systems before 10.4
-                #
-                # Try to detect 4-way universal builds, those have machine-type
-                # 'universal' instead of 'fat'.
-
-                machine = 'fat'
-                cflags = get_config_vars().get('CFLAGS')
-
-                archs = re.findall('-arch\s+(\S+)', cflags)
-                archs = tuple(sorted(set(archs)))
-
-                if len(archs) == 1:
-                    machine = archs[0]
-                elif archs == ('i386', 'ppc'):
-                    machine = 'fat'
-                elif archs == ('i386', 'x86_64'):
-                    machine = 'intel'
-                elif archs == ('i386', 'ppc', 'x86_64'):
-                    machine = 'fat3'
-                elif archs == ('ppc64', 'x86_64'):
-                    machine = 'fat64'
-                elif archs == ('i386', 'ppc', 'ppc64', 'x86_64'):
-                    machine = 'universal'
-                else:
-                    raise ValueError(
-                       "Don't know machine value for archs=%r" % (archs,))
-
-            elif machine == 'i386':
-                # On OSX the machine type returned by uname is always the
-                # 32-bit variant, even if the executable architecture is
-                # the 64-bit variant
-                if sys.maxsize >= 2**32:
-                    machine = 'x86_64'
-
-            elif machine in ('PowerPC', 'Power_Macintosh'):
-                # Pick a sane name for the PPC architecture.
-                # See 'i386' case
-                if sys.maxsize >= 2**32:
-                    machine = 'ppc64'
-                else:
-                    machine = 'ppc'
-
-    return "%s-%s-%s" % (osname, release, machine)
-
-
-def get_python_version():
-    return _PY_VERSION_SHORT
-
-
-def _print_dict(title, data):
-    for index, (key, value) in enumerate(sorted(data.items())):
-        if index == 0:
-            print('%s: ' % (title))
-        print('\t%s = "%s"' % (key, value))
-
-
-def _main():
-    """Display all information sysconfig detains."""
-    print('Platform: "%s"' % get_platform())
-    print('Python version: "%s"' % get_python_version())
-    print('Current installation scheme: "%s"' % _get_default_scheme())
-    print()
-    _print_dict('Paths', get_paths())
-    print()
-    _print_dict('Variables', get_config_vars())
-
-
-if __name__ == '__main__':
-    _main()

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 12 19:13:45 2011
From: python-checkins at python.org (alexis.metaireau)
Date: Mon, 12 Sep 2011 19:13:45 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Use_=60setup=2Ecfg=60_as?=
 =?utf8?q?_the_canonical_source_for_installation?=
Message-ID: <E1R3A4n-0006if-CH@dinsdale.python.org>

http://hg.python.org/distutils2/rev/288640098ea8
changeset:   1139:288640098ea8
user:        Jeremy Kloth <jeremy.kloth at gmail.com>
date:        Mon Sep 12 05:37:52 2011 -0600
summary:
  Use `setup.cfg` as the canonical source for installation

files:
  distutils2/config.py |    3 +-
  setup.cfg            |   35 ++-
  setup.py             |  383 +++++++++++++-----------------
  3 files changed, 204 insertions(+), 217 deletions(-)


diff --git a/distutils2/config.py b/distutils2/config.py
--- a/distutils2/config.py
+++ b/distutils2/config.py
@@ -221,7 +221,8 @@
                 if len(data) != 2:
                     continue  # XXX error should never pass silently
                 key, value = data
-                self.dist.package_data[key.strip()] = value.strip()
+                globs = self.dist.package_data.setdefault(key.strip(), [])
+                globs.extend([v.strip() for v in value.split(',')])
 
             self.dist.data_files = []
             for data in files.get('data_files', []):
diff --git a/setup.cfg b/setup.cfg
--- a/setup.cfg
+++ b/setup.cfg
@@ -8,5 +8,36 @@
 [metadata]
 name = Distutils2
 version = 1.0a3
-home-page = http://xxx
-author = Ok
+summary = Python Distribution Utilities
+description-file = README.txt
+home-page = http://bitbucket.org/tarek/distutils2/wiki/Home
+author = Tarek Ziade
+author-email = tarek at ziade.org
+classifier =
+    Development Status :: 3 - Alpha
+    Intended Audience :: Developers
+    License :: OSI Approved :: Python Software Foundation License
+    Operating System :: OS Independent
+    Programming Language :: Python
+    Topic :: Software Development :: Libraries :: Python Modules
+    Topic :: System :: Archiving :: Packaging
+    Topic :: System :: Systems Administration
+    Topic :: Utilities
+license = PSF
+keywords = packaging,distutils
+
+[files]
+packages =
+    distutils2
+    distutils2.command
+    distutils2.compiler
+    distutils2.pypi
+    distutils2.tests
+    distutils2.tests.fixer
+    distutils2._backport
+    distutils2._backport.tests
+package_data =
+    distutils2._backport = sysconfig.cfg
+    distutils2.command = wininst*.exe
+scripts =
+    pysetup
\ No newline at end of file
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1,249 +1,204 @@
 #!/usr/bin/env python
 # -*- encoding: utf-8 -*-
-import sys
 import os
 import re
+import sys
+import codecs
+from distutils import sysconfig
+from distutils.core import setup, Extension
+from distutils.ccompiler import new_compiler
+try:
+    from configparser import RawConfigParser
+except ImportError: #<3.0
+    from ConfigParser import RawConfigParser
 
-from distutils2 import __version__ as VERSION
-from distutils import log
-from distutils.ccompiler import new_compiler
-from distutils.command.sdist import sdist
-from distutils.command.install import install
+def cfg_to_args(path='setup.cfg'):
+    from distutils2.util import split_multiline
+    def split_elements(value):
+        return [ v.strip() for v in value.split(',') ]
+    def split_files(value):
+        return [ str(v) for v in split_multiline(value) ]
+    opts_to_args =  {
+        'metadata': (
+            ('name', 'name', None),
+            ('version', 'version', None),
+            ('author', 'author', None),
+            ('author-email', 'author_email', None),
+            ('maintainer', 'maintainer', None),
+            ('maintainer-email', 'maintainer_email', None),
+            ('home-page', 'url', None),
+            ('summary', 'description', None),
+            ('description', 'long_description', None),
+            ('download-url', 'download_url', None),
+            ('classifier', 'classifiers', split_multiline),
+            ('platform', 'platforms', split_multiline),
+            ('license', 'license', None),
+            ('keywords', 'keywords', split_elements),
+            ),
+        'files': (
+            ('packages', 'packages', split_files),
+            ('modules', 'py_modules', split_files),
+            ('scripts', 'scripts', split_files),
+            ('package_data', 'package_data', split_files),
+            ),
+        }
+    config = RawConfigParser()
+    config.optionxform = lambda x: x.lower().replace('_', '-')
+    fp = codecs.open(path, encoding='utf-8')
+    try:
+        config.readfp(fp)
+    finally:
+        fp.close()
+    kwargs = {}
+    for section in opts_to_args:
+        for optname, argname, xform in opts_to_args[section]:
+            if config.has_option(section, optname):
+                value = config.get(section, optname)
+                if xform:
+                    value = xform(value)
+                kwargs[argname] = value
+    # Handle `description-file`
+    if ('long_description' not in kwargs and
+            config.has_option('metadata', 'description-file')):
+        filenames = config.get('metadata', 'description-file')
+        for filename in split_multiline(filenames):
+            descriptions = []
+            fp = open(filename)
+            try:
+                descriptions.append(fp.read())
+            finally:
+                fp.close()
+        kwargs['long_description'] = '\n\n'.join(descriptions)
+    # Handle `package_data`
+    if 'package_data' in kwargs:
+        package_data = {}
+        for data in kwargs['package_data']:
+            key, value = data.split('=', 1)
+            globs = package_data.setdefault(key.strip(), [])
+            globs.extend(split_elements(value))
+        kwargs['package_data'] = package_data
+    return kwargs
 
-# Python 3.x hook to run 2to3 automatically
-try:
-    from distutils.command.build_py import build_py_2to3 as build_py
-    from distutils.core import setup, Extension
-except ImportError:
-    # 2.x, try to use setuptools if available
-    try :
-        from setuptools.command.build_py import build_py
-        from setuptools import setup, Extension
-    except ImportError:
-        from distutils.command.build_py import build_py
-        from distutils.core import setup, Extension
-
-
-f = open('README.txt')
-try:
-    README = f.read()
-finally:
-    f.close()
-
-def get_tip_revision(path=os.getcwd()):
-    from subprocess import Popen, PIPE
-    try:
-        cmd = Popen(['hg', 'tip', '--template', '{rev}', '-R', path],
-                    stdout=PIPE, stderr=PIPE)
-    except OSError:
-        return 0
-    rev = cmd.stdout.read()
-    if not rev :
-        # there has been an error in the command
-        return 0
-    return int(rev)
-
-DEV_SUFFIX = '.dev%d' % get_tip_revision('..')
-
-
-class install_hg(install):
-
-    user_options = install.user_options + [
-            ('dev', None, "Add a dev marker")
-            ]
-
-    def initialize_options(self):
-        install.initialize_options(self)
-        self.dev = 0
-
-    def run(self):
-        if self.dev:
-            self.distribution.metadata.version += DEV_SUFFIX
-        install.run(self)
-
-
-class sdist_hg(sdist):
-
-    user_options = sdist.user_options + [
-            ('dev', None, "Add a dev marker")
-            ]
-
-    def initialize_options(self):
-        sdist.initialize_options(self)
-        self.dev = 0
-
-    def run(self):
-        if self.dev:
-            self.distribution.metadata.version += DEV_SUFFIX
-        sdist.run(self)
-
-
-# additional paths to check, set from the command line
-SSL_INCDIR = ''   # --openssl-incdir=
-SSL_LIBDIR = ''   # --openssl-libdir=
-SSL_DIR = ''      # --openssl-prefix=
-
-def add_dir_to_list(dirlist, dir):
-    """Add the directory 'dir' to the list 'dirlist' (at the front) if
-    'dir' actually exists and is a directory.  If 'dir' is already in
-    'dirlist' it is moved to the front.
-    """
-    if dir is not None and os.path.isdir(dir) and dir not in dirlist:
-        if dir in dirlist:
-            dirlist.remove(dir)
-        dirlist.insert(0, dir)
-
-
+# (from Python's setup.py, in PyBuildExt.detect_modules())
 def prepare_hashlib_extensions():
     """Decide which C extensions to build and create the appropriate
     Extension objects to build them.  Return a list of Extensions.
     """
-    # this CCompiler object is only used to locate include files
-    compiler = new_compiler()
+    ssl_libs = None
+    ssl_inc_dir = None
+    ssl_lib_dirs = []
+    ssl_inc_dirs = []
+    if os.name == 'posix':
+        # (from Python's setup.py, in PyBuildExt.detect_modules())
+        # lib_dirs and inc_dirs are used to search for files;
+        # if a file is found in one of those directories, it can
+        # be assumed that no additional -I,-L directives are needed.
+        lib_dirs = []
+        inc_dirs = []
+        if os.path.normpath(sys.prefix) != '/usr':
+            lib_dirs.append(sysconfig.get_config_var('LIBDIR'))
+            inc_dirs.append(sysconfig.get_config_var('INCLUDEDIR'))
+        # Ensure that /usr/local is always used
+        lib_dirs.append('/usr/local/lib')
+        inc_dirs.append('/usr/local/include')
+        # Add the compiler defaults; this compiler object is only used
+        # to locate the OpenSSL files.
+        compiler = new_compiler()
+        lib_dirs.extend(compiler.library_dirs)
+        inc_dirs.extend(compiler.include_dirs)
+        # Now the platform defaults
+        lib_dirs.extend(['/lib64', '/usr/lib64', '/lib', '/usr/lib'])
+        inc_dirs.extend(['/usr/include'])
+        # Find the SSL library directory
+        ssl_libs = ['ssl', 'crypto']
+        ssl_lib = compiler.find_library_file(lib_dirs, 'ssl')
+        if ssl_lib is None:
+            ssl_lib_dirs = ['/usr/local/ssl/lib', '/usr/contrib/ssl/lib']
+            ssl_lib = compiler.find_library_file(ssl_lib_dirs, 'ssl')
+            if ssl_lib is not None:
+                ssl_lib_dirs.append(os.path.dirname(ssl_lib))
+            else:
+                ssl_libs = None
+        # Locate the SSL headers
+        for ssl_inc_dir in inc_dirs + ['/usr/local/ssl/include',
+                                       '/usr/contrib/ssl/include']:
+            ssl_h = os.path.join(ssl_inc_dir, 'openssl', 'ssl.h')
+            if os.path.exists(ssl_h):
+                if ssl_inc_dir not in inc_dirs:
+                    ssl_inc_dirs.append(ssl_inc_dir)
+                break
+    elif os.name == 'nt':
+        # (from Python's PCbuild/build_ssl.py, in find_best_ssl_dir())
+        # Look for SSL 1 level up from here.  That is, the same place the
+        # other externals for Python core live.
+        # note: do not abspath src_dir; the build will fail if any
+        # higher up directory name has spaces in it.
+        src_dir = '..'
+        try:
+            fnames = os.listdir(src_dir)
+        except OSError:
+            fnames = []
+        ssl_dir = None
+        best_parts = []
+        for fname in fnames:
+            fqn = os.path.join(src_dir, fname)
+            if os.path.isdir(fqn) and fname.startswith("openssl-"):
+                # We have a candidate, determine the best
+                parts = re.split("[.-]", fname)[1:]
+                # Ignore all "beta" or any other qualifiers;
+                # eg - openssl-0.9.7-beta1
+                if len(parts) < 4 and parts > best_parts:
+                    best_parts = parts
+                    ssl_dir = fqn
+        if ssl_dir is not None:
+            ssl_libs = ['gdi32', 'user32', 'advapi32',
+                        os.path.join(ssl_dir, 'out32', 'libeay32')]
+            ssl_inc_dir = os.path.join(ssl_dir, 'inc32')
+            ssl_inc_dirs.append(ssl_inc_dir)
 
-    # Ensure that these paths are always checked
-    if os.name == 'posix':
-        add_dir_to_list(compiler.library_dirs, '/usr/local/lib')
-        add_dir_to_list(compiler.include_dirs, '/usr/local/include')
-
-        add_dir_to_list(compiler.library_dirs, '/usr/local/ssl/lib')
-        add_dir_to_list(compiler.include_dirs, '/usr/local/ssl/include')
-
-        add_dir_to_list(compiler.library_dirs, '/usr/contrib/ssl/lib')
-        add_dir_to_list(compiler.include_dirs, '/usr/contrib/ssl/include')
-
-        add_dir_to_list(compiler.library_dirs, '/usr/lib')
-        add_dir_to_list(compiler.include_dirs, '/usr/include')
-
-    # look in paths supplied on the command line
-    if SSL_LIBDIR:
-        add_dir_to_list(compiler.library_dirs, SSL_LIBDIR)
-    if SSL_INCDIR:
-        add_dir_to_list(compiler.include_dirs, SSL_INCDIR)
-    if SSL_DIR:
-        if os.name == 'nt':
-            add_dir_to_list(compiler.library_dirs, os.path.join(SSL_DIR, 'out32dll'))
-            # prefer the static library
-            add_dir_to_list(compiler.library_dirs, os.path.join(SSL_DIR, 'out32'))
-        else:
-            add_dir_to_list(compiler.library_dirs, os.path.join(SSL_DIR, 'lib'))
-        add_dir_to_list(compiler.include_dirs, os.path.join(SSL_DIR, 'include'))
-
-    oslibs = {'posix': ['ssl', 'crypto'],
-              'nt': ['libeay32',  'gdi32', 'advapi32', 'user32']}
-
-    if os.name not in oslibs:
-        sys.stderr.write(
-            'unknown operating system, impossible to compile _hashlib')
-        sys.exit(1)
-
-    exts = []
-
-    ssl_inc_dirs = []
-    ssl_incs = []
-    for inc_dir in compiler.include_dirs:
-        f = os.path.join(inc_dir, 'openssl', 'ssl.h')
-        if os.path.exists(f):
-            ssl_incs.append(f)
-            ssl_inc_dirs.append(inc_dir)
-
-    ssl_lib = compiler.find_library_file(compiler.library_dirs, oslibs[os.name][0])
-
-    # find out which version of OpenSSL we have
+    # Find out which version of OpenSSL we have
     openssl_ver = 0
     openssl_ver_re = re.compile(
         '^\s*#\s*define\s+OPENSSL_VERSION_NUMBER\s+(0x[0-9a-fA-F]+)' )
-    ssl_inc_dir = ''
-    for ssl_inc_dir in ssl_inc_dirs:
-        name = os.path.join(ssl_inc_dir, 'openssl', 'opensslv.h')
-        if os.path.isfile(name):
-            try:
-                incfile = open(name, 'r')
-                for line in incfile:
-                    m = openssl_ver_re.match(line)
-                    if m:
-                        openssl_ver = int(m.group(1), 16)
-                        break
-            except IOError:
-                pass
+    if ssl_inc_dir is not None:
+        opensslv_h = os.path.join(ssl_inc_dir, 'openssl', 'opensslv.h')
+        try:
+            incfile = open(opensslv_h, 'r')
+            for line in incfile:
+                m = openssl_ver_re.match(line)
+                if m:
+                    openssl_ver = int(m.group(1), 16)
+        except IOError:
+            e = str(sys.last_value)
+            print("IOError while reading %s: %s" % (opensslv_h, e))
 
-        # first version found is what we'll use
-        if openssl_ver:
-            break
-
-    if (ssl_inc_dir and ssl_lib is not None and openssl_ver >= 0x00907000):
-
-        log.info('Using OpenSSL version 0x%08x from', openssl_ver)
-        log.info(' Headers:\t%s', ssl_inc_dir)
-        log.info(' Library:\t%s', ssl_lib)
-
+    # Now we can determine which extension modules need to be built.
+    exts = []
+    if ssl_libs is not None and openssl_ver >= 0x907000:
         # The _hashlib module wraps optimized implementations
         # of hash functions from the OpenSSL library.
         exts.append(Extension('distutils2._backport._hashlib',
                               ['distutils2/_backport/_hashopenssl.c'],
-                              include_dirs = [ssl_inc_dir],
-                              library_dirs = [os.path.dirname(ssl_lib)],
-                              libraries = oslibs[os.name]))
+                              include_dirs=ssl_inc_dirs,
+                              library_dirs=ssl_lib_dirs,
+                              libraries=ssl_libs))
     else:
+        # no openssl at all, use our own md5 and sha1
         exts.append(Extension('distutils2._backport._sha',
                               ['distutils2/_backport/shamodule.c']))
         exts.append(Extension('distutils2._backport._md5',
                               sources=['distutils2/_backport/md5module.c',
                                        'distutils2/_backport/md5.c'],
                               depends=['distutils2/_backport/md5.h']) )
-
-    if (not ssl_lib or openssl_ver < 0x00908000):
+    if openssl_ver < 0x908000:
         # OpenSSL doesn't do these until 0.9.8 so we'll bring our own
         exts.append(Extension('distutils2._backport._sha256',
                               ['distutils2/_backport/sha256module.c']))
         exts.append(Extension('distutils2._backport._sha512',
                               ['distutils2/_backport/sha512module.c']))
-
     return exts
 
-setup_kwargs = {'scripts': ['scripts/pysetup']}
-
-if sys.version < '2.6':
-    setup_kwargs['scripts'].append('distutils2/create.py')
-
+setup_kwargs = cfg_to_args('setup.cfg')
 if sys.version < '2.5':
     setup_kwargs['ext_modules'] = prepare_hashlib_extensions()
-
-_CLASSIFIERS = """\
-Development Status :: 3 - Alpha
-Intended Audience :: Developers
-License :: OSI Approved :: Python Software Foundation License
-Operating System :: OS Independent
-Programming Language :: Python
-Topic :: Software Development :: Libraries :: Python Modules
-Topic :: System :: Archiving :: Packaging
-Topic :: System :: Systems Administration
-Topic :: Utilities"""
-
-setup(name="Distutils2",
-      version=VERSION,
-      description="Python Distribution Utilities",
-      keywords=['packaging', 'distutils'],
-      author="Tarek Ziade",
-      author_email="tarek at ziade.org",
-      url="http://bitbucket.org/tarek/distutils2/wiki/Home",
-      license="PSF",
-      long_description=README,
-      classifiers=_CLASSIFIERS.split('\n'),
-      packages=[
-          'distutils2',
-          'distutils2.tests', 
-          'distutils2.compiler', 
-          'distutils2.command', 
-          'distutils2._backport', 
-          'distutils2.pypi', 
-          'distutils2.tests.fixer', 
-          'distutils2._backport.tests',
-      ],
-      cmdclass={
-          'build_py':build_py, 
-          'install_hg': install_hg,
-          'sdist_hg': sdist_hg, 
-      },
-      package_data={'distutils2._backport': ['sysconfig.cfg']},
-      **setup_kwargs)
+setup(**setup_kwargs)

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 12 19:13:45 2011
From: python-checkins at python.org (alexis.metaireau)
Date: Mon, 12 Sep 2011 19:13:45 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Add_support_for_building?=
 =?utf8?q?_OpenSSL_on_Windows_=28backported_from_CPython_with_fixes?=
Message-ID: <E1R3A4n-0006ij-Cp@dinsdale.python.org>

http://hg.python.org/distutils2/rev/e0098d4649a6
changeset:   1140:e0098d4649a6
user:        Jeremy Kloth <jeremy.kloth at gmail.com>
date:        Mon Sep 12 06:03:41 2011 -0600
summary:
  Add support for building OpenSSL on Windows (backported from CPython with fixes for 2.x support)

files:
  PC/build_ssl.py |  282 ++++++++++++++++++++++++++++++++++++
  1 files changed, 282 insertions(+), 0 deletions(-)


diff --git a/PC/build_ssl.py b/PC/build_ssl.py
new file mode 100644
--- /dev/null
+++ b/PC/build_ssl.py
@@ -0,0 +1,282 @@
+# Script for building the _ssl and _hashlib modules for Windows.
+# Uses Perl to setup the OpenSSL environment correctly
+# and build OpenSSL, then invokes a simple nmake session
+# for the actual _ssl.pyd and _hashlib.pyd DLLs.
+
+# THEORETICALLY, you can:
+# * Unpack the latest SSL release one level above your main Python source
+#   directory.  It is likely you will already find the zlib library and
+#   any other external packages there.
+# * Install ActivePerl and ensure it is somewhere on your path.
+# * Run this script from the PCBuild directory.
+#
+# it should configure and build SSL, then build the _ssl and _hashlib
+# Python extensions without intervention.
+
+# Modified by Christian Heimes
+# Now this script supports pre-generated makefiles and assembly files.
+# Developers don't need an installation of Perl anymore to build Python. A svn
+# checkout from our svn repository is enough.
+#
+# In Order to create the files in the case of an update you still need Perl.
+# Run build_ssl in this order:
+# python.exe build_ssl.py Release x64
+# python.exe build_ssl.py Release Win32
+
+import os, sys, re, shutil
+
+# Find all "foo.exe" files on the PATH.
+def find_all_on_path(filename, extras = None):
+    entries = os.environ["PATH"].split(os.pathsep)
+    ret = []
+    for p in entries:
+        fname = os.path.abspath(os.path.join(p, filename))
+        if os.path.isfile(fname) and fname not in ret:
+            ret.append(fname)
+    if extras:
+        for p in extras:
+            fname = os.path.abspath(os.path.join(p, filename))
+            if os.path.isfile(fname) and fname not in ret:
+                ret.append(fname)
+    return ret
+
+# Find a suitable Perl installation for OpenSSL.
+# cygwin perl does *not* work.  ActivePerl does.
+# Being a Perl dummy, the simplest way I can check is if the "Win32" package
+# is available.
+def find_working_perl(perls):
+    for perl in perls:
+        fh = os.popen('"%s" -e "use Win32;"' % perl)
+        fh.read()
+        rc = fh.close()
+        if rc:
+            continue
+        return perl
+    print("Can not find a suitable PERL:")
+    if perls:
+        print(" the following perl interpreters were found:")
+        for p in perls:
+            print(" ", p)
+        print(" None of these versions appear suitable for building OpenSSL")
+    else:
+        print(" NO perl interpreters were found on this machine at all!")
+    print(" Please install ActivePerl and ensure it appears on your path")
+    return None
+
+# Locate the best SSL directory given a few roots to look into.
+def find_best_ssl_dir(sources):
+    candidates = []
+    for s in sources:
+        try:
+            # note: do not abspath s; the build will fail if any
+            # higher up directory name has spaces in it.
+            fnames = os.listdir(s)
+        except os.error:
+            fnames = []
+        for fname in fnames:
+            fqn = os.path.join(s, fname)
+            if os.path.isdir(fqn) and fname.startswith("openssl-"):
+                candidates.append(fqn)
+    # Now we have all the candidates, locate the best.
+    best_parts = []
+    best_name = None
+    for c in candidates:
+        parts = re.split("[.-]", os.path.basename(c))[1:]
+        # eg - openssl-0.9.7-beta1 - ignore all "beta" or any other qualifiers
+        if len(parts) >= 4:
+            continue
+        if parts > best_parts:
+            best_parts = parts
+            best_name = c
+    if best_name is not None:
+        print("Found an SSL directory at '%s'" % (best_name,))
+    else:
+        print("Could not find an SSL directory in '%s'" % (sources,))
+    sys.stdout.flush()
+    return best_name
+
+def create_makefile64(makefile, m32):
+    """Create and fix makefile for 64bit
+
+    Replace 32 with 64bit directories
+    """
+    if not os.path.isfile(m32):
+        return
+    with open(m32) as fin:
+        with open(makefile, 'w') as fout:
+            for line in fin:
+                line = line.replace("=tmp32", "=tmp64")
+                line = line.replace("=out32", "=out64")
+                line = line.replace("=inc32", "=inc64")
+                # force 64 bit machine
+                line = line.replace("MKLIB=lib", "MKLIB=lib /MACHINE:X64")
+                line = line.replace("LFLAGS=", "LFLAGS=/MACHINE:X64 ")
+                # don't link against the lib on 64bit systems
+                line = line.replace("bufferoverflowu.lib", "")
+                fout.write(line)
+    os.unlink(m32)
+
+def fix_makefile(makefile):
+    """Fix some stuff in all makefiles
+    """
+    if not os.path.isfile(makefile):
+        return
+    with open(makefile) as fin:
+        lines = fin.readlines()
+    with open(makefile, 'w') as fout:
+        for line in lines:
+            if line.startswith("PERL="):
+                continue
+            if line.startswith("CP="):
+                line = "CP=copy\n"
+            if line.startswith("MKDIR="):
+                line = "MKDIR=mkdir\n"
+            if line.startswith("CFLAG="):
+                line = line.strip()
+                for algo in ("RC5", "MDC2", "IDEA"):
+                    noalgo = " -DOPENSSL_NO_%s" % algo
+                    if noalgo not in line:
+                        line = line + noalgo
+                line = line + '\n'
+            fout.write(line)
+
+def run_configure(configure, do_script):
+    print("perl Configure "+configure+" no-idea no-mdc2")
+    os.system("perl Configure "+configure+" no-idea no-mdc2")
+    print(do_script)
+    os.system(do_script)
+
+def cmp(f1, f2):
+    bufsize = 1024 * 8
+    with open(f1, 'rb') as fp1, open(f2, 'rb') as fp2:
+        while True:
+            b1 = fp1.read(bufsize)
+            b2 = fp2.read(bufsize)
+            if b1 != b2:
+                return False
+            if not b1:
+                return True
+
+def copy(src, dst):
+    if os.path.isfile(dst) and cmp(src, dst):
+        return
+    shutil.copy(src, dst)
+
+def main():
+    build_all = "-a" in sys.argv
+    # Default to 'Release' configuration on for the 'Win32' platform
+    try:
+        configuration, platform = sys.argv[1:3]
+    except ValueError:
+        configuration, platform = 'Release', 'Win32'
+    if configuration == "Release":
+        debug = False
+    elif configuration == "Debug":
+        debug = True
+    else:
+        raise ValueError(str(sys.argv))
+
+    if platform == "Win32":
+        arch = "x86"
+        configure = "VC-WIN32"
+        do_script = "ms\\do_nasm"
+        makefile="ms\\nt.mak"
+        m32 = makefile
+        dirsuffix = "32"
+    elif platform == "x64":
+        arch="amd64"
+        configure = "VC-WIN64A"
+        do_script = "ms\\do_win64a"
+        makefile = "ms\\nt64.mak"
+        m32 = makefile.replace('64', '')
+        dirsuffix = "64"
+        #os.environ["VSEXTCOMP_USECL"] = "MS_OPTERON"
+    else:
+        raise ValueError(str(sys.argv))
+
+    make_flags = ""
+    if build_all:
+        make_flags = "-a"
+    # perl should be on the path, but we also look in "\perl" and "c:\\perl"
+    # as "well known" locations
+    perls = find_all_on_path("perl.exe", ["\\perl\\bin", "C:\\perl\\bin"])
+    perl = find_working_perl(perls)
+    if perl:
+        print("Found a working perl at '%s'" % (perl,))
+    else:
+        print("No Perl installation was found. Existing Makefiles are used.")
+    sys.stdout.flush()
+    # Look for SSL 2 levels up from pcbuild - ie, same place zlib etc all live.
+    ssl_dir = find_best_ssl_dir(("..\\..",))
+    if ssl_dir is None:
+        sys.exit(1)
+
+    old_cd = os.getcwd()
+    try:
+        os.chdir(ssl_dir)
+        # rebuild makefile when we do the role over from 32 to 64 build
+        if arch == "amd64" and os.path.isfile(m32) and not os.path.isfile(makefile):
+            os.unlink(m32)
+
+        # If the ssl makefiles do not exist, we invoke Perl to generate them.
+        # Due to a bug in this script, the makefile sometimes ended up empty
+        # Force a regeneration if it is.
+        if not os.path.isfile(makefile) or os.path.getsize(makefile)==0:
+            if perl is None:
+                print("Perl is required to build the makefiles!")
+                sys.exit(1)
+
+            print("Creating the makefiles...")
+            sys.stdout.flush()
+            # Put our working Perl at the front of our path
+            os.environ["PATH"] = os.path.dirname(perl) + \
+                                          os.pathsep + \
+                                          os.environ["PATH"]
+            run_configure(configure, do_script)
+            if debug:
+                print("OpenSSL debug builds aren't supported.")
+            #if arch=="x86" and debug:
+            #    # the do_masm script in openssl doesn't generate a debug
+            #    # build makefile so we generate it here:
+            #    os.system("perl util\mk1mf.pl debug "+configure+" >"+makefile)
+
+            if arch == "amd64":
+                create_makefile64(makefile, m32)
+            fix_makefile(makefile)
+            copy(r"crypto\buildinf.h", r"crypto\buildinf_%s.h" % arch)
+            copy(r"crypto\opensslconf.h", r"crypto\opensslconf_%s.h" % arch)
+
+        # If the assembler files don't exist in tmpXX, copy them there
+        if perl is None and os.path.exists("asm"+dirsuffix):
+            if not os.path.exists("tmp"+dirsuffix):
+                os.mkdir("tmp"+dirsuffix)
+            for f in os.listdir("asm"+dirsuffix):
+                if not f.endswith(".asm"): continue
+                if os.path.isfile(r"tmp%s\%s" % (dirsuffix, f)): continue
+                shutil.copy(r"asm%s\%s" % (dirsuffix, f), "tmp"+dirsuffix)
+
+        # Now run make.
+        if arch == "amd64":
+            rc = os.system("ml64 -c -Foms\\uptable.obj ms\\uptable.asm")
+            if rc:
+                print("ml64 assembler has failed.")
+                sys.exit(rc)
+
+        copy(r"crypto\buildinf_%s.h" % arch, r"crypto\buildinf.h")
+        copy(r"crypto\opensslconf_%s.h" % arch, r"crypto\opensslconf.h")
+
+        #makeCommand = "nmake /nologo PERL=\"%s\" -f \"%s\"" %(perl, makefile)
+        makeCommand = "nmake /nologo -f \"%s\"" % makefile
+        print("Executing ssl makefiles: " + makeCommand)
+        sys.stdout.flush()
+        rc = os.system(makeCommand)
+        if rc:
+            print("Executing "+makefile+" failed")
+            print(rc)
+            sys.exit(rc)
+    finally:
+        os.chdir(old_cd)
+    sys.exit(rc)
+
+if __name__=='__main__':
+    main()

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 12 19:36:24 2011
From: python-checkins at python.org (alexis.metaireau)
Date: Mon, 12 Sep 2011 19:36:24 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Factor_out_the_distribution?=
 =?utf8?q?_file-system_safe_name_functions_from?=
Message-ID: <E1R3AQi-0000ff-MV@dinsdale.python.org>

http://hg.python.org/cpython/rev/a03da3dc478e
changeset:   72364:a03da3dc478e
parent:      72360:76edde9b63ae
user:        Jeremy Kloth <jeremy.kloth at gmail.com>
date:        Mon Sep 12 11:12:42 2011 -0600
summary:
  Factor out the distribution file-system safe name functions from install_distinfo to allow all metadata consumers access to them.

files:
  Lib/packaging/command/install_distinfo.py |  32 +----------
  Lib/packaging/dist.py                     |   4 +-
  Lib/packaging/metadata.py                 |  16 ++++-
  3 files changed, 16 insertions(+), 36 deletions(-)


diff --git a/Lib/packaging/command/install_distinfo.py b/Lib/packaging/command/install_distinfo.py
--- a/Lib/packaging/command/install_distinfo.py
+++ b/Lib/packaging/command/install_distinfo.py
@@ -63,9 +63,7 @@
 
         metadata = self.distribution.metadata
 
-        basename = "%s-%s.dist-info" % (
-            to_filename(safe_name(metadata['Name'])),
-            to_filename(safe_version(metadata['Version'])))
+        basename = metadata.get_fullname(filesafe=True) + ".dist-info"
 
         self.distinfo_dir = os.path.join(self.distinfo_dir, basename)
 
@@ -145,31 +143,3 @@
 
     def get_outputs(self):
         return self.outfiles
-
-
-# The following functions are taken from setuptools' pkg_resources module.
-
-def safe_name(name):
-    """Convert an arbitrary string to a standard distribution name
-
-    Any runs of non-alphanumeric/. characters are replaced with a single '-'.
-    """
-    return re.sub('[^A-Za-z0-9.]+', '-', name)
-
-
-def safe_version(version):
-    """Convert an arbitrary string to a standard version string
-
-    Spaces become dots, and all other non-alphanumeric characters become
-    dashes, with runs of multiple dashes condensed to a single dash.
-    """
-    version = version.replace(' ', '.')
-    return re.sub('[^A-Za-z0-9.]+', '-', version)
-
-
-def to_filename(name):
-    """Convert a project or version name to its filename-escaped form
-
-    Any '-' characters are currently replaced with '_'.
-    """
-    return name.replace('-', '_')
diff --git a/Lib/packaging/dist.py b/Lib/packaging/dist.py
--- a/Lib/packaging/dist.py
+++ b/Lib/packaging/dist.py
@@ -228,8 +228,8 @@
             d = self.command_options[command] = {}
         return d
 
-    def get_fullname(self):
-        return self.metadata.get_fullname()
+    def get_fullname(self, filesafe=False):
+        return self.metadata.get_fullname(filesafe)
 
     def dump_option_dicts(self, header=None, commands=None, indent=""):
         from pprint import pformat
diff --git a/Lib/packaging/metadata.py b/Lib/packaging/metadata.py
--- a/Lib/packaging/metadata.py
+++ b/Lib/packaging/metadata.py
@@ -182,6 +182,7 @@
 
 _MISSING = object()
 
+_FILESAFE = re.compile('[^A-Za-z0-9.]+')
 
 class Metadata:
     """The metadata of a release.
@@ -285,9 +286,18 @@
     #
     # Public API
     #
-    def get_fullname(self):
-        """Return the distribution name with version"""
-        return '%s-%s' % (self['Name'], self['Version'])
+    def get_fullname(self, filesafe=False):
+        """Return the distribution name with version.
+
+        If filesafe is true, return a filename-escaped form."""
+        name, version = self['Name'], self['Version']
+        if filesafe:
+            # For both name and version any runs of non-alphanumeric or '.'
+            # characters are replaced with a single '-'.  Additionally any
+            # spaces in the version string become '.'
+            name = _FILESAFE.sub('-', name)
+            version = _FILESAFE.sub('-', version.replace(' ', '.'))
+        return '%s-%s' % (name, version)
 
     def is_metadata_field(self, name):
         """return True if name is a valid metadata key"""

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 21:20:19 2011
From: python-checkins at python.org (amaury.forgeotdarc)
Date: Mon, 12 Sep 2011 21:20:19 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyNDgz?=
 =?utf8?q?=3A_ctypes=3A_Fix_a_crash_when_the_destruction_of_a_callback?=
Message-ID: <E1R3C3H-0002ej-6M@dinsdale.python.org>

http://hg.python.org/cpython/rev/eb9f566fd8db
changeset:   72365:eb9f566fd8db
branch:      2.7
parent:      72363:18676f0118e4
user:        Amaury Forgeot d'Arc <amauryfa at gmail.com>
date:        Mon Sep 12 20:12:09 2011 +0200
summary:
  Issue #12483: ctypes: Fix a crash when the destruction of a callback
object triggers the garbage collector.

files:
  Lib/ctypes/test/test_callbacks.py |  8 ++++++++
  Misc/NEWS                         |  3 +++
  Modules/_ctypes/callbacks.c       |  1 +
  3 files changed, 12 insertions(+), 0 deletions(-)


diff --git a/Lib/ctypes/test/test_callbacks.py b/Lib/ctypes/test/test_callbacks.py
--- a/Lib/ctypes/test/test_callbacks.py
+++ b/Lib/ctypes/test/test_callbacks.py
@@ -140,6 +140,14 @@
                 if isinstance(x, X)]
         self.assertEqual(len(live), 0)
 
+    def test_issue12483(self):
+        import gc
+        class Nasty:
+            def __del__(self):
+                gc.collect()
+        CFUNCTYPE(None)(lambda x=Nasty(): None)
+        
+
 try:
     WINFUNCTYPE
 except NameError:
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -192,6 +192,9 @@
 Extension Modules
 -----------------
 
+- Issue #12483: ctypes: Fix a crash when the destruction of a callback
+  object triggers the garbage collector.
+
 - Issue #12950: Fix passing file descriptors in multiprocessing, under
   OpenIndiana/Illumos.
 
diff --git a/Modules/_ctypes/callbacks.c b/Modules/_ctypes/callbacks.c
--- a/Modules/_ctypes/callbacks.c
+++ b/Modules/_ctypes/callbacks.c
@@ -18,6 +18,7 @@
 CThunkObject_dealloc(PyObject *_self)
 {
     CThunkObject *self = (CThunkObject *)_self;
+    PyObject_GC_UnTrack(self);
     Py_XDECREF(self->converters);
     Py_XDECREF(self->callable);
     Py_XDECREF(self->restype);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 21:20:20 2011
From: python-checkins at python.org (amaury.forgeotdarc)
Date: Mon, 12 Sep 2011 21:20:20 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyNDgz?=
 =?utf8?q?=3A_ctypes=3A_Fix_a_crash_when_the_destruction_of_a_callback?=
Message-ID: <E1R3C3I-0002fF-2S@dinsdale.python.org>

http://hg.python.org/cpython/rev/eae8e4ab0455
changeset:   72366:eae8e4ab0455
branch:      3.2
parent:      72358:882d3b78b1cc
user:        Amaury Forgeot d'Arc <amauryfa at gmail.com>
date:        Mon Sep 12 21:03:36 2011 +0200
summary:
  Issue #12483: ctypes: Fix a crash when the destruction of a callback
object triggers the garbage collector.

files:
  Lib/ctypes/test/test_callbacks.py |  8 ++++++++
  Misc/NEWS                         |  3 +++
  Modules/_ctypes/callbacks.c       |  1 +
  3 files changed, 12 insertions(+), 0 deletions(-)


diff --git a/Lib/ctypes/test/test_callbacks.py b/Lib/ctypes/test/test_callbacks.py
--- a/Lib/ctypes/test/test_callbacks.py
+++ b/Lib/ctypes/test/test_callbacks.py
@@ -134,6 +134,14 @@
                 if isinstance(x, X)]
         self.assertEqual(len(live), 0)
 
+    def test_issue12483(self):
+        import gc
+        class Nasty:
+            def __del__(self):
+                gc.collect()
+        CFUNCTYPE(None)(lambda x=Nasty(): None)
+        
+
 try:
     WINFUNCTYPE
 except NameError:
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -65,6 +65,9 @@
 Extension Modules
 -----------------
  
+- Issue #12483: ctypes: Fix a crash when the destruction of a callback
+  object triggers the garbage collector.
+
 - Issue #12950: Fix passing file descriptors in multiprocessing, under
   OpenIndiana/Illumos.
 
diff --git a/Modules/_ctypes/callbacks.c b/Modules/_ctypes/callbacks.c
--- a/Modules/_ctypes/callbacks.c
+++ b/Modules/_ctypes/callbacks.c
@@ -13,6 +13,7 @@
 CThunkObject_dealloc(PyObject *_self)
 {
     CThunkObject *self = (CThunkObject *)_self;
+    PyObject_GC_UnTrack(self);
     Py_XDECREF(self->converters);
     Py_XDECREF(self->callable);
     Py_XDECREF(self->restype);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 21:20:20 2011
From: python-checkins at python.org (amaury.forgeotdarc)
Date: Mon, 12 Sep 2011 21:20:20 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_3=2E2=3A_Issue_=2312483=3A_ctypes=3A_Fix_a_crash_when_?=
 =?utf8?q?the_destruction_of_a_callback?=
Message-ID: <E1R3C3I-0002fJ-W9@dinsdale.python.org>

http://hg.python.org/cpython/rev/fe125a3fda54
changeset:   72367:fe125a3fda54
parent:      72364:a03da3dc478e
parent:      72366:eae8e4ab0455
user:        Amaury Forgeot d'Arc <amauryfa at gmail.com>
date:        Mon Sep 12 21:09:12 2011 +0200
summary:
  Merge 3.2: Issue #12483: ctypes: Fix a crash when the destruction of a callback
  object triggers the garbage collector.

files:
  Lib/ctypes/test/test_callbacks.py |  8 ++++++++
  Misc/NEWS                         |  3 +++
  Modules/_ctypes/callbacks.c       |  1 +
  3 files changed, 12 insertions(+), 0 deletions(-)


diff --git a/Lib/ctypes/test/test_callbacks.py b/Lib/ctypes/test/test_callbacks.py
--- a/Lib/ctypes/test/test_callbacks.py
+++ b/Lib/ctypes/test/test_callbacks.py
@@ -134,6 +134,14 @@
                 if isinstance(x, X)]
         self.assertEqual(len(live), 0)
 
+    def test_issue12483(self):
+        import gc
+        class Nasty:
+            def __del__(self):
+                gc.collect()
+        CFUNCTYPE(None)(lambda x=Nasty(): None)
+        
+
 try:
     WINFUNCTYPE
 except NameError:
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1274,6 +1274,9 @@
 Extension Modules
 -----------------
 
+- Issue #12483: ctypes: Fix a crash when the destruction of a callback
+  object triggers the garbage collector.
+
 - Issue #12950: Fix passing file descriptors in multiprocessing, under
   OpenIndiana/Illumos.
 
diff --git a/Modules/_ctypes/callbacks.c b/Modules/_ctypes/callbacks.c
--- a/Modules/_ctypes/callbacks.c
+++ b/Modules/_ctypes/callbacks.c
@@ -13,6 +13,7 @@
 CThunkObject_dealloc(PyObject *_self)
 {
     CThunkObject *self = (CThunkObject *)_self;
+    PyObject_GC_UnTrack(self);
     Py_XDECREF(self->converters);
     Py_XDECREF(self->callable);
     Py_XDECREF(self->restype);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 21:20:27 2011
From: python-checkins at python.org (amaury.forgeotdarc)
Date: Mon, 12 Sep 2011 21:20:27 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_trailing_spaces?=
Message-ID: <E1R3C3P-0002fT-Lm@dinsdale.python.org>

http://hg.python.org/cpython/rev/c3fd4f956020
changeset:   72368:c3fd4f956020
user:        Amaury Forgeot d'Arc <amauryfa at gmail.com>
date:        Mon Sep 12 21:17:09 2011 +0200
summary:
  Remove trailing spaces

files:
  Lib/ctypes/test/test_callbacks.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/ctypes/test/test_callbacks.py b/Lib/ctypes/test/test_callbacks.py
--- a/Lib/ctypes/test/test_callbacks.py
+++ b/Lib/ctypes/test/test_callbacks.py
@@ -140,7 +140,7 @@
             def __del__(self):
                 gc.collect()
         CFUNCTYPE(None)(lambda x=Nasty(): None)
-        
+
 
 try:
     WINFUNCTYPE

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 12 21:20:31 2011
From: python-checkins at python.org (amaury.forgeotdarc)
Date: Mon, 12 Sep 2011 21:20:31 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Remove_trailing?=
	=?utf8?q?_spaces?=
Message-ID: <E1R3C3T-0002fY-50@dinsdale.python.org>

http://hg.python.org/cpython/rev/a698ad2741da
changeset:   72369:a698ad2741da
branch:      2.7
parent:      72365:eb9f566fd8db
user:        Amaury Forgeot d'Arc <amauryfa at gmail.com>
date:        Mon Sep 12 21:19:53 2011 +0200
summary:
  Remove trailing spaces

files:
  Lib/ctypes/test/test_callbacks.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/ctypes/test/test_callbacks.py b/Lib/ctypes/test/test_callbacks.py
--- a/Lib/ctypes/test/test_callbacks.py
+++ b/Lib/ctypes/test/test_callbacks.py
@@ -146,7 +146,7 @@
             def __del__(self):
                 gc.collect()
         CFUNCTYPE(None)(lambda x=Nasty(): None)
-        
+
 
 try:
     WINFUNCTYPE

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep 13 00:42:43 2011
From: python-checkins at python.org (senthil.kumaran)
Date: Tue, 13 Sep 2011 00:42:43 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Add_the_quote?=
	=?utf8?q?=5Fplus_call_in_the_test=2E?=
Message-ID: <E1R3FD9-0004JD-5y@dinsdale.python.org>

http://hg.python.org/cpython/rev/8f6d958f95c2
changeset:   72370:8f6d958f95c2
branch:      3.2
parent:      72366:eae8e4ab0455
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Tue Sep 13 06:40:27 2011 +0800
summary:
  Add the quote_plus call in the test.

files:
  Lib/test/test_urllib.py |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_urllib.py b/Lib/test/test_urllib.py
--- a/Lib/test/test_urllib.py
+++ b/Lib/test/test_urllib.py
@@ -490,6 +490,7 @@
         result = urllib.parse.quote(partial_quote)
         self.assertEqual(expected, result,
                          "using quote(): %r != %r" % (expected, result))
+        result = urllib.parse.quote_plus(partial_quote)
         self.assertEqual(expected, result,
                          "using quote_plus(): %r != %r" % (expected, result))
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep 13 00:42:43 2011
From: python-checkins at python.org (senthil.kumaran)
Date: Tue, 13 Sep 2011 00:42:43 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_merge_from_3=2E2_-_Add_the_missing_quote=5Fplus_call=2E_Fix_?=
 =?utf8?q?closes_Issue12924?=
Message-ID: <E1R3FD9-0004JH-T7@dinsdale.python.org>

http://hg.python.org/cpython/rev/e25526865339
changeset:   72371:e25526865339
parent:      72368:c3fd4f956020
parent:      72370:8f6d958f95c2
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Tue Sep 13 06:41:43 2011 +0800
summary:
  merge from 3.2 - Add the missing quote_plus call. Fix closes Issue12924

files:
  Lib/test/test_urllib.py |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_urllib.py b/Lib/test/test_urllib.py
--- a/Lib/test/test_urllib.py
+++ b/Lib/test/test_urllib.py
@@ -490,6 +490,7 @@
         result = urllib.parse.quote(partial_quote)
         self.assertEqual(expected, result,
                          "using quote(): %r != %r" % (expected, result))
+        result = urllib.parse.quote_plus(partial_quote)
         self.assertEqual(expected, result,
                          "using quote_plus(): %r != %r" % (expected, result))
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep 13 00:42:44 2011
From: python-checkins at python.org (senthil.kumaran)
Date: Tue, 13 Sep 2011 00:42:44 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Port_the_fix_fo?=
 =?utf8?q?r_Issue12924_=28missing_quote=5Fplus=29_to_2=2E7_branch=2E?=
Message-ID: <E1R3FDA-0004JL-Ji@dinsdale.python.org>

http://hg.python.org/cpython/rev/e8d8eb9e05fd
changeset:   72372:e8d8eb9e05fd
branch:      2.7
parent:      72369:a698ad2741da
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Tue Sep 13 06:42:21 2011 +0800
summary:
  Port the fix for Issue12924 (missing quote_plus) to 2.7 branch.

files:
  Lib/test/test_urllib.py |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_urllib.py b/Lib/test/test_urllib.py
--- a/Lib/test/test_urllib.py
+++ b/Lib/test/test_urllib.py
@@ -407,6 +407,7 @@
         result = urllib.quote(partial_quote)
         self.assertEqual(expected, result,
                          "using quote(): %s != %s" % (expected, result))
+        result = urllib.quote_plus(partial_quote)
         self.assertEqual(expected, result,
                          "using quote_plus(): %s != %s" % (expected, result))
         self.assertRaises(TypeError, urllib.quote, None)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep 13 01:14:59 2011
From: python-checkins at python.org (senthil.kumaran)
Date: Tue, 13 Sep 2011 01:14:59 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_issue12938_?=
 =?utf8?q?-_Update_the_docstring_of_html=2Eescape=2E_Include_the_informati?=
 =?utf8?q?on?=
Message-ID: <E1R3FiN-0006j6-LH@dinsdale.python.org>

http://hg.python.org/cpython/rev/bc5b96c92770
changeset:   72373:bc5b96c92770
branch:      3.2
parent:      72370:8f6d958f95c2
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Tue Sep 13 07:14:13 2011 +0800
summary:
  Fix issue12938 - Update the docstring of html.escape. Include the information on single quote.

files:
  Lib/html/__init__.py |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Lib/html/__init__.py b/Lib/html/__init__.py
--- a/Lib/html/__init__.py
+++ b/Lib/html/__init__.py
@@ -13,7 +13,8 @@
     """
     Replace special characters "&", "<" and ">" to HTML-safe sequences.
     If the optional flag quote is true (the default), the quotation mark
-    character (") is also translated.
+    characters, both double quote (") and single quote (') characters are also
+    translated.
     """
     if quote:
         return s.translate(_escape_map_full)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep 13 01:15:00 2011
From: python-checkins at python.org (senthil.kumaran)
Date: Tue, 13 Sep 2011 01:15:00 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_merge_from_3=2E2?=
Message-ID: <E1R3FiO-0006jc-MZ@dinsdale.python.org>

http://hg.python.org/cpython/rev/60f2739564f3
changeset:   72374:60f2739564f3
parent:      72371:e25526865339
parent:      72373:bc5b96c92770
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Tue Sep 13 07:14:39 2011 +0800
summary:
  merge from 3.2

files:
  Lib/html/__init__.py |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Lib/html/__init__.py b/Lib/html/__init__.py
--- a/Lib/html/__init__.py
+++ b/Lib/html/__init__.py
@@ -13,7 +13,8 @@
     """
     Replace special characters "&", "<" and ">" to HTML-safe sequences.
     If the optional flag quote is true (the default), the quotation mark
-    character (") is also translated.
+    characters, both double quote (") and single quote (') characters are also
+    translated.
     """
     if quote:
         return s.translate(_escape_map_full)

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Tue Sep 13 05:18:32 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Tue, 13 Sep 2011 05:18:32 +0200
Subject: [Python-checkins] Daily reference leaks (60f2739564f3): sum=0
Message-ID: <E1R3JW4-0004ks-Pq@ap.vmr.nerim.net>

results for 60f2739564f3 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogTMgThe', '-x']

From python-checkins at python.org  Tue Sep 13 13:43:49 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 13 Sep 2011 13:43:49 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Let=E2=80=99s_not_use_in?=
	=?utf8?q?put_in_2=2Ex?=
Message-ID: <E1R3RP3-0002Yx-8w@dinsdale.python.org>

http://hg.python.org/distutils2/rev/467abb891c1a
changeset:   1142:467abb891c1a
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 12 01:40:43 2011 +0200
summary:
  Let?s not use input in 2.x

files:
  distutils2/command/register.py |  12 ++++++------
  distutils2/util.py             |   2 +-
  2 files changed, 7 insertions(+), 7 deletions(-)


diff --git a/distutils2/command/register.py b/distutils2/command/register.py
--- a/distutils2/command/register.py
+++ b/distutils2/command/register.py
@@ -144,7 +144,7 @@
  4. quit
 Your selection [default 1]: ''')
 
-            choice = input()
+            choice = raw_input()
             if not choice:
                 choice = '1'
             elif choice not in choices:
@@ -153,7 +153,7 @@
         if choice == '1':
             # get the username and password
             while not username:
-                username = input('Username: ')
+                username = raw_input('Username: ')
             while not password:
                 password = getpass.getpass('Password: ')
 
@@ -179,7 +179,7 @@
                         get_pypirc_path())
                     choice = 'X'
                     while choice.lower() not in 'yn':
-                        choice = input('Save your login (y/N)?')
+                        choice = raw_input('Save your login (y/N)?')
                         if not choice:
                             choice = 'n'
                     if choice.lower() == 'y':
@@ -190,7 +190,7 @@
             data['name'] = data['password'] = data['email'] = ''
             data['confirm'] = None
             while not data['name']:
-                data['name'] = input('Username: ')
+                data['name'] = raw_input('Username: ')
             while data['password'] != data['confirm']:
                 while not data['password']:
                     data['password'] = getpass.getpass('Password: ')
@@ -201,7 +201,7 @@
                     data['confirm'] = None
                     print "Password and confirm don't match!"
             while not data['email']:
-                data['email'] = input('   EMail: ')
+                data['email'] = raw_input('   EMail: ')
             code, result = self.post_to_server(data)
             if code != 200:
                 logger.info('server response (%s): %s', code, result)
@@ -212,7 +212,7 @@
             data = {':action': 'password_reset'}
             data['email'] = ''
             while not data['email']:
-                data['email'] = input('Your email address: ')
+                data['email'] = raw_input('Your email address: ')
             code, result = self.post_to_server(data)
             logger.info('server response (%s): %s', code, result)
 
diff --git a/distutils2/util.py b/distutils2/util.py
--- a/distutils2/util.py
+++ b/distutils2/util.py
@@ -1156,7 +1156,7 @@
 def ask(message, options):
     """Prompt the user with *message*; *options* contains allowed responses."""
     while True:
-        response = input(message)
+        response = raw_input(message)
         response = response.strip().lower()
         if response not in options:
             print 'invalid response:', repr(response)

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Tue Sep 13 13:43:49 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 13 Sep 2011 13:43:49 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Let_the_test_suite_run_a?=
	=?utf8?q?gain?=
Message-ID: <E1R3RP3-0002Z2-9z@dinsdale.python.org>

http://hg.python.org/distutils2/rev/a8ef48dacaff
changeset:   1143:a8ef48dacaff
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 12 02:13:02 2011 +0200
summary:
  Let the test suite run again

files:
  distutils2/_backport/tests/test_sysconfig.py  |  11 +++-
  distutils2/command/install_data.py            |   2 +-
  distutils2/compiler/bcppcompiler.py           |   6 +-
  distutils2/tests/__init__.py                  |   8 +++
  distutils2/tests/support.py                   |   5 ++
  distutils2/tests/test_command_install_dist.py |  10 ++--
  distutils2/tests/test_command_register.py     |  12 ++--
  distutils2/util.py                            |  22 +++++-----
  8 files changed, 45 insertions(+), 31 deletions(-)


diff --git a/distutils2/_backport/tests/test_sysconfig.py b/distutils2/_backport/tests/test_sysconfig.py
--- a/distutils2/_backport/tests/test_sysconfig.py
+++ b/distutils2/_backport/tests/test_sysconfig.py
@@ -15,7 +15,7 @@
         get_scheme_names, _main, _SCHEMES)
 
 from distutils2.tests import unittest, TESTFN, unlink
-from distutils2.tests.support import EnvironGuard
+from distutils2.tests.support import EnvironRestorer
 from test.test_support import TESTFN, unlink
 
 try:
@@ -24,7 +24,10 @@
     skip_unless_symlink = unittest.skip(
         'requires test.test_support.skip_unless_symlink')
 
-class TestSysConfig(EnvironGuard, unittest.TestCase):
+
+class TestSysConfig(EnvironRestorer, unittest.TestCase):
+
+    restore_environ = ['MACOSX_DEPLOYMENT_TARGET', 'PATH']
 
     def setUp(self):
         super(TestSysConfig, self).setUp()
@@ -245,8 +248,8 @@
         # On Windows, the EXE needs to know where pythonXY.dll is at so we have
         # to add the directory to the path.
         if sys.platform == 'win32':
-            os.environ['Path'] = ';'.join((
-                os.path.dirname(sys.executable), os.environ['Path']))
+            os.environ['PATH'] = ';'.join((
+                os.path.dirname(sys.executable), os.environ['PATH']))
 
         # Issue 7880
         def get(python):
diff --git a/distutils2/command/install_data.py b/distutils2/command/install_data.py
--- a/distutils2/command/install_data.py
+++ b/distutils2/command/install_data.py
@@ -4,10 +4,10 @@
 
 import os, sys
 from shutil import Error
-from sysconfig import get_paths, format_value
 from distutils2 import logger
 from distutils2.util import convert_path
 from distutils2.command.cmd import Command
+from distutils2._backport.sysconfig import get_paths, format_value
 
 
 class install_data(Command):
diff --git a/distutils2/compiler/bcppcompiler.py b/distutils2/compiler/bcppcompiler.py
--- a/distutils2/compiler/bcppcompiler.py
+++ b/distutils2/compiler/bcppcompiler.py
@@ -351,7 +351,5 @@
                 self.mkpath(os.path.dirname(output_file))
             try:
                 self.spawn(pp_args)
-            except PackagingExecError:
-                msg = sys.exc_info()[1]
-                print(msg)
-                raise CompileError(msg)
+            except PackagingExecError, exc:
+                raise CompileError(exc)
diff --git a/distutils2/tests/__init__.py b/distutils2/tests/__init__.py
--- a/distutils2/tests/__init__.py
+++ b/distutils2/tests/__init__.py
@@ -16,6 +16,7 @@
 import os
 import sys
 import unittest2 as unittest
+from distutils2.tests.support import TESTFN
 
 # XXX move helpers to support, add tests for them, remove things that
 # duplicate test.support (or keep them for the backport; needs thinking)
@@ -130,3 +131,10 @@
         del sys.modules[name]
     except KeyError:
         pass
+
+
+def unlink(filename):
+    try:
+        os.unlink(filename)
+    except OSError:
+        pass
diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -503,3 +503,8 @@
     except KeyError:
         pass
 
+try:
+    from test.test_support import skip_unless_symlink
+except ImportError:
+    skip_unless_symlink = unittest.skip(
+        'requires test.test_support.skip_unless_symlink')
diff --git a/distutils2/tests/test_command_install_dist.py b/distutils2/tests/test_command_install_dist.py
--- a/distutils2/tests/test_command_install_dist.py
+++ b/distutils2/tests/test_command_install_dist.py
@@ -3,17 +3,17 @@
 import os
 import sys
 
-from sysconfig import (get_scheme_names, get_config_vars,
-                       _SCHEMES, get_config_var, get_path)
-
-_CONFIG_VARS = get_config_vars()
-
 from distutils2.command.install_dist import install_dist
 from distutils2.dist import Distribution
 from distutils2.errors import PackagingOptionError
 
 from distutils2.tests import unittest, support
 
+from distutils2._backport.sysconfig import (
+    get_scheme_names, get_config_vars, _SCHEMES, get_config_var, get_path)
+
+_CONFIG_VARS = get_config_vars()
+
 
 class InstallTestCase(support.TempdirManager,
                       support.LoggingCatcher,
diff --git a/distutils2/tests/test_command_register.py b/distutils2/tests/test_command_register.py
--- a/distutils2/tests/test_command_register.py
+++ b/distutils2/tests/test_command_register.py
@@ -120,7 +120,7 @@
         # Password : 'password'
         # Save your login (y/N)? : 'y'
         inputs = Inputs('1', 'tarek', 'y')
-        register_module.input = inputs
+        register_module.raw_input = inputs
         cmd.ensure_finalized()
         cmd.run()
 
@@ -168,7 +168,7 @@
         # this test runs choice 2
         cmd = self._get_cmd()
         inputs = Inputs('2', 'tarek', 'tarek at ziade.org')
-        register_module.input = inputs
+        register_module.raw_input = inputs
         # let's run the command
         # FIXME does this send a real request? use a mock server
         cmd.ensure_finalized()
@@ -185,7 +185,7 @@
         # this test runs choice 3
         cmd = self._get_cmd()
         inputs = Inputs('3', 'tarek at ziade.org')
-        register_module.input = inputs
+        register_module.raw_input = inputs
         cmd.ensure_finalized()
         cmd.run()
 
@@ -208,7 +208,7 @@
         cmd.ensure_finalized()
         cmd.strict = True
         inputs = Inputs('1', 'tarek', 'y')
-        register_module.input = inputs
+        register_module.raw_input = inputs
         self.assertRaises(PackagingSetupError, cmd.run)
 
         # metadata is OK but long_description is broken
@@ -229,7 +229,7 @@
         cmd.ensure_finalized()
         cmd.strict = True
         inputs = Inputs('1', 'tarek', 'y')
-        register_module.input = inputs
+        register_module.raw_input = inputs
         cmd.ensure_finalized()
         cmd.run()
 
@@ -237,7 +237,7 @@
         cmd = self._get_cmd()
         cmd.ensure_finalized()
         inputs = Inputs('1', 'tarek', 'y')
-        register_module.input = inputs
+        register_module.raw_input = inputs
         cmd.ensure_finalized()
         cmd.run()
 
diff --git a/distutils2/util.py b/distutils2/util.py
--- a/distutils2/util.py
+++ b/distutils2/util.py
@@ -1,34 +1,34 @@
 """Miscellaneous utility functions."""
 
-import codecs
 import os
 import re
 import csv
 import sys
 import errno
+import codecs
 import shutil
 import string
-try:
-    import hashlib
-except ImportError: #<2.5
-    from _backport import hashlib
 import tarfile
 import zipfile
 import posixpath
 import subprocess
 import sysconfig
-try:
-    from glob import iglob as std_iglob
-except ImportError:#<2.5
-    from glob import glob as std_iglob
 from fnmatch import fnmatchcase
 from inspect import getsource
 from ConfigParser import RawConfigParser
+try:
+    from glob import iglob as std_iglob
+except ImportError:
+    from glob import glob as std_iglob
+try:
+    import hashlib
+except ImportError:
+    from distutils2._backport import hashlib
 
 from distutils2 import logger
 from distutils2.errors import (PackagingPlatformError, PackagingFileError,
-                              PackagingByteCompileError, PackagingExecError,
-                              InstallationException, PackagingInternalError)
+                               PackagingByteCompileError, PackagingExecError,
+                               InstallationException, PackagingInternalError)
 
 _PLATFORM = None
 _DEFAULT_INSTALLER = 'distutils2'

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Tue Sep 13 13:43:49 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 13 Sep 2011 13:43:49 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Convert_print_function_c?=
 =?utf8?q?alls_back_to_print_statements=2E?=
Message-ID: <E1R3RP3-0002Yo-8F@dinsdale.python.org>

http://hg.python.org/distutils2/rev/61f2bd107258
changeset:   1141:61f2bd107258
parent:      1133:20f69a01ae38
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 12 01:39:51 2011 +0200
summary:
  Convert print function calls back to print statements.

This caused pysetup to print out tuples.  When multi-line strings are
enclosed in parens for line-wrapping purposes, which would not print out
a tuple, I have added a space for clarity anyway.

files:
  distutils2/_backport/tests/test_sysconfig.py |   2 +-
  distutils2/command/register.py               |   4 +-
  distutils2/create.py                         |  38 +++++-----
  distutils2/depgraph.py                       |  20 ++--
  distutils2/dist.py                           |  24 +++---
  distutils2/run.py                            |  38 +++++-----
  distutils2/tests/support.py                  |   6 +-
  distutils2/tests/test_command_build_ext.py   |   4 +-
  distutils2/util.py                           |   4 +-
  9 files changed, 70 insertions(+), 70 deletions(-)


diff --git a/distutils2/_backport/tests/test_sysconfig.py b/distutils2/_backport/tests/test_sysconfig.py
--- a/distutils2/_backport/tests/test_sysconfig.py
+++ b/distutils2/_backport/tests/test_sysconfig.py
@@ -251,7 +251,7 @@
         # Issue 7880
         def get(python):
             cmd = [python, '-c',
-                   'import sysconfig; print(sysconfig.get_platform())']
+                   'import sysconfig; print sysconfig.get_platform()']
             p = subprocess.Popen(cmd, stdout=subprocess.PIPE, env=os.environ)
             return p.communicate()
         real = os.path.realpath(sys.executable)
diff --git a/distutils2/command/register.py b/distutils2/command/register.py
--- a/distutils2/command/register.py
+++ b/distutils2/command/register.py
@@ -148,7 +148,7 @@
             if not choice:
                 choice = '1'
             elif choice not in choices:
-                print('Please choose one of the four options!')
+                print 'Please choose one of the four options!'
 
         if choice == '1':
             # get the username and password
@@ -199,7 +199,7 @@
                 if data['password'] != data['confirm']:
                     data['password'] = ''
                     data['confirm'] = None
-                    print("Password and confirm don't match!")
+                    print "Password and confirm don't match!"
             while not data['email']:
                 data['email'] = input('   EMail: ')
             code, result = self.post_to_server(data)
diff --git a/distutils2/create.py b/distutils2/create.py
--- a/distutils2/create.py
+++ b/distutils2/create.py
@@ -130,7 +130,7 @@
         if answer and answer[0].lower() in 'yn':
             return answer[0].lower()
 
-        print('\nERROR: You must select "Y" or "N".\n')
+        print '\nERROR: You must select "Y" or "N".\n'
 
 
 def ask(question, default=None, helptext=None, required=True,
@@ -154,19 +154,19 @@
 
         line = sys.stdin.readline().strip()
         if line == '?':
-            print('=' * 70)
-            print(helptext)
-            print('=' * 70)
+            print '=' * 70
+            print helptext
+            print '=' * 70
             continue
         if default and not line:
             return default
         if not line and required:
-            print('*' * 70)
-            print('This value cannot be empty.')
-            print('===========================')
+            print '*' * 70
+            print 'This value cannot be empty.'
+            print '==========================='
             if helptext:
-                print(helptext)
-            print('*' * 70)
+                print helptext
+            print '*' * 70
             continue
         return line
 
@@ -273,9 +273,9 @@
     def _write_cfg(self):
         if os.path.exists(_FILENAME):
             if os.path.exists('%s.old' % _FILENAME):
-                print("ERROR: %(name)s.old backup exists, please check that "
-                      "current %(name)s is correct and remove %(name)s.old" %
-                      {'name': _FILENAME})
+                print ('ERROR: %(name)s.old backup exists, please check that '
+                       'current %(name)s is correct and remove %(name)s.old' %
+                       {'name': _FILENAME})
                 return
             shutil.move(_FILENAME, '%s.old' % _FILENAME)
 
@@ -324,7 +324,7 @@
         fp.close()
 
         os.chmod(_FILENAME, 00644)
-        print('Wrote "%s".' % _FILENAME)
+        print 'Wrote %r.' % _FILENAME
 
     def convert_py_to_cfg(self):
         """Generate a setup.cfg from an existing setup.py.
@@ -631,8 +631,8 @@
                         break
 
             if len(found_list) == 0:
-                print('ERROR: Could not find a matching license for "%s"' %
-                      license)
+                print ('ERROR: Could not find a matching license for "%s"' %
+                       license)
                 continue
 
             question = 'Matching licenses:\n\n'
@@ -653,8 +653,8 @@
             try:
                 index = found_list[int(choice) - 1]
             except ValueError:
-                print("ERROR: Invalid selection, type a number from the list "
-                      "above.")
+                print ('ERROR: Invalid selection, type a number from the list '
+                       'above.')
 
             classifiers.add(_CLASSIFIERS_LIST[index])
 
@@ -677,8 +677,8 @@
                     classifiers.add(key)
                     return
                 except (IndexError, ValueError):
-                    print("ERROR: Invalid selection, type a single digit "
-                          "number.")
+                    print ('ERROR: Invalid selection, type a single digit '
+                           'number.')
 
 
 def main():
diff --git a/distutils2/depgraph.py b/distutils2/depgraph.py
--- a/distutils2/depgraph.py
+++ b/distutils2/depgraph.py
@@ -238,19 +238,19 @@
         e = sys.exc_info()[1]
         tempout.seek(0)
         tempout = tempout.read()
-        print(u'Could not generate the graph')
-        print(tempout)
-        print(e)
+        print 'Could not generate the graph'
+        print tempout
+        print e
         sys.exit(1)
 
     for dist, reqs in graph.missing.items():
         if len(reqs) > 0:
-            print(u"Warning: Missing dependencies for %r:" % dist.name,
-                  ", ".join(reqs))
+            print 'Warning: Missing dependencies for %r:' % dist.name, \
+                  ', '.join(reqs)
     # XXX replace with argparse
     if len(sys.argv) == 1:
-        print(u'Dependency graph:')
-        print(u'   ', repr(graph).replace(u'\n', u'\n    '))
+        print 'Dependency graph:'
+        print '   ', repr(graph).replace('\n', '\n    ')
         sys.exit(0)
     elif len(sys.argv) > 1 and sys.argv[1] in ('-d', '--dot'):
         if len(sys.argv) > 2:
@@ -263,11 +263,11 @@
         f.close()
         tempout.seek(0)
         tempout = tempout.read()
-        print(tempout)
-        print('Dot file written at %r' % filename)
+        print tempout
+        print 'Dot file written at %r' % filename
         sys.exit(0)
     else:
-        print('Supported option: -d [filename]')
+        print 'Supported option: -d [filename]'
         sys.exit(1)
 
 
diff --git a/distutils2/dist.py b/distutils2/dist.py
--- a/distutils2/dist.py
+++ b/distutils2/dist.py
@@ -511,14 +511,14 @@
                 options = self.global_options
             parser.set_option_table(options)
             parser.print_help(self.common_usage + "\nGlobal options:")
-            print(u'')
+            print
 
         if display_options:
             parser.set_option_table(self.display_options)
             parser.print_help(
                 "Information display options (just display " +
                 "information, ignore any commands)")
-            print(u'')
+            print
 
         for command in self.commands:
             if isinstance(command, type) and issubclass(command, Command):
@@ -531,9 +531,9 @@
             else:
                 parser.set_option_table(cls.user_options)
             parser.print_help("Options for %r command:" % cls.__name__)
-            print(u'')
+            print
 
-        print(gen_usage(self.script_name))
+        print gen_usage(self.script_name)
 
     def handle_display_options(self, option_order):
         """If there were any non-global "display-only" options
@@ -546,8 +546,8 @@
         # we ignore "foo bar").
         if self.help_commands:
             self.print_commands()
-            print()
-            print(gen_usage(self.script_name))
+            print
+            print gen_usage(self.script_name)
             return 1
 
         # If user supplied any of the "display metadata" options, then
@@ -563,12 +563,12 @@
                 opt = opt.replace('-', '_')
                 value = self.metadata[opt]
                 if opt in ('keywords', 'platform'):
-                    print(','.join(value))
+                    print ','.join(value)
                 elif opt in ('classifier', 'provides', 'requires',
                              'obsoletes'):
-                    print('\n'.join(value))
+                    print '\n'.join(value)
                 else:
-                    print(value)
+                    print value
                 any_display_options = True
 
         return any_display_options
@@ -577,14 +577,14 @@
         """Print a subset of the list of all commands -- used by
         'print_commands()'.
         """
-        print(header + ":")
+        print header + ":"
 
         for cmd in commands:
             cls = self.cmdclass.get(cmd) or get_command_class(cmd)
             description = getattr(cls, 'description',
                                   '(no description available)')
 
-            print("  %-*s  %s" % (max_length, cmd, description))
+            print "  %-*s  %s" % (max_length, cmd, description)
 
     def _get_command_groups(self):
         """Helper function to retrieve all the command class names divided
@@ -614,7 +614,7 @@
                                 "Standard commands",
                                 max_length)
         if extra_commands:
-            print()
+            print
             self.print_command_list(extra_commands,
                                     "Extra commands",
                                     max_length)
diff --git a/distutils2/run.py b/distutils2/run.py
--- a/distutils2/run.py
+++ b/distutils2/run.py
@@ -199,7 +199,7 @@
         def wrapper(*args, **kwargs):
             f_args = args[1]
             if '--help' in f_args or '-h' in f_args:
-                print(self.help_msg)
+                print self.help_msg
                 return
             return f(*args, **kwargs)
         return wrapper
@@ -227,7 +227,7 @@
     else:
         dists = get_distributions(use_egg_info=True)
         graph = generate_graph(dists)
-        print(graph.repr_node(dist))
+        print graph.repr_node(dist)
 
 
 @action_help(install_usage)
@@ -279,13 +279,13 @@
 
     for key in keys:
         if key in metadata:
-            print(metadata._convert_name(key) + ':')
+            print metadata._convert_name(key) + ':'
             value = metadata[key]
             if isinstance(value, list):
                 for v in value:
-                    print('   ', v)
+                    print '   ', v
             else:
-                print('   ', value.replace('\n', '\n    '))
+                print '   ', value.replace('\n', '\n    ')
 
 
 @action_help(remove_usage)
@@ -315,14 +315,14 @@
     commands = STANDARD_COMMANDS  # + extra commands
 
     if args == ['--list-commands']:
-        print('List of available commands:')
+        print 'List of available commands:'
         cmds = sorted(commands)
 
         for cmd in cmds:
             cls = dispatcher.cmdclass.get(cmd) or get_command_class(cmd)
             desc = getattr(cls, 'description',
                             '(no description available)')
-            print('  %s: %s' % (cmd, desc))
+            print '  %s: %s' % (cmd, desc)
         return
 
     while args:
@@ -360,7 +360,7 @@
 
     number = 0
     for dist in results:
-        print('%r %s (from %r)' % (dist.name, dist.version, dist.path))
+        print '%r %s (from %r)' % (dist.name, dist.version, dist.path)
         number += 1
 
     if number == 0:
@@ -573,18 +573,18 @@
         # late import because of mutual dependence between these modules
         from distutils2.command.cmd import Command
 
-        print('Usage: pysetup [options] action [action_options]')
-        print()
+        print 'Usage: pysetup [options] action [action_options]'
+        print
         if global_options_:
             self.print_usage(self.parser)
-            print()
+            print
 
         if display_options_:
             parser.set_option_table(display_options)
             parser.print_help(
                 "Information display options (just display " +
                 "information, ignore any commands)")
-            print()
+            print
 
         for command in commands:
             if isinstance(command, type) and issubclass(command, Command):
@@ -598,15 +598,15 @@
                 parser.set_option_table(cls.user_options)
 
             parser.print_help("Options for %r command:" % cls.__name__)
-            print()
+            print
 
     def _show_command_help(self, command):
         if isinstance(command, basestring):
             command = get_command_class(command)
 
         desc = getattr(command, 'description', '(no description available)')
-        print('Description:', desc)
-        print()
+        print 'Description:', desc
+        print
 
         if (hasattr(command, 'help_options') and
             isinstance(command.help_options, list)):
@@ -616,7 +616,7 @@
             self.parser.set_option_table(command.user_options)
 
         self.parser.print_help("Options:")
-        print()
+        print
 
     def _get_command_groups(self):
         """Helper function to retrieve all the command class names divided
@@ -643,7 +643,7 @@
 
         self.print_command_list(std_commands, "Standard commands", max_length)
         if extra_commands:
-            print()
+            print
             self.print_command_list(extra_commands, "Extra commands",
                                     max_length)
 
@@ -651,14 +651,14 @@
         """Print a subset of the list of all commands -- used by
         'print_commands()'.
         """
-        print(header + ":")
+        print header + ":"
 
         for cmd in commands:
             cls = self.cmdclass.get(cmd) or get_command_class(cmd)
             description = getattr(cls, 'description',
                                   '(no description available)')
 
-            print("  %-*s  %s" % (max_length, cmd, description))
+            print "  %-*s  %s" % (max_length, cmd, description)
 
     def __call__(self):
         if self.action is None:
diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -326,9 +326,9 @@
         except UnicodeEncodeError:
             pass
         else:
-            print('WARNING: The filename %r CAN be encoded by the filesystem encoding (%s). '
-                  'Unicode filename tests may not be effective'
-                  % (TESTFN_UNENCODABLE, TESTFN_ENCODING))
+            print ('WARNING: The filename %r CAN be encoded by the filesystem encoding (%s). '
+                   'Unicode filename tests may not be effective'
+                   % (TESTFN_UNENCODABLE, TESTFN_ENCODING))
             TESTFN_UNENCODABLE = None
 # Mac OS X denies unencodable filenames (invalid utf-8)
 elif sys.platform != 'darwin':
diff --git a/distutils2/tests/test_command_build_ext.py b/distutils2/tests/test_command_build_ext.py
--- a/distutils2/tests/test_command_build_ext.py
+++ b/distutils2/tests/test_command_build_ext.py
@@ -465,8 +465,8 @@
     src = _get_source_filename()
     if not os.path.exists(src):
         if verbose:
-            print('test_command_build_ext: Cannot find source code (test'
-                  ' must run in python build dir)')
+            print ('test_command_build_ext: Cannot find source code (test'
+                   ' must run in python build dir)')
         return unittest.TestSuite()
     else:
         return unittest.makeSuite(BuildExtTestCase)
diff --git a/distutils2/util.py b/distutils2/util.py
--- a/distutils2/util.py
+++ b/distutils2/util.py
@@ -1159,8 +1159,8 @@
         response = input(message)
         response = response.strip().lower()
         if response not in options:
-            print('invalid response:', repr(response))
-            print('choose one of', ', '.join(repr(o) for o in options))
+            print 'invalid response:', repr(response)
+            print 'choose one of', ', '.join(repr(o) for o in options)
         else:
             return response
 

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Tue Sep 13 13:43:49 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 13 Sep 2011 13:43:49 +0200
Subject: [Python-checkins] =?utf8?q?distutils2_=28merge_default_-=3E_defau?=
 =?utf8?q?lt=29=3A_Branch_merge=2C_reverting_some_of_Alexis=E2=80=99_chang?=
 =?utf8?b?ZXMu?=
Message-ID: <E1R3RP3-0002Z6-BJ@dinsdale.python.org>

http://hg.python.org/distutils2/rev/eee8d2a46e62
changeset:   1144:eee8d2a46e62
parent:      1140:e0098d4649a6
parent:      1143:a8ef48dacaff
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Sep 13 13:28:48 2011 +0200
summary:
  Branch merge, reverting some of Alexis? changes.

The print statement fixes that were in my branch override the print(u'')
calls added by Alexis; the import at function scope were removed for the
usual reasons; d2._backport.hashlib is used if hashlib is not available
instead of md5.

files:
  distutils2/_backport/tests/test_sysconfig.py  |  13 +-
  distutils2/command/install_data.py            |   2 +-
  distutils2/command/register.py                |  16 +-
  distutils2/compiler/bcppcompiler.py           |   6 +-
  distutils2/create.py                          |  38 ++++----
  distutils2/depgraph.py                        |  20 ++--
  distutils2/dist.py                            |  24 ++--
  distutils2/run.py                             |  43 ++++-----
  distutils2/tests/__init__.py                  |   8 +
  distutils2/tests/support.py                   |  11 +-
  distutils2/tests/test_command_build_ext.py    |   4 +-
  distutils2/tests/test_command_install_dist.py |  10 +-
  distutils2/tests/test_command_register.py     |  12 +-
  distutils2/util.py                            |  30 +++---
  14 files changed, 125 insertions(+), 112 deletions(-)


diff --git a/distutils2/_backport/tests/test_sysconfig.py b/distutils2/_backport/tests/test_sysconfig.py
--- a/distutils2/_backport/tests/test_sysconfig.py
+++ b/distutils2/_backport/tests/test_sysconfig.py
@@ -15,7 +15,7 @@
         get_scheme_names, _main, _SCHEMES)
 
 from distutils2.tests import unittest, TESTFN, unlink
-from distutils2.tests.support import EnvironGuard
+from distutils2.tests.support import EnvironRestorer
 from test.test_support import TESTFN, unlink
 
 try:
@@ -24,7 +24,10 @@
     skip_unless_symlink = unittest.skip(
         'requires test.test_support.skip_unless_symlink')
 
-class TestSysConfig(EnvironGuard, unittest.TestCase):
+
+class TestSysConfig(EnvironRestorer, unittest.TestCase):
+
+    restore_environ = ['MACOSX_DEPLOYMENT_TARGET', 'PATH']
 
     def setUp(self):
         super(TestSysConfig, self).setUp()
@@ -245,13 +248,13 @@
         # On Windows, the EXE needs to know where pythonXY.dll is at so we have
         # to add the directory to the path.
         if sys.platform == 'win32':
-            os.environ['Path'] = ';'.join((
-                os.path.dirname(sys.executable), os.environ['Path']))
+            os.environ['PATH'] = ';'.join((
+                os.path.dirname(sys.executable), os.environ['PATH']))
 
         # Issue 7880
         def get(python):
             cmd = [python, '-c',
-                   'import sysconfig; print(sysconfig.get_platform())']
+                   'import sysconfig; print sysconfig.get_platform()']
             p = subprocess.Popen(cmd, stdout=subprocess.PIPE, env=os.environ)
             return p.communicate()
         real = os.path.realpath(sys.executable)
diff --git a/distutils2/command/install_data.py b/distutils2/command/install_data.py
--- a/distutils2/command/install_data.py
+++ b/distutils2/command/install_data.py
@@ -4,10 +4,10 @@
 
 import os, sys
 from shutil import Error
-from distutils2._backport.sysconfig import get_paths, format_value
 from distutils2 import logger
 from distutils2.util import convert_path
 from distutils2.command.cmd import Command
+from distutils2._backport.sysconfig import get_paths, format_value
 
 
 class install_data(Command):
diff --git a/distutils2/command/register.py b/distutils2/command/register.py
--- a/distutils2/command/register.py
+++ b/distutils2/command/register.py
@@ -144,16 +144,16 @@
  4. quit
 Your selection [default 1]: ''')
 
-            choice = input()
+            choice = raw_input()
             if not choice:
                 choice = '1'
             elif choice not in choices:
-                print('Please choose one of the four options!')
+                print 'Please choose one of the four options!'
 
         if choice == '1':
             # get the username and password
             while not username:
-                username = input('Username: ')
+                username = raw_input('Username: ')
             while not password:
                 password = getpass.getpass('Password: ')
 
@@ -179,7 +179,7 @@
                         get_pypirc_path())
                     choice = 'X'
                     while choice.lower() not in 'yn':
-                        choice = input('Save your login (y/N)?')
+                        choice = raw_input('Save your login (y/N)?')
                         if not choice:
                             choice = 'n'
                     if choice.lower() == 'y':
@@ -190,7 +190,7 @@
             data['name'] = data['password'] = data['email'] = ''
             data['confirm'] = None
             while not data['name']:
-                data['name'] = input('Username: ')
+                data['name'] = raw_input('Username: ')
             while data['password'] != data['confirm']:
                 while not data['password']:
                     data['password'] = getpass.getpass('Password: ')
@@ -199,9 +199,9 @@
                 if data['password'] != data['confirm']:
                     data['password'] = ''
                     data['confirm'] = None
-                    print("Password and confirm don't match!")
+                    print "Password and confirm don't match!"
             while not data['email']:
-                data['email'] = input('   EMail: ')
+                data['email'] = raw_input('   EMail: ')
             code, result = self.post_to_server(data)
             if code != 200:
                 logger.info('server response (%s): %s', code, result)
@@ -212,7 +212,7 @@
             data = {':action': 'password_reset'}
             data['email'] = ''
             while not data['email']:
-                data['email'] = input('Your email address: ')
+                data['email'] = raw_input('Your email address: ')
             code, result = self.post_to_server(data)
             logger.info('server response (%s): %s', code, result)
 
diff --git a/distutils2/compiler/bcppcompiler.py b/distutils2/compiler/bcppcompiler.py
--- a/distutils2/compiler/bcppcompiler.py
+++ b/distutils2/compiler/bcppcompiler.py
@@ -351,7 +351,5 @@
                 self.mkpath(os.path.dirname(output_file))
             try:
                 self.spawn(pp_args)
-            except PackagingExecError:
-                msg = sys.exc_info()[1]
-                print(msg)
-                raise CompileError(msg)
+            except PackagingExecError, exc:
+                raise CompileError(exc)
diff --git a/distutils2/create.py b/distutils2/create.py
--- a/distutils2/create.py
+++ b/distutils2/create.py
@@ -130,7 +130,7 @@
         if answer and answer[0].lower() in 'yn':
             return answer[0].lower()
 
-        print('\nERROR: You must select "Y" or "N".\n')
+        print '\nERROR: You must select "Y" or "N".\n'
 
 
 def ask(question, default=None, helptext=None, required=True,
@@ -154,19 +154,19 @@
 
         line = sys.stdin.readline().strip()
         if line == '?':
-            print('=' * 70)
-            print(helptext)
-            print('=' * 70)
+            print '=' * 70
+            print helptext
+            print '=' * 70
             continue
         if default and not line:
             return default
         if not line and required:
-            print('*' * 70)
-            print('This value cannot be empty.')
-            print('===========================')
+            print '*' * 70
+            print 'This value cannot be empty.'
+            print '==========================='
             if helptext:
-                print(helptext)
-            print('*' * 70)
+                print helptext
+            print '*' * 70
             continue
         return line
 
@@ -273,9 +273,9 @@
     def _write_cfg(self):
         if os.path.exists(_FILENAME):
             if os.path.exists('%s.old' % _FILENAME):
-                print("ERROR: %(name)s.old backup exists, please check that "
-                      "current %(name)s is correct and remove %(name)s.old" %
-                      {'name': _FILENAME})
+                print ('ERROR: %(name)s.old backup exists, please check that '
+                       'current %(name)s is correct and remove %(name)s.old' %
+                       {'name': _FILENAME})
                 return
             shutil.move(_FILENAME, '%s.old' % _FILENAME)
 
@@ -324,7 +324,7 @@
         fp.close()
 
         os.chmod(_FILENAME, 00644)
-        print('Wrote "%s".' % _FILENAME)
+        print 'Wrote %r.' % _FILENAME
 
     def convert_py_to_cfg(self):
         """Generate a setup.cfg from an existing setup.py.
@@ -631,8 +631,8 @@
                         break
 
             if len(found_list) == 0:
-                print('ERROR: Could not find a matching license for "%s"' %
-                      license)
+                print ('ERROR: Could not find a matching license for "%s"' %
+                       license)
                 continue
 
             question = 'Matching licenses:\n\n'
@@ -653,8 +653,8 @@
             try:
                 index = found_list[int(choice) - 1]
             except ValueError:
-                print("ERROR: Invalid selection, type a number from the list "
-                      "above.")
+                print ('ERROR: Invalid selection, type a number from the list '
+                       'above.')
 
             classifiers.add(_CLASSIFIERS_LIST[index])
 
@@ -677,8 +677,8 @@
                     classifiers.add(key)
                     return
                 except (IndexError, ValueError):
-                    print("ERROR: Invalid selection, type a single digit "
-                          "number.")
+                    print ('ERROR: Invalid selection, type a single digit '
+                           'number.')
 
 
 def main():
diff --git a/distutils2/depgraph.py b/distutils2/depgraph.py
--- a/distutils2/depgraph.py
+++ b/distutils2/depgraph.py
@@ -238,19 +238,19 @@
         e = sys.exc_info()[1]
         tempout.seek(0)
         tempout = tempout.read()
-        print(u'Could not generate the graph')
-        print(tempout)
-        print(e)
+        print 'Could not generate the graph'
+        print tempout
+        print e
         sys.exit(1)
 
     for dist, reqs in graph.missing.items():
         if len(reqs) > 0:
-            print(u"Warning: Missing dependencies for %r:" % dist.name,
-                  ", ".join(reqs))
+            print 'Warning: Missing dependencies for %r:' % dist.name, \
+                  ', '.join(reqs)
     # XXX replace with argparse
     if len(sys.argv) == 1:
-        print(u'Dependency graph:')
-        print(u'   ', repr(graph).replace(u'\n', u'\n    '))
+        print 'Dependency graph:'
+        print '   ', repr(graph).replace('\n', '\n    ')
         sys.exit(0)
     elif len(sys.argv) > 1 and sys.argv[1] in ('-d', '--dot'):
         if len(sys.argv) > 2:
@@ -263,11 +263,11 @@
         f.close()
         tempout.seek(0)
         tempout = tempout.read()
-        print(tempout)
-        print('Dot file written at %r' % filename)
+        print tempout
+        print 'Dot file written at %r' % filename
         sys.exit(0)
     else:
-        print('Supported option: -d [filename]')
+        print 'Supported option: -d [filename]'
         sys.exit(1)
 
 
diff --git a/distutils2/dist.py b/distutils2/dist.py
--- a/distutils2/dist.py
+++ b/distutils2/dist.py
@@ -511,14 +511,14 @@
                 options = self.global_options
             parser.set_option_table(options)
             parser.print_help(self.common_usage + "\nGlobal options:")
-            print(u'')
+            print
 
         if display_options:
             parser.set_option_table(self.display_options)
             parser.print_help(
                 "Information display options (just display " +
                 "information, ignore any commands)")
-            print(u'')
+            print
 
         for command in self.commands:
             if isinstance(command, type) and issubclass(command, Command):
@@ -531,9 +531,9 @@
             else:
                 parser.set_option_table(cls.user_options)
             parser.print_help("Options for %r command:" % cls.__name__)
-            print(u'')
+            print
 
-        print(gen_usage(self.script_name))
+        print gen_usage(self.script_name)
 
     def handle_display_options(self, option_order):
         """If there were any non-global "display-only" options
@@ -546,8 +546,8 @@
         # we ignore "foo bar").
         if self.help_commands:
             self.print_commands()
-            print(u'')
-            print(gen_usage(self.script_name))
+            print
+            print gen_usage(self.script_name)
             return 1
 
         # If user supplied any of the "display metadata" options, then
@@ -563,12 +563,12 @@
                 opt = opt.replace('-', '_')
                 value = self.metadata[opt]
                 if opt in ('keywords', 'platform'):
-                    print(','.join(value))
+                    print ','.join(value)
                 elif opt in ('classifier', 'provides', 'requires',
                              'obsoletes'):
-                    print('\n'.join(value))
+                    print '\n'.join(value)
                 else:
-                    print(value)
+                    print value
                 any_display_options = True
 
         return any_display_options
@@ -577,14 +577,14 @@
         """Print a subset of the list of all commands -- used by
         'print_commands()'.
         """
-        print(header + ":")
+        print header + ":"
 
         for cmd in commands:
             cls = self.cmdclass.get(cmd) or get_command_class(cmd)
             description = getattr(cls, 'description',
                                   '(no description available)')
 
-            print("  %-*s  %s" % (max_length, cmd, description))
+            print "  %-*s  %s" % (max_length, cmd, description)
 
     def _get_command_groups(self):
         """Helper function to retrieve all the command class names divided
@@ -614,7 +614,7 @@
                                 "Standard commands",
                                 max_length)
         if extra_commands:
-            print(u'')
+            print
             self.print_command_list(extra_commands,
                                     "Extra commands",
                                     max_length)
diff --git a/distutils2/run.py b/distutils2/run.py
--- a/distutils2/run.py
+++ b/distutils2/run.py
@@ -10,7 +10,9 @@
 from distutils2.dist import Distribution
 from distutils2.util import _is_archive_file, generate_setup_py
 from distutils2.command import get_command_class, STANDARD_COMMANDS
+from distutils2.install import install, install_local_project, remove
 from distutils2.database import get_distribution, get_distributions
+from distutils2.depgraph import generate_graph
 from distutils2.fancy_getopt import FancyGetopt
 from distutils2.errors import (PackagingArgError, PackagingError,
                               PackagingModuleError, PackagingClassError,
@@ -197,7 +199,7 @@
         def wrapper(*args, **kwargs):
             f_args = args[1]
             if '--help' in f_args or '-h' in f_args:
-                print(self.help_msg)
+                print self.help_msg
                 return
             return f(*args, **kwargs)
         return wrapper
@@ -217,7 +219,6 @@
 
 @action_help(graph_usage)
 def _graph(dispatcher, args, **kw):
-    from distutils2.depgraph import generate_graph
     name = args[1]
     dist = get_distribution(name, use_egg_info=True)
     if dist is None:
@@ -226,12 +227,11 @@
     else:
         dists = get_distributions(use_egg_info=True)
         graph = generate_graph(dists)
-        print(graph.repr_node(dist))
+        print graph.repr_node(dist)
 
 
 @action_help(install_usage)
 def _install(dispatcher, args, **kw):
-    from distutils2.install import install, install_local_project
     # first check if we are in a source directory
     if len(args) < 2:
         # are we inside a project dir?
@@ -279,18 +279,17 @@
 
     for key in keys:
         if key in metadata:
-            print(metadata._convert_name(key) + ':')
+            print metadata._convert_name(key) + ':'
             value = metadata[key]
             if isinstance(value, list):
                 for v in value:
-                    print('   ', v)
+                    print '   ', v
             else:
-                print('   ', value.replace('\n', '\n    '))
+                print '   ', value.replace('\n', '\n    ')
 
 
 @action_help(remove_usage)
 def _remove(distpatcher, args, **kw):
-    from distutils2.install import remove
     opts = _parse_args(args[1:], 'y', [])
     if 'y' in opts:
         auto_confirm = True
@@ -316,14 +315,14 @@
     commands = STANDARD_COMMANDS  # + extra commands
 
     if args == ['--list-commands']:
-        print('List of available commands:')
+        print 'List of available commands:'
         cmds = sorted(commands)
 
         for cmd in cmds:
             cls = dispatcher.cmdclass.get(cmd) or get_command_class(cmd)
             desc = getattr(cls, 'description',
                             '(no description available)')
-            print('  %s: %s' % (cmd, desc))
+            print '  %s: %s' % (cmd, desc)
         return
 
     while args:
@@ -361,7 +360,7 @@
 
     number = 0
     for dist in results:
-        print('%r %s (from %r)' % (dist.name, dist.version, dist.path))
+        print '%r %s (from %r)' % (dist.name, dist.version, dist.path)
         number += 1
 
     if number == 0:
@@ -574,18 +573,18 @@
         # late import because of mutual dependence between these modules
         from distutils2.command.cmd import Command
 
-        print('Usage: pysetup [options] action [action_options]')
-        print(u'')
+        print 'Usage: pysetup [options] action [action_options]'
+        print
         if global_options_:
             self.print_usage(self.parser)
-            print(u'')
+            print
 
         if display_options_:
             parser.set_option_table(display_options)
             parser.print_help(
                 "Information display options (just display " +
                 "information, ignore any commands)")
-            print(u'')
+            print
 
         for command in commands:
             if isinstance(command, type) and issubclass(command, Command):
@@ -599,15 +598,15 @@
                 parser.set_option_table(cls.user_options)
 
             parser.print_help("Options for %r command:" % cls.__name__)
-            print(u'')
+            print
 
     def _show_command_help(self, command):
         if isinstance(command, basestring):
             command = get_command_class(command)
 
         desc = getattr(command, 'description', '(no description available)')
-        print('Description:', desc)
-        print(u'')
+        print 'Description:', desc
+        print
 
         if (hasattr(command, 'help_options') and
             isinstance(command.help_options, list)):
@@ -617,7 +616,7 @@
             self.parser.set_option_table(command.user_options)
 
         self.parser.print_help("Options:")
-        print(u'')
+        print
 
     def _get_command_groups(self):
         """Helper function to retrieve all the command class names divided
@@ -644,7 +643,7 @@
 
         self.print_command_list(std_commands, "Standard commands", max_length)
         if extra_commands:
-            print(u'')
+            print
             self.print_command_list(extra_commands, "Extra commands",
                                     max_length)
 
@@ -652,14 +651,14 @@
         """Print a subset of the list of all commands -- used by
         'print_commands()'.
         """
-        print(header + ":")
+        print header + ":"
 
         for cmd in commands:
             cls = self.cmdclass.get(cmd) or get_command_class(cmd)
             description = getattr(cls, 'description',
                                   '(no description available)')
 
-            print("  %-*s  %s" % (max_length, cmd, description))
+            print "  %-*s  %s" % (max_length, cmd, description)
 
     def __call__(self):
         if self.action is None:
diff --git a/distutils2/tests/__init__.py b/distutils2/tests/__init__.py
--- a/distutils2/tests/__init__.py
+++ b/distutils2/tests/__init__.py
@@ -16,6 +16,7 @@
 import os
 import sys
 import unittest2 as unittest
+from distutils2.tests.support import TESTFN
 
 # XXX move helpers to support, add tests for them, remove things that
 # duplicate test.support (or keep them for the backport; needs thinking)
@@ -130,3 +131,10 @@
         del sys.modules[name]
     except KeyError:
         pass
+
+
+def unlink(filename):
+    try:
+        os.unlink(filename)
+    except OSError:
+        pass
diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -326,9 +326,9 @@
         except UnicodeEncodeError:
             pass
         else:
-            print('WARNING: The filename %r CAN be encoded by the filesystem encoding (%s). '
-                  'Unicode filename tests may not be effective'
-                  % (TESTFN_UNENCODABLE, TESTFN_ENCODING))
+            print ('WARNING: The filename %r CAN be encoded by the filesystem encoding (%s). '
+                   'Unicode filename tests may not be effective'
+                   % (TESTFN_UNENCODABLE, TESTFN_ENCODING))
             TESTFN_UNENCODABLE = None
 # Mac OS X denies unencodable filenames (invalid utf-8)
 elif sys.platform != 'darwin':
@@ -503,3 +503,8 @@
     except KeyError:
         pass
 
+try:
+    from test.test_support import skip_unless_symlink
+except ImportError:
+    skip_unless_symlink = unittest.skip(
+        'requires test.test_support.skip_unless_symlink')
diff --git a/distutils2/tests/test_command_build_ext.py b/distutils2/tests/test_command_build_ext.py
--- a/distutils2/tests/test_command_build_ext.py
+++ b/distutils2/tests/test_command_build_ext.py
@@ -465,8 +465,8 @@
     src = _get_source_filename()
     if not os.path.exists(src):
         if verbose:
-            print('test_command_build_ext: Cannot find source code (test'
-                  ' must run in python build dir)')
+            print ('test_command_build_ext: Cannot find source code (test'
+                   ' must run in python build dir)')
         return unittest.TestSuite()
     else:
         return unittest.makeSuite(BuildExtTestCase)
diff --git a/distutils2/tests/test_command_install_dist.py b/distutils2/tests/test_command_install_dist.py
--- a/distutils2/tests/test_command_install_dist.py
+++ b/distutils2/tests/test_command_install_dist.py
@@ -3,17 +3,17 @@
 import os
 import sys
 
-from sysconfig import (get_scheme_names, get_config_vars,
-                       _SCHEMES, get_config_var, get_path)
-
-_CONFIG_VARS = get_config_vars()
-
 from distutils2.command.install_dist import install_dist
 from distutils2.dist import Distribution
 from distutils2.errors import PackagingOptionError
 
 from distutils2.tests import unittest, support
 
+from distutils2._backport.sysconfig import (
+    get_scheme_names, get_config_vars, _SCHEMES, get_config_var, get_path)
+
+_CONFIG_VARS = get_config_vars()
+
 
 class InstallTestCase(support.TempdirManager,
                       support.LoggingCatcher,
diff --git a/distutils2/tests/test_command_register.py b/distutils2/tests/test_command_register.py
--- a/distutils2/tests/test_command_register.py
+++ b/distutils2/tests/test_command_register.py
@@ -120,7 +120,7 @@
         # Password : 'password'
         # Save your login (y/N)? : 'y'
         inputs = Inputs('1', 'tarek', 'y')
-        register_module.input = inputs
+        register_module.raw_input = inputs
         cmd.ensure_finalized()
         cmd.run()
 
@@ -168,7 +168,7 @@
         # this test runs choice 2
         cmd = self._get_cmd()
         inputs = Inputs('2', 'tarek', 'tarek at ziade.org')
-        register_module.input = inputs
+        register_module.raw_input = inputs
         # let's run the command
         # FIXME does this send a real request? use a mock server
         cmd.ensure_finalized()
@@ -185,7 +185,7 @@
         # this test runs choice 3
         cmd = self._get_cmd()
         inputs = Inputs('3', 'tarek at ziade.org')
-        register_module.input = inputs
+        register_module.raw_input = inputs
         cmd.ensure_finalized()
         cmd.run()
 
@@ -208,7 +208,7 @@
         cmd.ensure_finalized()
         cmd.strict = True
         inputs = Inputs('1', 'tarek', 'y')
-        register_module.input = inputs
+        register_module.raw_input = inputs
         self.assertRaises(PackagingSetupError, cmd.run)
 
         # metadata is OK but long_description is broken
@@ -229,7 +229,7 @@
         cmd.ensure_finalized()
         cmd.strict = True
         inputs = Inputs('1', 'tarek', 'y')
-        register_module.input = inputs
+        register_module.raw_input = inputs
         cmd.ensure_finalized()
         cmd.run()
 
@@ -237,7 +237,7 @@
         cmd = self._get_cmd()
         cmd.ensure_finalized()
         inputs = Inputs('1', 'tarek', 'y')
-        register_module.input = inputs
+        register_module.raw_input = inputs
         cmd.ensure_finalized()
         cmd.run()
 
diff --git a/distutils2/util.py b/distutils2/util.py
--- a/distutils2/util.py
+++ b/distutils2/util.py
@@ -1,33 +1,33 @@
 """Miscellaneous utility functions."""
 
-import codecs
 import os
 import re
 import csv
 import sys
 import errno
+import codecs
 import shutil
 import string
-try:
-    from hashlib import md5
-except ImportError: #<2.5
-    from md5 import md5
 import tarfile
 import zipfile
 import posixpath
 import subprocess
-try:
-    from glob import iglob as std_iglob
-except ImportError:#<2.5
-    from glob import glob as std_iglob
 from fnmatch import fnmatchcase
 from inspect import getsource
 from ConfigParser import RawConfigParser
+try:
+    from glob import iglob as std_iglob
+except ImportError:
+    from glob import glob as std_iglob
+try:
+    import hashlib
+except ImportError:
+    from distutils2._backport import hashlib
 
 from distutils2 import logger
 from distutils2.errors import (PackagingPlatformError, PackagingFileError,
-                              PackagingByteCompileError, PackagingExecError,
-                              InstallationException, PackagingInternalError)
+                               PackagingByteCompileError, PackagingExecError,
+                               InstallationException, PackagingInternalError)
 from distutils2._backport import sysconfig
 
 _PLATFORM = None
@@ -1156,11 +1156,11 @@
 def ask(message, options):
     """Prompt the user with *message*; *options* contains allowed responses."""
     while True:
-        response = input(message)
+        response = raw_input(message)
         response = response.strip().lower()
         if response not in options:
-            print('invalid response:', repr(response))
-            print('choose one of', ', '.join(repr(o) for o in options))
+            print 'invalid response:', repr(response)
+            print 'choose one of', ', '.join(repr(o) for o in options)
         else:
             return response
 
@@ -1212,7 +1212,7 @@
             # do not put size and md5 hash, as in PEP-376
             writer.writerow((fpath, '', ''))
         else:
-            hash = md5()
+            hash = hashlib.md5()
             fp = open(fpath, 'rb')
             hash.update(fp.read())
             fp.close()

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Tue Sep 13 13:43:49 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 13 Sep 2011 13:43:49 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Remove_obsolete_files=2E?=
Message-ID: <E1R3RP3-0002ZA-D2@dinsdale.python.org>

http://hg.python.org/distutils2/rev/ee75e8d9738c
changeset:   1145:ee75e8d9738c
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Sep 13 13:30:34 2011 +0200
summary:
  Remove obsolete files.

test_distutils2.py does nothing that runtests.py or python -m unittest
does; pkgutil is no longer the home of the PEP 376 implementation
(d2.database is).

files:
  distutils2/_backport/pkgutil.py            |  1222 ----------
  distutils2/_backport/tests/test_pkgutil.py |   608 ----
  test_distutils2.py                         |     5 -
  3 files changed, 0 insertions(+), 1835 deletions(-)


diff --git a/distutils2/_backport/pkgutil.py b/distutils2/_backport/pkgutil.py
deleted file mode 100644
--- a/distutils2/_backport/pkgutil.py
+++ /dev/null
@@ -1,1222 +0,0 @@
-"""Utilities to support packages."""
-
-import imp
-import sys
-
-from csv import reader as csv_reader
-import os
-import re
-from stat import ST_SIZE
-from types import ModuleType
-import warnings
-
-try:
-    from hashlib import md5
-except ImportError:
-    from md5 import md5
-
-from distutils2.errors import DistutilsError
-from distutils2.metadata import Metadata
-from distutils2.version import suggest_normalized_version, VersionPredicate
-try:
-    import cStringIO as StringIO
-except ImportError:
-    import StringIO
-
-
-__all__ = [
-    'get_importer', 'iter_importers', 'get_loader', 'find_loader',
-    'walk_packages', 'iter_modules', 'get_data',
-    'ImpImporter', 'ImpLoader', 'read_code', 'extend_path',
-    'Distribution', 'EggInfoDistribution', 'distinfo_dirname',
-    'get_distributions', 'get_distribution', 'get_file_users',
-    'provides_distribution', 'obsoletes_distribution',
-    'enable_cache', 'disable_cache', 'clear_cache',
-]
-
-
-##########################
-# PEP 302 Implementation #
-##########################
-
-def read_code(stream):
-    # This helper is needed in order for the :pep:`302` emulation to
-    # correctly handle compiled files
-    import marshal
-
-    magic = stream.read(4)
-    if magic != imp.get_magic():
-        return None
-
-    stream.read(4)  # Skip timestamp
-    return marshal.load(stream)
-
-
-def simplegeneric(func):
-    """Make a trivial single-dispatch generic function"""
-    registry = {}
-
-    def wrapper(*args, ** kw):
-        ob = args[0]
-        try:
-            cls = ob.__class__
-        except AttributeError:
-            cls = type(ob)
-        try:
-            mro = cls.__mro__
-        except AttributeError:
-            try:
-
-                class cls(cls, object):
-                    pass
-                mro = cls.__mro__[1:]
-            except TypeError:
-                mro = object, # must be an ExtensionClass or some such  :(
-        for t in mro:
-            if t in registry:
-                return registry[t](*args, ** kw)
-        else:
-            return func(*args, ** kw)
-    try:
-        wrapper.__name__ = func.__name__
-    except (TypeError, AttributeError):
-        pass    # Python 2.3 doesn't allow functions to be renamed
-
-    def register(typ, func=None):
-        if func is None:
-            return lambda f: register(typ, f)
-        registry[typ] = func
-        return func
-
-    wrapper.__dict__ = func.__dict__
-    wrapper.__doc__ = func.__doc__
-    wrapper.register = register
-    return wrapper
-
-
-def walk_packages(path=None, prefix='', onerror=None):
-    """Yields ``(module_loader, name, ispkg)`` for all modules recursively
-    on *path*, or, if *path* is ``None``, all accessible modules.
-
-    :parameter path: should be either ``None`` or a list of paths to look for
-                     modules in.
-    :parameter prefix: is a string to output on the front of every module name
-                       on output.
-
-    Note that this function must import all packages (NOT all
-    modules!) on the given path, in order to access the ``__path__``
-    attribute to find submodules.
-
-    *onerror* is a function which gets called with one argument (the
-    name of the package which was being imported) if any exception
-    occurs while trying to import a package.  If no onerror function is
-    supplied, ``ImportErrors`` are caught and ignored, while all other
-    exceptions are propagated, terminating the search.
-
-    Examples:
-
-    * list all modules python can access::
-
-        walk_packages()
-
-    * list all submodules of ctypes::
-
-        walk_packages(ctypes.__path__, ctypes.__name__+'.')
-
-    """
-
-    def seen(p, m={}):
-        if p in m:
-            return True
-        m[p] = True
-
-    for importer, name, ispkg in iter_modules(path, prefix):
-        yield importer, name, ispkg
-
-        if ispkg:
-            try:
-                __import__(name)
-            except ImportError:
-                if onerror is not None:
-                    onerror(name)
-            except Exception:
-                if onerror is not None:
-                    onerror(name)
-                else:
-                    raise
-            else:
-                path = getattr(sys.modules[name], '__path__', None) or []
-
-                # don't traverse path items we've seen before
-                path = [p for p in path if not seen(p)]
-
-                for item in walk_packages(path, name + '.', onerror):
-                    yield item
-
-
-def iter_modules(path=None, prefix=''):
-    """Yields ``(module_loader, name, ispkg)`` for all submodules on path,
-    or, if *path* is ``None``, all top-level modules on ``sys.path``.
-
-    :parameter path: should be either None or a list of paths to look for
-                     modules in.
-    :parameter prefix: is a string to output on the front of every module name
-                       on output.
-
-    """
-
-    if path is None:
-        importers = iter_importers()
-    else:
-        importers = map(get_importer, path)
-
-    yielded = {}
-    for i in importers:
-        for name, ispkg in iter_importer_modules(i, prefix):
-            if name not in yielded:
-                yielded[name] = 1
-                yield i, name, ispkg
-
-
-#@simplegeneric
-def iter_importer_modules(importer, prefix=''):
-    if not hasattr(importer, 'iter_modules'):
-        return []
-    return importer.iter_modules(prefix)
-
-iter_importer_modules = simplegeneric(iter_importer_modules)
-
-
-class ImpImporter(object):
-    """:pep:`302` Importer that wraps Python's "classic" import algorithm
-
-    ``ImpImporter(dirname)`` produces a :pep:`302` importer that searches that
-    directory. ``ImpImporter(None)`` produces a :pep:`302` importer that
-    searches the current ``sys.path``, plus any modules that are frozen
-    or built-in.
-
-    Note that :class:`ImpImporter` does not currently support being used by
-    placement on ``sys.meta_path``.
-    """
-
-    def __init__(self, path=None):
-        self.path = path
-
-    def find_module(self, fullname, path=None):
-        # Note: we ignore 'path' argument since it is only used via meta_path
-        subname = fullname.split(".")[-1]
-        if subname != fullname and self.path is None:
-            return None
-        if self.path is None:
-            path = None
-        else:
-            path = [os.path.realpath(self.path)]
-        try:
-            file, filename, etc = imp.find_module(subname, path)
-        except ImportError:
-            return None
-        return ImpLoader(fullname, file, filename, etc)
-
-    def iter_modules(self, prefix=''):
-        if self.path is None or not os.path.isdir(self.path):
-            return
-
-        yielded = {}
-        import inspect
-
-        filenames = os.listdir(self.path)
-        filenames.sort()  # handle packages before same-named modules
-
-        for fn in filenames:
-            modname = inspect.getmodulename(fn)
-            if modname == '__init__' or modname in yielded:
-                continue
-
-            path = os.path.join(self.path, fn)
-            ispkg = False
-
-            if not modname and os.path.isdir(path) and '.' not in fn:
-                modname = fn
-                for fn in os.listdir(path):
-                    subname = inspect.getmodulename(fn)
-                    if subname == '__init__':
-                        ispkg = True
-                        break
-                else:
-                    continue    # not a package
-
-            if modname and '.' not in modname:
-                yielded[modname] = 1
-                yield prefix + modname, ispkg
-
-
-class ImpLoader(object):
-    """:pep:`302` Loader that wraps Python's "classic" import algorithm """
-
-    code = source = None
-
-    def __init__(self, fullname, file, filename, etc):
-        self.file = file
-        self.filename = filename
-        self.fullname = fullname
-        self.etc = etc
-
-    def load_module(self, fullname):
-        self._reopen()
-        try:
-            mod = imp.load_module(fullname, self.file, self.filename, self.etc)
-        finally:
-            if self.file:
-                self.file.close()
-        # Note: we don't set __loader__ because we want the module to look
-        # normal; i.e. this is just a wrapper for standard import machinery
-        return mod
-
-    def get_data(self, pathname):
-        return open(pathname, "rb").read()
-
-    def _reopen(self):
-        if self.file and self.file.closed:
-            mod_type = self.etc[2]
-            if mod_type == imp.PY_SOURCE:
-                self.file = open(self.filename, 'rU')
-            elif mod_type in (imp.PY_COMPILED, imp.C_EXTENSION):
-                self.file = open(self.filename, 'rb')
-
-    def _fix_name(self, fullname):
-        if fullname is None:
-            fullname = self.fullname
-        elif fullname != self.fullname:
-            raise ImportError("Loader for module %s cannot handle "
-                              "module %s" % (self.fullname, fullname))
-        return fullname
-
-    def is_package(self, fullname):
-        fullname = self._fix_name(fullname)
-        return self.etc[2] == imp.PKG_DIRECTORY
-
-    def get_code(self, fullname=None):
-        fullname = self._fix_name(fullname)
-        if self.code is None:
-            mod_type = self.etc[2]
-            if mod_type == imp.PY_SOURCE:
-                source = self.get_source(fullname)
-                self.code = compile(source, self.filename, 'exec')
-            elif mod_type == imp.PY_COMPILED:
-                self._reopen()
-                try:
-                    self.code = read_code(self.file)
-                finally:
-                    self.file.close()
-            elif mod_type == imp.PKG_DIRECTORY:
-                self.code = self._get_delegate().get_code()
-        return self.code
-
-    def get_source(self, fullname=None):
-        fullname = self._fix_name(fullname)
-        if self.source is None:
-            mod_type = self.etc[2]
-            if mod_type == imp.PY_SOURCE:
-                self._reopen()
-                try:
-                    self.source = self.file.read()
-                finally:
-                    self.file.close()
-            elif mod_type == imp.PY_COMPILED:
-                if os.path.exists(self.filename[:-1]):
-                    f = open(self.filename[:-1], 'rU')
-                    self.source = f.read()
-                    f.close()
-            elif mod_type == imp.PKG_DIRECTORY:
-                self.source = self._get_delegate().get_source()
-        return self.source
-
-    def _get_delegate(self):
-        return ImpImporter(self.filename).find_module('__init__')
-
-    def get_filename(self, fullname=None):
-        fullname = self._fix_name(fullname)
-        mod_type = self.etc[2]
-        if mod_type == imp.PKG_DIRECTORY:
-            return self._get_delegate().get_filename()
-        elif mod_type in (imp.PY_SOURCE, imp.PY_COMPILED, imp.C_EXTENSION):
-            return self.filename
-        return None
-
-
-try:
-    import zipimport
-    from zipimport import zipimporter
-
-    def iter_zipimport_modules(importer, prefix=''):
-        dirlist = sorted(zipimport._zip_directory_cache[importer.archive])
-        _prefix = importer.prefix
-        plen = len(_prefix)
-        yielded = {}
-        import inspect
-        for fn in dirlist:
-            if not fn.startswith(_prefix):
-                continue
-
-            fn = fn[plen:].split(os.sep)
-
-            if len(fn) == 2 and fn[1].startswith('__init__.py'):
-                if fn[0] not in yielded:
-                    yielded[fn[0]] = 1
-                    yield fn[0], True
-
-            if len(fn) != 1:
-                continue
-
-            modname = inspect.getmodulename(fn[0])
-            if modname == '__init__':
-                continue
-
-            if modname and '.' not in modname and modname not in yielded:
-                yielded[modname] = 1
-                yield prefix + modname, False
-
-    iter_importer_modules.register(zipimporter, iter_zipimport_modules)
-
-except ImportError:
-    pass
-
-
-def get_importer(path_item):
-    """Retrieve a  :pep:`302` importer for the given path item
-
-    The returned importer is cached in ``sys.path_importer_cache``
-    if it was newly created by a path hook.
-
-    If there is no importer, a wrapper around the basic import
-    machinery is returned. This wrapper is never inserted into
-    the importer cache (``None`` is inserted instead).
-
-    The cache (or part of it) can be cleared manually if a
-    rescan of ``sys.path_hooks`` is necessary.
-    """
-    try:
-        importer = sys.path_importer_cache[path_item]
-    except KeyError:
-        for path_hook in sys.path_hooks:
-            try:
-                importer = path_hook(path_item)
-                break
-            except ImportError:
-                pass
-        else:
-            importer = None
-        sys.path_importer_cache.setdefault(path_item, importer)
-
-    if importer is None:
-        try:
-            importer = ImpImporter(path_item)
-        except ImportError:
-            importer = None
-    return importer
-
-
-def iter_importers(fullname=""):
-    """Yield :pep:`302` importers for the given module name
-
-    If fullname contains a '.', the importers will be for the package
-    containing fullname, otherwise they will be importers for sys.meta_path,
-    sys.path, and Python's "classic" import machinery, in that order.  If
-    the named module is in a package, that package is imported as a side
-    effect of invoking this function.
-
-    Non :pep:`302` mechanisms (e.g. the Windows registry) used by the
-    standard import machinery to find files in alternative locations
-    are partially supported, but are searched AFTER ``sys.path``. Normally,
-    these locations are searched BEFORE sys.path, preventing ``sys.path``
-    entries from shadowing them.
-
-    For this to cause a visible difference in behaviour, there must
-    be a module or package name that is accessible via both sys.path
-    and one of the non :pep:`302` file system mechanisms. In this case,
-    the emulation will find the former version, while the builtin
-    import mechanism will find the latter.
-
-    Items of the following types can be affected by this discrepancy:
-    :data:`imp.C_EXTENSION`, :data:`imp.PY_SOURCE`, :data:`imp.PY_COMPILED`,
-    :data:`imp.PKG_DIRECTORY`
-    """
-    if fullname.startswith('.'):
-        raise ImportError("Relative module names not supported")
-    if '.' in fullname:
-        # Get the containing package's __path__
-        pkg = '.'.join(fullname.split('.')[:-1])
-        if pkg not in sys.modules:
-            __import__(pkg)
-        path = getattr(sys.modules[pkg], '__path__', None) or []
-    else:
-        for importer in sys.meta_path:
-            yield importer
-        path = sys.path
-    for item in path:
-        yield get_importer(item)
-    if '.' not in fullname:
-        yield ImpImporter()
-
-
-def get_loader(module_or_name):
-    """Get a :pep:`302` "loader" object for module_or_name
-
-    If the module or package is accessible via the normal import
-    mechanism, a wrapper around the relevant part of that machinery
-    is returned.  Returns None if the module cannot be found or imported.
-    If the named module is not already imported, its containing package
-    (if any) is imported, in order to establish the package ``__path__``.
-
-    This function uses :func:`iter_importers`, and is thus subject to the same
-    limitations regarding platform-specific special import locations such
-    as the Windows registry.
-    """
-    if module_or_name in sys.modules:
-        module_or_name = sys.modules[module_or_name]
-    if isinstance(module_or_name, ModuleType):
-        module = module_or_name
-        loader = getattr(module, '__loader__', None)
-        if loader is not None:
-            return loader
-        fullname = module.__name__
-    else:
-        fullname = module_or_name
-    return find_loader(fullname)
-
-
-def find_loader(fullname):
-    """Find a :pep:`302` "loader" object for fullname
-
-    If fullname contains dots, path must be the containing package's
-    ``__path__``. Returns ``None`` if the module cannot be found or imported.
-    This function uses :func:`iter_importers`, and is thus subject to the same
-    limitations regarding platform-specific special import locations such as
-    the Windows registry.
-    """
-    for importer in iter_importers(fullname):
-        loader = importer.find_module(fullname)
-        if loader is not None:
-            return loader
-
-    return None
-
-
-def extend_path(path, name):
-    """Extend a package's path.
-
-    Intended use is to place the following code in a package's
-    ``__init__.py``::
-
-        from pkgutil import extend_path
-        __path__ = extend_path(__path__, __name__)
-
-    This will add to the package's ``__path__`` all subdirectories of
-    directories on ``sys.path`` named after the package.  This is useful
-    if one wants to distribute different parts of a single logical
-    package as multiple directories.
-
-    It also looks for ``*.pkg`` files beginning where ``*`` matches the name
-    argument.  This feature is similar to ``*.pth`` files (see ``site.py``),
-    except that it doesn't special-case lines starting with ``import``.
-    A ``*.pkg`` file is trusted at face value: apart from checking for
-    duplicates, all entries found in a ``*.pkg`` file are added to the
-    path, regardless of whether they are exist the filesystem.  (This
-    is a feature.)
-
-    If the input path is not a list (as is the case for frozen
-    packages) it is returned unchanged.  The input path is not
-    modified; an extended copy is returned.  Items are only appended
-    to the copy at the end.
-
-    It is assumed that sys.path is a sequence.  Items of sys.path that
-    are not (unicode or 8-bit) strings referring to existing
-    directories are ignored.  Unicode items of sys.path that cause
-    errors when used as filenames may cause this function to raise an
-    exception (in line with ``os.path.isdir()`` behavior).
-    """
-
-    if not isinstance(path, list):
-        # This could happen e.g. when this is called from inside a
-        # frozen package.  Return the path unchanged in that case.
-        return path
-
-    pname = os.path.join(*name.split('.'))  # Reconstitute as relative path
-    # Just in case os.extsep != '.'
-    sname = os.extsep.join(name.split('.'))
-    sname_pkg = sname + os.extsep + "pkg"
-    init_py = "__init__" + os.extsep + "py"
-
-    path = path[:]  # Start with a copy of the existing path
-
-    for dir in sys.path:
-        if not isinstance(dir, basestring) or not os.path.isdir(dir):
-            continue
-        subdir = os.path.join(dir, pname)
-        # XXX This may still add duplicate entries to path on
-        # case-insensitive filesystems
-        initfile = os.path.join(subdir, init_py)
-        if subdir not in path and os.path.isfile(initfile):
-            path.append(subdir)
-        # XXX Is this the right thing for subpackages like zope.app?
-        # It looks for a file named "zope.app.pkg"
-        pkgfile = os.path.join(dir, sname_pkg)
-        if os.path.isfile(pkgfile):
-            try:
-                f = open(pkgfile)
-            except IOError, msg:
-                sys.stderr.write("Can't open %s: %s\n" %
-                                 (pkgfile, msg))
-            else:
-                for line in f:
-                    line = line.rstrip('\n')
-                    if not line or line.startswith('#'):
-                        continue
-                    path.append(line)  # Don't check for existence!
-                f.close()
-
-    return path
-
-
-def get_data(package, resource):
-    """Get a resource from a package.
-
-    This is a wrapper round the :pep:`302` loader get_data API. The package
-    argument should be the name of a package, in standard module format
-    (``foo.bar``). The resource argument should be in the form of a relative
-    filename, using ``'/'`` as the path separator. The parent directory name
-    ``'..'`` is not allowed, and nor is a rooted name (starting with a
-    ``'/'``).
-
-    The function returns a binary string, which is the contents of the
-    specified resource.
-
-    For packages located in the filesystem, which have already been imported,
-    this is the rough equivalent of::
-
-        d = os.path.dirname(sys.modules[package].__file__)
-        data = open(os.path.join(d, resource), 'rb').read()
-
-    If the package cannot be located or loaded, or it uses a :pep:`302` loader
-    which does not support :func:`get_data`, then ``None`` is returned.
-    """
-
-    loader = get_loader(package)
-    if loader is None or not hasattr(loader, 'get_data'):
-        return None
-    mod = sys.modules.get(package) or loader.load_module(package)
-    if mod is None or not hasattr(mod, '__file__'):
-        return None
-
-    # Modify the resource name to be compatible with the loader.get_data
-    # signature - an os.path format "filename" starting with the dirname of
-    # the package's __file__
-    parts = resource.split('/')
-    parts.insert(0, os.path.dirname(mod.__file__))
-    resource_name = os.path.join(*parts)
-    return loader.get_data(resource_name)
-
-
-##########################
-# PEP 376 Implementation #
-##########################
-
-DIST_FILES = ('INSTALLER', 'METADATA', 'RECORD', 'REQUESTED', 'RESOURCES')
-
-# Cache
-_cache_name = {}  # maps names to Distribution instances
-_cache_name_egg = {}  # maps names to EggInfoDistribution instances
-_cache_path = {}  # maps paths to Distribution instances
-_cache_path_egg = {}  # maps paths to EggInfoDistribution instances
-_cache_generated = False  # indicates if .dist-info distributions are cached
-_cache_generated_egg = False  # indicates if .dist-info and .egg are cached
-_cache_enabled = True
-
-
-def enable_cache():
-    """
-    Enables the internal cache.
-
-    Note that this function will not clear the cache in any case, for that
-    functionality see :func:`clear_cache`.
-    """
-    global _cache_enabled
-
-    _cache_enabled = True
-
-
-def disable_cache():
-    """
-    Disables the internal cache.
-
-    Note that this function will not clear the cache in any case, for that
-    functionality see :func:`clear_cache`.
-    """
-    global _cache_enabled
-
-    _cache_enabled = False
-
-
-def clear_cache():
-    """ Clears the internal cache. """
-    global _cache_name, _cache_name_egg, _cache_path, _cache_path_egg, \
-        _cache_generated, _cache_generated_egg
-
-    _cache_name = {}
-    _cache_name_egg = {}
-    _cache_path = {}
-    _cache_path_egg = {}
-    _cache_generated = False
-    _cache_generated_egg = False
-
-
-def _yield_distributions(include_dist, include_egg, paths=sys.path):
-    """
-    Yield .dist-info and .egg(-info) distributions, based on the arguments
-
-    :parameter include_dist: yield .dist-info distributions
-    :parameter include_egg: yield .egg(-info) distributions
-    """
-    for path in paths:
-        realpath = os.path.realpath(path)
-        if not os.path.isdir(realpath):
-            continue
-        for dir in os.listdir(realpath):
-            dist_path = os.path.join(realpath, dir)
-            if include_dist and dir.endswith('.dist-info'):
-                yield Distribution(dist_path)
-            elif include_egg and (dir.endswith('.egg-info') or
-                                  dir.endswith('.egg')):
-                yield EggInfoDistribution(dist_path)
-
-def _generate_cache(use_egg_info=False, paths=sys.path):
-    global _cache_generated, _cache_generated_egg
-
-    if _cache_generated_egg or (_cache_generated and not use_egg_info):
-        return
-    else:
-        gen_dist = not _cache_generated
-        gen_egg = use_egg_info
-
-        for dist in _yield_distributions(gen_dist, gen_egg, paths):
-            if isinstance(dist, Distribution):
-                _cache_path[dist.path] = dist
-                if not dist.name in _cache_name:
-                    _cache_name[dist.name] = []
-                _cache_name[dist.name].append(dist)
-            else:
-                _cache_path_egg[dist.path] = dist
-                if not dist.name in _cache_name_egg:
-                    _cache_name_egg[dist.name] = []
-                _cache_name_egg[dist.name].append(dist)
-
-        if gen_dist:
-            _cache_generated = True
-        if gen_egg:
-            _cache_generated_egg = True
-
-
-class Distribution(object):
-    """Created with the *path* of the ``.dist-info`` directory provided to the
-    constructor. It reads the metadata contained in ``METADATA`` when it is
-    instantiated."""
-
-    # Attribute documenting for Sphinx style documentation, see for more info:
-    #   http://sphinx.pocoo.org/ext/autodoc.html#dir-autoattribute
-    name = ''
-    """The name of the distribution."""
-    metadata = None
-    """A :class:`distutils2.metadata.Metadata` instance loaded with
-    the distribution's ``METADATA`` file."""
-    requested = False
-    """A boolean that indicates whether the ``REQUESTED`` metadata file is
-    present (in other words, whether the package was installed by user
-    request or it was installed as a dependency)."""
-
-    def __init__(self, path):
-        if _cache_enabled and path in _cache_path:
-            self.metadata = _cache_path[path].metadata
-        else:
-            metadata_path = os.path.join(path, 'METADATA')
-            self.metadata = Metadata(path=metadata_path)
-
-        self.path = path
-        self.name = self.metadata['name']
-
-        if _cache_enabled and not path in _cache_path:
-            _cache_path[path] = self
-
-    def __repr__(self):
-        return '%s-%s at %s' % (self.name, self.metadata.version, self.path)
-
-    def _get_records(self, local=False):
-        RECORD = self.get_distinfo_file('RECORD')
-        record_reader = csv_reader(RECORD, delimiter=',')
-        for row in record_reader:
-            path, md5, size = row[:] + [None for i in xrange(len(row), 3)]
-            if local:
-                path = path.replace('/', os.sep)
-                path = os.path.join(sys.prefix, path)
-            yield path, md5, size
-
-    def get_resource_path(self, relative_path):
-        resources_file = self.get_distinfo_file('RESOURCES')
-        resources_reader = csv_reader(resources_file, delimiter=',')
-        for relative, destination in resources_reader:
-            if relative == relative_path:
-                return destination
-        raise KeyError('No resource file with relative path %s were installed' %
-                       relative_path)
-
-    def get_installed_files(self, local=False):
-        """
-        Iterates over the ``RECORD`` entries and returns a tuple
-        ``(path, md5, size)`` for each line. If *local* is ``True``,
-        the returned path is transformed into a local absolute path.
-        Otherwise the raw value from RECORD is returned.
-
-        A local absolute path is an absolute path in which occurrences of
-        ``'/'`` have been replaced by the system separator given by ``os.sep``.
-
-        :parameter local: flag to say if the path should be returned a local
-                          absolute path
-
-        :type local: boolean
-        :returns: iterator of (path, md5, size)
-        """
-        return self._get_records(local)
-
-    def uses(self, path):
-        """
-        Returns ``True`` if path is listed in ``RECORD``. *path* can be a local
-        absolute path or a relative ``'/'``-separated path.
-
-        :rtype: boolean
-        """
-        for p, md5, size in self._get_records():
-            local_absolute = os.path.join(sys.prefix, p)
-            if path == p or path == local_absolute:
-                return True
-        return False
-
-    def get_distinfo_file(self, path, binary=False):
-        """
-        Returns a file located under the ``.dist-info`` directory. Returns a
-        ``file`` instance for the file pointed by *path*.
-
-        :parameter path: a ``'/'``-separated path relative to the
-                         ``.dist-info`` directory or an absolute path;
-                         If *path* is an absolute path and doesn't start
-                         with the ``.dist-info`` directory path,
-                         a :class:`DistutilsError` is raised
-        :type path: string
-        :parameter binary: If *binary* is ``True``, opens the file in read-only
-                           binary mode (``rb``), otherwise opens it in
-                           read-only mode (``r``).
-        :rtype: file object
-        """
-        open_flags = 'r'
-        if binary:
-            open_flags += 'b'
-
-        # Check if it is an absolute path
-        if path.find(os.sep) >= 0:
-            # it's an absolute path?
-            distinfo_dirname, path = path.split(os.sep)[-2:]
-            if distinfo_dirname != self.path.split(os.sep)[-1]:
-                raise DistutilsError("Requested dist-info file does not "
-                                     "belong to the %s distribution. '%s' was requested." \
-                                     % (self.name, os.sep.join([distinfo_dirname, path])))
-
-        # The file must be relative
-        if path not in DIST_FILES:
-            raise DistutilsError("Requested an invalid dist-info file: "
-                                 "%s" % path)
-
-        # Convert the relative path back to absolute
-        path = os.path.join(self.path, path)
-        return open(path, open_flags)
-
-    def get_distinfo_files(self, local=False):
-        """
-        Iterates over the ``RECORD`` entries and returns paths for each line if
-        the path is pointing to a file located in the ``.dist-info`` directory
-        or one of its subdirectories.
-
-        :parameter local: If *local* is ``True``, each returned path is
-                          transformed into a local absolute path. Otherwise the
-                          raw value from ``RECORD`` is returned.
-        :type local: boolean
-        :returns: iterator of paths
-        """
-        for path, md5, size in self._get_records(local):
-            yield path
-
-    def __eq__(self, other):
-        return isinstance(other, Distribution) and self.path == other.path
-
-    # See http://docs.python.org/reference/datamodel#object.__hash__
-    __hash__ = object.__hash__
-
-
-class EggInfoDistribution(object):
-    """Created with the *path* of the ``.egg-info`` directory or file provided
-    to the constructor. It reads the metadata contained in the file itself, or
-    if the given path happens to be a directory, the metadata is read from the
-    file ``PKG-INFO`` under that directory."""
-
-    name = ''
-    """The name of the distribution."""
-    metadata = None
-    """A :class:`distutils2.metadata.Metadata` instance loaded with
-    the distribution's ``METADATA`` file."""
-    _REQUIREMENT = re.compile(\
-                              r'(?P<name>[-A-Za-z0-9_.]+)\s*' \
-                              r'(?P<first>(?:<|<=|!=|==|>=|>)[-A-Za-z0-9_.]+)?\s*' \
-                              r'(?P<rest>(?:\s*,\s*(?:<|<=|!=|==|>=|>)[-A-Za-z0-9_.]+)*)\s*' \
-                              r'(?P<extras>\[.*\])?')
-
-    def __init__(self, path, display_warnings=False):
-        self.path = path
-        self.display_warnings = display_warnings
-        if _cache_enabled and path in _cache_path_egg:
-            self.metadata = _cache_path_egg[path].metadata
-            self.name = self.metadata['Name']
-            return
-
-        # reused from Distribute's pkg_resources
-        def yield_lines(strs):
-            """Yield non-empty/non-comment lines of a ``basestring``
-            or sequence"""
-            if isinstance(strs, basestring):
-                for s in strs.splitlines():
-                    s = s.strip()
-                    # skip blank lines/comments
-                    if s and not s.startswith('#'):
-                        yield s
-            else:
-                for ss in strs:
-                    for s in yield_lines(ss):
-                        yield s
-
-        requires = None
-        if path.endswith('.egg'):
-            if os.path.isdir(path):
-                meta_path = os.path.join(path, 'EGG-INFO', 'PKG-INFO')
-                self.metadata = Metadata(path=meta_path)
-                try:
-                    req_path = os.path.join(path, 'EGG-INFO', 'requires.txt')
-                    requires = open(req_path, 'r').read()
-                except IOError:
-                    requires = None
-            else:
-                # FIXME handle the case where zipfile is not available
-                zipf = zipimport.zipimporter(path)
-                fileobj = StringIO.StringIO(zipf.get_data('EGG-INFO/PKG-INFO'))
-                self.metadata = Metadata(fileobj=fileobj)
-                try:
-                    requires = zipf.get_data('EGG-INFO/requires.txt')
-                except IOError:
-                    requires = None
-            self.name = self.metadata['Name']
-        elif path.endswith('.egg-info'):
-            if os.path.isdir(path):
-                path = os.path.join(path, 'PKG-INFO')
-                try:
-                    req_f = open(os.path.join(path, 'requires.txt'), 'r')
-                    requires = req_f.read()
-                except IOError:
-                    requires = None
-            self.metadata = Metadata(path=path)
-            self.name = self.metadata['name']
-        else:
-            raise ValueError('The path must end with .egg-info or .egg')
-
-
-        if requires is not None:
-            if self.metadata['Metadata-Version'] == '1.1':
-                # we can't have 1.1 metadata *and* Setuptools requires
-                for field in ('Obsoletes', 'Requires', 'Provides'):
-                    del self.metadata[field]
-
-        reqs = []
-
-        if requires is not None:
-            for line in yield_lines(requires):
-                if line[0] == '[' and self.display_warnings:
-                    warnings.warn('distutils2 does not support extensions '
-                                  'in requires.txt')
-                    break
-                else:
-                    match = self._REQUIREMENT.match(line.strip())
-                    if not match:
-                        # this happens when we encounter extras
-                        # since they are written at the end of the file
-                        # we just exit
-                        break
-                        #raise ValueError('Distribution %s has ill formed '
-                        #                 'requires.txt file (%s)' %
-                        #                 (self.name, line))
-                    else:
-                        if match.group('extras'):
-                            s = (('Distribution %s uses extra requirements '
-                                 'which are not supported in distutils') \
-                                 % (self.name))
-                            warnings.warn(s)
-                        name = match.group('name')
-                        version = None
-                        if match.group('first'):
-                            version = match.group('first')
-                            if match.group('rest'):
-                                version += match.group('rest')
-                            version = version.replace(' ', '')  # trim spaces
-                        if version is None:
-                            reqs.append(name)
-                        else:
-                            reqs.append('%s (%s)' % (name, version))
-
-            if len(reqs) > 0:
-                self.metadata['Requires-Dist'] += reqs
-
-
-        if _cache_enabled:
-            _cache_path_egg[self.path] = self
-
-    def __repr__(self):
-        return '%s-%s at %s' % (self.name, self.metadata.version, self.path)
-
-    def get_installed_files(self, local=False):
-
-        def _md5(path):
-            f = open(path)
-            try:
-                content = f.read()
-            finally:
-                f.close()
-            return md5(content).hexdigest()
-
-        def _size(path):
-            return os.stat(path)[ST_SIZE]
-
-        path = self.path
-        if local:
-            path = path.replace('/', os.sep)
-
-        # XXX What about scripts and data files ?
-        if os.path.isfile(path):
-            return [(path, _md5(path), _size(path))]
-        else:
-            files = []
-            for root, dir, files_ in os.walk(path):
-                for item in files_:
-                    item = os.path.join(root, item)
-                    files.append((item, _md5(item), _size(item)))
-            return files
-
-        return []
-
-    def uses(self, path):
-        return False
-
-    def __eq__(self, other):
-        return isinstance(other, EggInfoDistribution) and \
-            self.path == other.path
-
-    # See http://docs.python.org/reference/datamodel#object.__hash__
-    __hash__ = object.__hash__
-
-
-def distinfo_dirname(name, version):
-    """
-    The *name* and *version* parameters are converted into their
-    filename-escaped form, i.e. any ``'-'`` characters are replaced
-    with ``'_'`` other than the one in ``'dist-info'`` and the one
-    separating the name from the version number.
-
-    :parameter name: is converted to a standard distribution name by replacing
-                     any runs of non- alphanumeric characters with a single
-                     ``'-'``.
-    :type name: string
-    :parameter version: is converted to a standard version string. Spaces
-                        become dots, and all other non-alphanumeric characters
-                        (except dots) become dashes, with runs of multiple
-                        dashes condensed to a single dash.
-    :type version: string
-    :returns: directory name
-    :rtype: string"""
-    file_extension = '.dist-info'
-    name = name.replace('-', '_')
-    normalized_version = suggest_normalized_version(version)
-    # Because this is a lookup procedure, something will be returned even if
-    #   it is a version that cannot be normalized
-    if normalized_version is None:
-        # Unable to achieve normality?
-        normalized_version = version
-    return '-'.join([name, normalized_version]) + file_extension
-
-
-def get_distributions(use_egg_info=False, paths=sys.path):
-    """
-    Provides an iterator that looks for ``.dist-info`` directories in
-    ``sys.path`` and returns :class:`Distribution` instances for each one of
-    them. If the parameters *use_egg_info* is ``True``, then the ``.egg-info``
-    files and directores are iterated as well.
-
-    :rtype: iterator of :class:`Distribution` and :class:`EggInfoDistribution`
-            instances
-    """
-    if not _cache_enabled:
-        for dist in _yield_distributions(True, use_egg_info, paths):
-            yield dist
-    else:
-        _generate_cache(use_egg_info, paths)
-
-        for dist in _cache_path.itervalues():
-            yield dist
-
-        if use_egg_info:
-            for dist in _cache_path_egg.itervalues():
-                yield dist
-
-
-def get_distribution(name, use_egg_info=False, paths=None):
-    """
-    Scans all elements in ``sys.path`` and looks for all directories
-    ending with ``.dist-info``. Returns a :class:`Distribution`
-    corresponding to the ``.dist-info`` directory that contains the
-    ``METADATA`` that matches *name* for the *name* metadata field.
-    If no distribution exists with the given *name* and the parameter
-    *use_egg_info* is set to ``True``, then all files and directories ending
-    with ``.egg-info`` are scanned. A :class:`EggInfoDistribution` instance is
-    returned if one is found that has metadata that matches *name* for the
-    *name* metadata field.
-
-    This function only returns the first result found, as no more than one
-    value is expected. If the directory is not found, ``None`` is returned.
-
-    :rtype: :class:`Distribution` or :class:`EggInfoDistribution` or None
-    """
-    if paths == None:
-        paths = sys.path
-
-    if not _cache_enabled:
-        for dist in _yield_distributions(True, use_egg_info, paths):
-            if dist.name == name:
-                return dist
-    else:
-        _generate_cache(use_egg_info, paths)
-
-        if name in _cache_name:
-            return _cache_name[name][0]
-        elif use_egg_info and name in _cache_name_egg:
-            return _cache_name_egg[name][0]
-        else:
-            return None
-
-
-def obsoletes_distribution(name, version=None, use_egg_info=False):
-    """
-    Iterates over all distributions to find which distributions obsolete
-    *name*.
-
-    If a *version* is provided, it will be used to filter the results.
-    If the argument *use_egg_info* is set to ``True``, then ``.egg-info``
-    distributions will be considered as well.
-
-    :type name: string
-    :type version: string
-    :parameter name:
-    """
-    for dist in get_distributions(use_egg_info):
-        obsoleted = (dist.metadata['Obsoletes-Dist'] +
-                     dist.metadata['Obsoletes'])
-        for obs in obsoleted:
-            o_components = obs.split(' ', 1)
-            if len(o_components) == 1 or version is None:
-                if name == o_components[0]:
-                    yield dist
-                    break
-            else:
-                try:
-                    predicate = VersionPredicate(obs)
-                except ValueError:
-                    raise DistutilsError(('Distribution %s has ill formed' +
-                                         ' obsoletes field') % (dist.name,))
-                if name == o_components[0] and predicate.match(version):
-                    yield dist
-                    break
-
-
-def provides_distribution(name, version=None, use_egg_info=False):
-    """
-    Iterates over all distributions to find which distributions provide *name*.
-    If a *version* is provided, it will be used to filter the results. Scans
-    all elements in ``sys.path``  and looks for all directories ending with
-    ``.dist-info``. Returns a :class:`Distribution`  corresponding to the
-    ``.dist-info`` directory that contains a ``METADATA`` that matches *name*
-    for the name metadata. If the argument *use_egg_info* is set to ``True``,
-    then all files and directories ending with ``.egg-info`` are considered
-    as well and returns an :class:`EggInfoDistribution` instance.
-
-    This function only returns the first result found, since no more than
-    one values are expected. If the directory is not found, returns ``None``.
-
-    :parameter version: a version specifier that indicates the version
-                        required, conforming to the format in ``PEP-345``
-
-    :type name: string
-    :type version: string
-    """
-    predicate = None
-    if not version is None:
-        try:
-            predicate = VersionPredicate(name + ' (' + version + ')')
-        except ValueError:
-            raise DistutilsError('Invalid name or version')
-
-    for dist in get_distributions(use_egg_info):
-        provided = dist.metadata['Provides-Dist'] + dist.metadata['Provides']
-
-        for p in provided:
-            p_components = p.rsplit(' ', 1)
-            if len(p_components) == 1 or predicate is None:
-                if name == p_components[0]:
-                    yield dist
-                    break
-            else:
-                p_name, p_ver = p_components
-                if len(p_ver) < 2 or p_ver[0] != '(' or p_ver[-1] != ')':
-                    raise DistutilsError(('Distribution %s has invalid ' +
-                                         'provides field: %s') \
-                                         % (dist.name, p))
-                p_ver = p_ver[1:-1]  # trim off the parenthesis
-                if p_name == name and predicate.match(p_ver):
-                    yield dist
-                    break
-
-
-def get_file_users(path):
-    """
-    Iterates over all distributions to find out which distributions uses
-    *path*.
-
-    :parameter path: can be a local absolute path or a relative
-                     ``'/'``-separated path.
-    :type path: string
-    :rtype: iterator of :class:`Distribution` instances
-    """
-    for dist in get_distributions():
-        if dist.uses(path):
-            yield dist
-
-def resource_path(distribution_name, relative_path):
-    dist = get_distribution(distribution_name)
-    if dist != None:
-        return dist.get_resource_path(relative_path)
-    raise LookupError('No distribution named %s is installed.' %
-                      distribution_name)
-
-def resource_open(distribution_name, relative_path, * args, ** kwargs):
-    file = open(resource_path(distribution_name, relative_path), * args,
-                ** kwargs)
-    return file
\ No newline at end of file
diff --git a/distutils2/_backport/tests/test_pkgutil.py b/distutils2/_backport/tests/test_pkgutil.py
deleted file mode 100644
--- a/distutils2/_backport/tests/test_pkgutil.py
+++ /dev/null
@@ -1,608 +0,0 @@
-# -*- coding: utf-8 -*-
-"""Tests for PEP 376 pkgutil functionality"""
-import imp
-import sys
-
-import csv
-import os
-import shutil
-import tempfile
-import zipfile
-try:
-    from hashlib import md5
-except ImportError:
-    from distutils2._backport.hashlib import md5
-
-from distutils2.errors import DistutilsError
-from distutils2.metadata import Metadata
-from distutils2.tests import unittest, run_unittest, support, TESTFN
-
-from distutils2._backport import pkgutil
-from distutils2._backport.pkgutil import (
-                                          Distribution, EggInfoDistribution, get_distribution, get_distributions,
-                                          provides_distribution, obsoletes_distribution, get_file_users,
-                                          distinfo_dirname, _yield_distributions)
-
-try:
-    from os.path import relpath
-except ImportError:
-    try:
-        from unittest.compatibility import relpath
-    except ImportError:
-        from unittest2.compatibility import relpath
-
-# Adapted from Python 2.7's trunk
-
-# TODO Add a test for getting a distribution that is provided by another
-# distribution.
-
-# TODO Add a test for absolute pathed RECORD items (e.g. /etc/myapp/config.ini)
-
-
-class TestPkgUtilData(unittest.TestCase):
-
-    def setUp(self):
-        super(TestPkgUtilData, self).setUp()
-        self.dirname = tempfile.mkdtemp()
-        sys.path.insert(0, self.dirname)
-        pkgutil.disable_cache()
-
-    def tearDown(self):
-        super(TestPkgUtilData, self).tearDown()
-        del sys.path[0]
-        pkgutil.enable_cache()
-        shutil.rmtree(self.dirname)
-
-    def test_getdata_filesys(self):
-        pkg = 'test_getdata_filesys'
-
-        # Include a LF and a CRLF, to test that binary data is read back
-        RESOURCE_DATA = 'Hello, world!\nSecond line\r\nThird line'
-
-        # Make a package with some resources
-        package_dir = os.path.join(self.dirname, pkg)
-        os.mkdir(package_dir)
-        # Empty init.py
-        f = open(os.path.join(package_dir, '__init__.py'), "wb")
-        try:
-            pass
-        finally:
-            f.close()
-        # Resource files, res.txt, sub/res.txt
-        f = open(os.path.join(package_dir, 'res.txt'), "wb")
-        try:
-            f.write(RESOURCE_DATA)
-        finally:
-            f.close()
-        os.mkdir(os.path.join(package_dir, 'sub'))
-        f = open(os.path.join(package_dir, 'sub', 'res.txt'), "wb")
-        try:
-            f.write(RESOURCE_DATA)
-        finally:
-            f.close()
-
-        # Check we can read the resources
-        res1 = pkgutil.get_data(pkg, 'res.txt')
-        self.assertEqual(res1, RESOURCE_DATA)
-        res2 = pkgutil.get_data(pkg, 'sub/res.txt')
-        self.assertEqual(res2, RESOURCE_DATA)
-
-        del sys.modules[pkg]
-
-    def test_getdata_zipfile(self):
-        zip = 'test_getdata_zipfile.zip'
-        pkg = 'test_getdata_zipfile'
-
-        # Include a LF and a CRLF, to test that binary data is read back
-        RESOURCE_DATA = 'Hello, world!\nSecond line\r\nThird line'
-
-        # Make a package with some resources
-        zip_file = os.path.join(self.dirname, zip)
-        z = zipfile.ZipFile(zip_file, 'w')
-        try:
-            # Empty init.py
-            z.writestr(pkg + '/__init__.py', "")
-            # Resource files, res.txt, sub/res.txt
-            z.writestr(pkg + '/res.txt', RESOURCE_DATA)
-            z.writestr(pkg + '/sub/res.txt', RESOURCE_DATA)
-        finally:
-            z.close()
-
-        # Check we can read the resources
-        sys.path.insert(0, zip_file)
-        res1 = pkgutil.get_data(pkg, 'res.txt')
-        self.assertEqual(res1, RESOURCE_DATA)
-        res2 = pkgutil.get_data(pkg, 'sub/res.txt')
-        self.assertEqual(res2, RESOURCE_DATA)
-
-        names = []
-        for loader, name, ispkg in pkgutil.iter_modules([zip_file]):
-            names.append(name)
-        self.assertEqual(names, ['test_getdata_zipfile'])
-
-        del sys.path[0]
-
-        del sys.modules[pkg]
-
-# Adapted from Python 2.7's trunk
-
-
-class TestPkgUtilPEP302(unittest.TestCase):
-
-    class MyTestLoader(object):
-
-        def load_module(self, fullname):
-            # Create an empty module
-            mod = sys.modules.setdefault(fullname, imp.new_module(fullname))
-            mod.__file__ = "<%s>" % self.__class__.__name__
-            mod.__loader__ = self
-            # Make it a package
-            mod.__path__ = []
-            # Count how many times the module is reloaded
-            mod.__dict__['loads'] = mod.__dict__.get('loads', 0) + 1
-            return mod
-
-        def get_data(self, path):
-            return "Hello, world!"
-
-    class MyTestImporter(object):
-
-        def find_module(self, fullname, path=None):
-            return TestPkgUtilPEP302.MyTestLoader()
-
-    def setUp(self):
-        super(TestPkgUtilPEP302, self).setUp()
-        pkgutil.disable_cache()
-        sys.meta_path.insert(0, self.MyTestImporter())
-
-    def tearDown(self):
-        del sys.meta_path[0]
-        pkgutil.enable_cache()
-        super(TestPkgUtilPEP302, self).tearDown()
-
-    def test_getdata_pep302(self):
-        # Use a dummy importer/loader
-        self.assertEqual(pkgutil.get_data('foo', 'dummy'), "Hello, world!")
-        del sys.modules['foo']
-
-    def test_alreadyloaded(self):
-        # Ensure that get_data works without reloading - the "loads" module
-        # variable in the example loader should count how many times a reload
-        # occurs.
-        import foo
-        self.assertEqual(foo.loads, 1)
-        self.assertEqual(pkgutil.get_data('foo', 'dummy'), "Hello, world!")
-        self.assertEqual(foo.loads, 1)
-        del sys.modules['foo']
-
-
-class TestPkgUtilDistribution(unittest.TestCase):
-    # Tests the pkgutil.Distribution class
-
-    def setUp(self):
-        super(TestPkgUtilDistribution, self).setUp()
-        self.fake_dists_path = os.path.abspath(
-                                               os.path.join(os.path.dirname(__file__), 'fake_dists'))
-        pkgutil.disable_cache()
-
-        self.distinfo_dirs = [os.path.join(self.fake_dists_path, dir)
-            for dir in os.listdir(self.fake_dists_path)
-            if dir.endswith('.dist-info')]
-
-        def get_hexdigest(file):
-            md5_hash = md5()
-            md5_hash.update(open(file).read())
-            return md5_hash.hexdigest()
-
-        def record_pieces(file):
-            path = relpath(file, sys.prefix)
-            digest = get_hexdigest(file)
-            size = os.path.getsize(file)
-            return [path, digest, size]
-
-        self.records = {}
-        for distinfo_dir in self.distinfo_dirs:
-            # Setup the RECORD file for this dist
-            record_file = os.path.join(distinfo_dir, 'RECORD')
-            record_writer = csv.writer(open(record_file, 'w'), delimiter=',',
-                                       quoting=csv.QUOTE_NONE)
-            dist_location = distinfo_dir.replace('.dist-info', '')
-
-            for path, dirs, files in os.walk(dist_location):
-                for f in files:
-                    record_writer.writerow(record_pieces(
-                                           os.path.join(path, f)))
-            for file in ['INSTALLER', 'METADATA', 'REQUESTED']:
-                record_writer.writerow(record_pieces(
-                                       os.path.join(distinfo_dir, file)))
-            record_writer.writerow([relpath(record_file, sys.prefix)])
-            del record_writer  # causes the RECORD file to close
-            record_reader = csv.reader(open(record_file, 'rb'))
-            record_data = []
-            for row in record_reader:
-                path, md5_, size = row[:] + \
-                    [None for i in xrange(len(row), 3)]
-                record_data.append([path, (md5_, size, )])
-            self.records[distinfo_dir] = dict(record_data)
-
-    def tearDown(self):
-        self.records = None
-        for distinfo_dir in self.distinfo_dirs:
-            record_file = os.path.join(distinfo_dir, 'RECORD')
-            open(record_file, 'w').close()
-        pkgutil.enable_cache()
-        super(TestPkgUtilDistribution, self).tearDown()
-
-    def test_instantiation(self):
-        # Test the Distribution class's instantiation provides us with usable
-        # attributes.
-        here = os.path.abspath(os.path.dirname(__file__))
-        name = 'choxie'
-        version = '2.0.0.9'
-        dist_path = os.path.join(here, 'fake_dists',
-                                 distinfo_dirname(name, version))
-        dist = Distribution(dist_path)
-
-        self.assertEqual(dist.name, name)
-        self.assertTrue(isinstance(dist.metadata, Metadata))
-        self.assertEqual(dist.metadata['version'], version)
-        self.assertTrue(isinstance(dist.requested, type(bool())))
-
-    def test_installed_files(self):
-        # Test the iteration of installed files.
-        # Test the distribution's installed files
-        for distinfo_dir in self.distinfo_dirs:
-            dist = Distribution(distinfo_dir)
-            for path, md5_, size in dist.get_installed_files():
-                record_data = self.records[dist.path]
-                self.assertIn(path, record_data)
-                self.assertEqual(md5_, record_data[path][0])
-                self.assertEqual(size, record_data[path][1])
-
-    def test_uses(self):
-        # Test to determine if a distribution uses a specified file.
-        # Criteria to test against
-        distinfo_name = 'grammar-1.0a4'
-        distinfo_dir = os.path.join(self.fake_dists_path,
-                                    distinfo_name + '.dist-info')
-        true_path = [self.fake_dists_path, distinfo_name, \
-            'grammar', 'utils.py']
-        true_path = relpath(os.path.join(*true_path), sys.prefix)
-        false_path = [self.fake_dists_path, 'towel_stuff-0.1', 'towel_stuff',
-            '__init__.py']
-        false_path = relpath(os.path.join(*false_path), sys.prefix)
-
-        # Test if the distribution uses the file in question
-        dist = Distribution(distinfo_dir)
-        self.assertTrue(dist.uses(true_path))
-        self.assertFalse(dist.uses(false_path))
-
-    def test_get_distinfo_file(self):
-        # Test the retrieval of dist-info file objects.
-        distinfo_name = 'choxie-2.0.0.9'
-        other_distinfo_name = 'grammar-1.0a4'
-        distinfo_dir = os.path.join(self.fake_dists_path,
-                                    distinfo_name + '.dist-info')
-        dist = Distribution(distinfo_dir)
-        # Test for known good file matches
-        distinfo_files = [
-            # Relative paths
-            'INSTALLER', 'METADATA',
-            # Absolute paths
-            os.path.join(distinfo_dir, 'RECORD'),
-            os.path.join(distinfo_dir, 'REQUESTED'),
-            ]
-
-        for distfile in distinfo_files:
-            value = dist.get_distinfo_file(distfile)
-            self.assertTrue(isinstance(value, file))
-            # Is it the correct file?
-            self.assertEqual(value.name, os.path.join(distinfo_dir, distfile))
-
-        # Test an absolute path that is part of another distributions dist-info
-        other_distinfo_file = os.path.join(self.fake_dists_path,
-                                           other_distinfo_name + '.dist-info', 'REQUESTED')
-        self.assertRaises(DistutilsError, dist.get_distinfo_file,
-                          other_distinfo_file)
-        # Test for a file that does not exist and should not exist
-        self.assertRaises(DistutilsError, dist.get_distinfo_file, \
-                          'ENTRYPOINTS')
-
-    def test_get_distinfo_files(self):
-        # Test for the iteration of RECORD path entries.
-        distinfo_name = 'towel_stuff-0.1'
-        distinfo_dir = os.path.join(self.fake_dists_path,
-                                    distinfo_name + '.dist-info')
-        dist = Distribution(distinfo_dir)
-        # Test for the iteration of the raw path
-        distinfo_record_paths = self.records[distinfo_dir].keys()
-        found = [path for path in dist.get_distinfo_files()]
-        self.assertEqual(sorted(found), sorted(distinfo_record_paths))
-        # Test for the iteration of local absolute paths
-        distinfo_record_paths = [os.path.join(sys.prefix, path)
-            for path in self.records[distinfo_dir]]
-        found = [path for path in dist.get_distinfo_files(local=True)]
-        self.assertEqual(sorted(found), sorted(distinfo_record_paths))
-
-    def test_get_resources_path(self):
-        distinfo_name = 'babar-0.1'
-        distinfo_dir = os.path.join(self.fake_dists_path,
-                                    distinfo_name + '.dist-info')
-        dist = Distribution(distinfo_dir)
-        resource_path = dist.get_resource_path('babar.png')
-        self.assertEqual(resource_path, 'babar.png')
-        self.assertRaises(KeyError, dist.get_resource_path, 'notexist')
-
-
-
-class TestPkgUtilPEP376(support.LoggingCatcher, support.WarningsCatcher,
-                        unittest.TestCase):
-    # Tests for the new functionality added in PEP 376.
-
-    def setUp(self):
-        super(TestPkgUtilPEP376, self).setUp()
-        pkgutil.disable_cache()
-        # Setup the path environment with our fake distributions
-        current_path = os.path.abspath(os.path.dirname(__file__))
-        self.sys_path = sys.path[:]
-        self.fake_dists_path = os.path.join(current_path, 'fake_dists')
-        sys.path.insert(0, self.fake_dists_path)
-
-    def tearDown(self):
-        sys.path[:] = self.sys_path
-        pkgutil.enable_cache()
-        super(TestPkgUtilPEP376, self).tearDown()
-
-    def test_distinfo_dirname(self):
-        # Given a name and a version, we expect the distinfo_dirname function
-        # to return a standard distribution information directory name.
-
-        items = [# (name, version, standard_dirname)
-            # Test for a very simple single word name and decimal
-            # version number
-            ('docutils', '0.5', 'docutils-0.5.dist-info'),
-            # Test for another except this time with a '-' in the name, which
-            # needs to be transformed during the name lookup
-            ('python-ldap', '2.5', 'python_ldap-2.5.dist-info'),
-            # Test for both '-' in the name and a funky version number
-            ('python-ldap', '2.5 a---5', 'python_ldap-2.5 a---5.dist-info'),
-            ]
-
-        # Loop through the items to validate the results
-        for name, version, standard_dirname in items:
-            dirname = distinfo_dirname(name, version)
-            self.assertEqual(dirname, standard_dirname)
-
-    def test_get_distributions(self):
-        # Lookup all distributions found in the ``sys.path``.
-        # This test could potentially pick up other installed distributions
-        fake_dists = [('grammar', '1.0a4'), ('choxie', '2.0.0.9'),
-                      ('towel-stuff', '0.1'), ('babar', '0.1')]
-        found_dists = []
-
-        # Verify the fake dists have been found.
-        dists = [dist for dist in get_distributions()]
-        for dist in dists:
-            if not isinstance(dist, Distribution):
-                self.fail("item received was not a Distribution instance: "
-                          "%s" % type(dist))
-            if dist.name in dict(fake_dists) and \
-                dist.path.startswith(self.fake_dists_path):
-                    found_dists.append((dist.name, dist.metadata['version'], ))
-            else:
-                # check that it doesn't find anything more than this
-                self.assertFalse(dist.path.startswith(self.fake_dists_path))
-            # otherwise we don't care what other distributions are found
-
-        # Finally, test that we found all that we were looking for
-        self.assertListEqual(sorted(found_dists), sorted(fake_dists))
-
-        # Now, test if the egg-info distributions are found correctly as well
-        fake_dists += [('bacon', '0.1'), ('cheese', '2.0.2'),
-                       ('coconuts-aster', '10.3'),
-                       ('banana', '0.4'), ('strawberry', '0.6'),
-                       ('truffles', '5.0'), ('nut', 'funkyversion')]
-        found_dists = []
-
-        dists = [dist for dist in get_distributions(use_egg_info=True)]
-        for dist in dists:
-            if not (isinstance(dist, Distribution) or \
-                    isinstance(dist, EggInfoDistribution)):
-                self.fail("item received was not a Distribution or "
-                          "EggInfoDistribution instance: %s" % type(dist))
-            if dist.name in dict(fake_dists) and \
-                dist.path.startswith(self.fake_dists_path):
-                    found_dists.append((dist.name, dist.metadata['version']))
-            else:
-                self.assertFalse(dist.path.startswith(self.fake_dists_path))
-
-        self.assertListEqual(sorted(fake_dists), sorted(found_dists))
-
-    def test_get_distribution(self):
-        # Test for looking up a distribution by name.
-        # Test the lookup of the towel-stuff distribution
-        name = 'towel-stuff'  # Note: This is different from the directory name
-
-        # Lookup the distribution
-        dist = get_distribution(name)
-        self.assertTrue(isinstance(dist, Distribution))
-        self.assertEqual(dist.name, name)
-
-        # Verify that an unknown distribution returns None
-        self.assertEqual(None, get_distribution('bogus'))
-
-        # Verify partial name matching doesn't work
-        self.assertEqual(None, get_distribution('towel'))
-
-        # Verify that it does not find egg-info distributions, when not
-        # instructed to
-        self.assertEqual(None, get_distribution('bacon'))
-        self.assertEqual(None, get_distribution('cheese'))
-        self.assertEqual(None, get_distribution('strawberry'))
-        self.assertEqual(None, get_distribution('banana'))
-
-        # Now check that it works well in both situations, when egg-info
-        # is a file and directory respectively.
-        dist = get_distribution('cheese', use_egg_info=True)
-        self.assertTrue(isinstance(dist, EggInfoDistribution))
-        self.assertEqual(dist.name, 'cheese')
-
-        dist = get_distribution('bacon', use_egg_info=True)
-        self.assertTrue(isinstance(dist, EggInfoDistribution))
-        self.assertEqual(dist.name, 'bacon')
-
-        dist = get_distribution('banana', use_egg_info=True)
-        self.assertTrue(isinstance(dist, EggInfoDistribution))
-        self.assertEqual(dist.name, 'banana')
-
-        dist = get_distribution('strawberry', use_egg_info=True)
-        self.assertTrue(isinstance(dist, EggInfoDistribution))
-        self.assertEqual(dist.name, 'strawberry')
-
-    def test_get_file_users(self):
-        # Test the iteration of distributions that use a file.
-        name = 'towel_stuff-0.1'
-        path = os.path.join(self.fake_dists_path, name,
-                            'towel_stuff', '__init__.py')
-        for dist in get_file_users(path):
-            self.assertTrue(isinstance(dist, Distribution))
-            self.assertEqual(dist.name, name)
-
-    def test_provides(self):
-        # Test for looking up distributions by what they provide
-        checkLists = lambda x, y: self.assertListEqual(sorted(x), sorted(y))
-
-        l = [dist.name for dist in provides_distribution('truffles')]
-        checkLists(l, ['choxie', 'towel-stuff'])
-
-        l = [dist.name for dist in provides_distribution('truffles', '1.0')]
-        checkLists(l, ['choxie'])
-
-        l = [dist.name for dist in provides_distribution('truffles', '1.0',
-                                                         use_egg_info=True)]
-        checkLists(l, ['choxie', 'cheese'])
-
-        l = [dist.name for dist in provides_distribution('truffles', '1.1.2')]
-        checkLists(l, ['towel-stuff'])
-
-        l = [dist.name for dist in provides_distribution('truffles', '1.1')]
-        checkLists(l, ['towel-stuff'])
-
-        l = [dist.name for dist in provides_distribution('truffles', \
-                                                         '!=1.1,<=2.0')]
-        checkLists(l, ['choxie'])
-
-        l = [dist.name for dist in provides_distribution('truffles', \
-                                                         '!=1.1,<=2.0',
-                                                          use_egg_info=True)]
-        checkLists(l, ['choxie', 'bacon', 'cheese'])
-
-        l = [dist.name for dist in provides_distribution('truffles', '>1.0')]
-        checkLists(l, ['towel-stuff'])
-
-        l = [dist.name for dist in provides_distribution('truffles', '>1.5')]
-        checkLists(l, [])
-
-        l = [dist.name for dist in provides_distribution('truffles', '>1.5',
-                                                         use_egg_info=True)]
-        checkLists(l, ['bacon'])
-
-        l = [dist.name for dist in provides_distribution('truffles', '>=1.0')]
-        checkLists(l, ['choxie', 'towel-stuff'])
-
-        l = [dist.name for dist in provides_distribution('strawberry', '0.6',
-                                                         use_egg_info=True)]
-        checkLists(l, ['coconuts-aster'])
-
-        l = [dist.name for dist in provides_distribution('strawberry', '>=0.5',
-                                                         use_egg_info=True)]
-        checkLists(l, ['coconuts-aster'])
-
-        l = [dist.name for dist in provides_distribution('strawberry', '>0.6',
-                                                         use_egg_info=True)]
-        checkLists(l, [])
-
-        l = [dist.name for dist in provides_distribution('banana', '0.4',
-                                                         use_egg_info=True)]
-        checkLists(l, ['coconuts-aster'])
-
-        l = [dist.name for dist in provides_distribution('banana', '>=0.3',
-                                                         use_egg_info=True)]
-        checkLists(l, ['coconuts-aster'])
-
-        l = [dist.name for dist in provides_distribution('banana', '!=0.4',
-                                                         use_egg_info=True)]
-        checkLists(l, [])
-
-    def test_obsoletes(self):
-        # Test looking for distributions based on what they obsolete
-        checkLists = lambda x, y: self.assertListEqual(sorted(x), sorted(y))
-
-        l = [dist.name for dist in obsoletes_distribution('truffles', '1.0')]
-        checkLists(l, [])
-
-        l = [dist.name for dist in obsoletes_distribution('truffles', '1.0',
-                                                          use_egg_info=True)]
-        checkLists(l, ['cheese', 'bacon'])
-
-        l = [dist.name for dist in obsoletes_distribution('truffles', '0.8')]
-        checkLists(l, ['choxie'])
-
-        l = [dist.name for dist in obsoletes_distribution('truffles', '0.8',
-                                                          use_egg_info=True)]
-        checkLists(l, ['choxie', 'cheese'])
-
-        l = [dist.name for dist in obsoletes_distribution('truffles', '0.9.6')]
-        checkLists(l, ['choxie', 'towel-stuff'])
-
-        l = [dist.name for dist in obsoletes_distribution('truffles', \
-                                                          '0.5.2.3')]
-        checkLists(l, ['choxie', 'towel-stuff'])
-
-        l = [dist.name for dist in obsoletes_distribution('truffles', '0.2')]
-        checkLists(l, ['towel-stuff'])
-
-    def test_yield_distribution(self):
-        # tests the internal function _yield_distributions
-        checkLists = lambda x, y: self.assertListEqual(sorted(x), sorted(y))
-
-        eggs = [('bacon', '0.1'), ('banana', '0.4'), ('strawberry', '0.6'),
-                ('truffles', '5.0'), ('cheese', '2.0.2'),
-                ('coconuts-aster', '10.3'), ('nut', 'funkyversion')]
-        dists = [('choxie', '2.0.0.9'), ('grammar', '1.0a4'),
-                 ('towel-stuff', '0.1'), ('babar', '0.1')]
-
-        checkLists([], _yield_distributions(False, False))
-
-        found = [(dist.name, dist.metadata['Version'])
-                for dist in _yield_distributions(False, True)
-                if dist.path.startswith(self.fake_dists_path)]
-        checkLists(eggs, found)
-
-        found = [(dist.name, dist.metadata['Version'])
-                for dist in _yield_distributions(True, False)
-                if dist.path.startswith(self.fake_dists_path)]
-        checkLists(dists, found)
-
-        found = [(dist.name, dist.metadata['Version'])
-                for dist in _yield_distributions(True, True)
-                if dist.path.startswith(self.fake_dists_path)]
-        checkLists(dists + eggs, found)
-
-
-def test_suite():
-    suite = unittest.TestSuite()
-    load = unittest.defaultTestLoader.loadTestsFromTestCase
-    suite.addTest(load(TestPkgUtilData))
-    suite.addTest(load(TestPkgUtilDistribution))
-    suite.addTest(load(TestPkgUtilPEP302))
-    suite.addTest(load(TestPkgUtilPEP376))
-    return suite
-
-
-def test_main():
-    run_unittest(test_suite())
-
-
-if __name__ == "__main__":
-    test_main()
diff --git a/test_distutils2.py b/test_distutils2.py
deleted file mode 100644
--- a/test_distutils2.py
+++ /dev/null
@@ -1,5 +0,0 @@
-import sys
-from distutils2.tests.__main__ import test_main
-
-if __name__ == '__main__':
-    test_main()

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Tue Sep 13 16:34:41 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 13 Sep 2011 16:34:41 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_The_value_is_the_dotted_mod?=
 =?utf8?q?ule_name_to_the_command_class=2E?=
Message-ID: <E1R3U4P-0002vv-0V@dinsdale.python.org>

http://hg.python.org/cpython/rev/688c53548f6b
changeset:   72375:688c53548f6b
user:        Jeremy Kloth <jeremy.kloth at gmail.com>
date:        Tue Sep 13 08:26:25 2011 -0600
summary:
  The value is the dotted module name to the command class.

files:
  Lib/packaging/command/__init__.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/packaging/command/__init__.py b/Lib/packaging/command/__init__.py
--- a/Lib/packaging/command/__init__.py
+++ b/Lib/packaging/command/__init__.py
@@ -33,7 +33,7 @@
 
 # XXX this is crappy
 if os.name == 'nt':
-    _COMMANDS['bdist_msi'] = 'packaging.command.bdist_msi'
+    _COMMANDS['bdist_msi'] = 'packaging.command.bdist_msi.bdist_msi'
 
 # XXX use OrderedDict to preserve the grouping (build-related, install-related,
 # distribution-related)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep 13 23:20:55 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Tue, 13 Sep 2011 23:20:55 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Use_xattr_functions_from_sy?=
 =?utf8?q?s/xattr=2Eh_instead_of_attr/xattr=2Eh_=28closes_=2312720=29?=
Message-ID: <E1R3aPX-0001P1-3t@dinsdale.python.org>

http://hg.python.org/cpython/rev/33f7044b5682
changeset:   72376:33f7044b5682
user:        Benjamin Peterson <benjamin at python.org>
date:        Tue Sep 13 17:20:47 2011 -0400
summary:
  Use xattr functions from sys/xattr.h instead of attr/xattr.h (closes #12720)

sys/xattr.h is glibc while attr/xattr.h is a separate library.

files:
  Modules/posixmodule.c |  12 ++++++------
  configure             |   4 ++--
  configure.in          |   4 ++--
  pyconfig.h.in         |   8 ++++----
  4 files changed, 14 insertions(+), 14 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -107,8 +107,8 @@
 #include <sched.h>
 #endif
 
-#ifdef HAVE_ATTR_XATTR_H
-#include <attr/xattr.h>
+#ifdef HAVE_SYS_XATTR_H
+#include <sys/xattr.h>
 #endif
 
 #if defined(__FreeBSD__) || defined(__DragonFly__) || defined(__APPLE__)
@@ -10032,7 +10032,7 @@
 }
 #endif
 
-#ifdef HAVE_ATTR_XATTR_H
+#ifdef HAVE_SYS_XATTR_H
 
 static int
 try_getxattr(const char *path, const char *name,
@@ -10408,7 +10408,7 @@
     return listxattr_common((const char *)(Py_uintptr_t)fd, wrap_flistxattr);
 }
 
-#endif /* HAVE_ATTR_XATTR_H */
+#endif /* HAVE_SYS_XATTR_H */
 
 static PyMethodDef posix_methods[] = {
     {"access",          posix_access, METH_VARARGS, posix_access__doc__},
@@ -10861,7 +10861,7 @@
 #ifdef HAVE_MKFIFOAT
     {"mkfifoat",        posix_mkfifoat, METH_VARARGS, posix_mkfifoat__doc__},
 #endif
-#ifdef HAVE_ATTR_XATTR_H
+#ifdef HAVE_SYS_XATTR_H
     {"setxattr", posix_setxattr, METH_VARARGS, posix_setxattr__doc__},
     {"lsetxattr", posix_lsetxattr, METH_VARARGS, posix_lsetxattr__doc__},
     {"fsetxattr", posix_fsetxattr, METH_VARARGS, posix_fsetxattr__doc__},
@@ -11336,7 +11336,7 @@
 #endif
 #endif
 
-#ifdef HAVE_ATTR_XATTR_H
+#ifdef HAVE_SYS_XATTR_H
     if (ins(d, "XATTR_CREATE", (long)XATTR_CREATE)) return -1;
     if (ins(d, "XATTR_REPLACE", (long)XATTR_REPLACE)) return -1;
     if (ins(d, "XATTR_SIZE_MAX", (long)XATTR_SIZE_MAX)) return -1;
diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -6090,12 +6090,12 @@
 
 fi
 
-for ac_header in asm/types.h attr/xattr.h conio.h curses.h direct.h dlfcn.h errno.h \
+for ac_header in asm/types.h conio.h curses.h direct.h dlfcn.h errno.h \
 fcntl.h grp.h \
 ieeefp.h io.h langinfo.h libintl.h ncurses.h poll.h process.h pthread.h \
 sched.h shadow.h signal.h stdint.h stropts.h termios.h \
 unistd.h utime.h \
-sys/audioio.h sys/bsdtty.h sys/epoll.h sys/event.h sys/file.h sys/loadavg.h \
+sys/audioio.h sys/xattr.h sys/bsdtty.h sys/epoll.h sys/event.h sys/file.h sys/loadavg.h \
 sys/lock.h sys/mkdev.h sys/modem.h \
 sys/param.h sys/poll.h sys/select.h sys/sendfile.h sys/socket.h sys/statvfs.h \
 sys/stat.h sys/termio.h sys/time.h \
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -1299,12 +1299,12 @@
 
 # checks for header files
 AC_HEADER_STDC
-AC_CHECK_HEADERS(asm/types.h attr/xattr.h conio.h curses.h direct.h dlfcn.h errno.h \
+AC_CHECK_HEADERS(asm/types.h conio.h curses.h direct.h dlfcn.h errno.h \
 fcntl.h grp.h \
 ieeefp.h io.h langinfo.h libintl.h ncurses.h poll.h process.h pthread.h \
 sched.h shadow.h signal.h stdint.h stropts.h termios.h \
 unistd.h utime.h \
-sys/audioio.h sys/bsdtty.h sys/epoll.h sys/event.h sys/file.h sys/loadavg.h \
+sys/audioio.h sys/xattr.h sys/bsdtty.h sys/epoll.h sys/event.h sys/file.h sys/loadavg.h \
 sys/lock.h sys/mkdev.h sys/modem.h \
 sys/param.h sys/poll.h sys/select.h sys/sendfile.h sys/socket.h sys/statvfs.h \
 sys/stat.h sys/termio.h sys/time.h \
diff --git a/pyconfig.h.in b/pyconfig.h.in
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -64,9 +64,6 @@
 /* Define if GCC supports __attribute__((format(PyArg_ParseTuple, 2, 3))) */
 #undef HAVE_ATTRIBUTE_FORMAT_PARSETUPLE
 
-/* Define to 1 if you have the <attr/xattr.h> header file. */
-#undef HAVE_ATTR_XATTR_H
-
 /* Define to 1 if you have the `bind_textdomain_codeset' function. */
 #undef HAVE_BIND_TEXTDOMAIN_CODESET
 
@@ -948,6 +945,9 @@
 /* Define to 1 if you have the <sys/wait.h> header file. */
 #undef HAVE_SYS_WAIT_H
 
+/* Define to 1 if you have the <sys/xattr.h> header file. */
+#undef HAVE_SYS_XATTR_H
+
 /* Define to 1 if you have the `tcgetpgrp' function. */
 #undef HAVE_TCGETPGRP
 
@@ -1312,7 +1312,7 @@
    this defined. */
 #undef _POSIX_1_SOURCE
 
-/* Define to activate features from IEEE Stds 1003.1-2001 */
+/* Define to activate features from IEEE Stds 1003.1-2008 */
 #undef _POSIX_C_SOURCE
 
 /* Define to 1 if you need to in order for `stat' and other things to work. */

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Wed Sep 14 05:22:23 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Wed, 14 Sep 2011 05:22:23 +0200
Subject: [Python-checkins] Daily reference leaks (33f7044b5682): sum=0
Message-ID: <E1R3g3L-0004Wd-PR@ap.vmr.nerim.net>

results for 33f7044b5682 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogRyCqAQ', '-x']

From python-checkins at python.org  Wed Sep 14 16:17:34 2011
From: python-checkins at python.org (stefan.krah)
Date: Wed, 14 Sep 2011 16:17:34 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzExMTQ5?=
 =?utf8?q?=3A_recent_versions_of_clang_require_the_-fwrapv_flag=2E?=
Message-ID: <E1R3qHO-0006f5-Dv@dinsdale.python.org>

http://hg.python.org/cpython/rev/0f1e8c246a7b
changeset:   72377:0f1e8c246a7b
branch:      3.2
parent:      72373:bc5b96c92770
user:        Stefan Krah <skrah at bytereef.org>
date:        Wed Sep 14 15:14:08 2011 +0200
summary:
  Issue #11149: recent versions of clang require the -fwrapv flag.

files:
  configure    |  6 ++++++
  configure.in |  6 ++++++
  2 files changed, 12 insertions(+), 0 deletions(-)


diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -5504,6 +5504,12 @@
         if "$CC" -v --help 2>/dev/null |grep -- -fwrapv > /dev/null; then
            WRAP="-fwrapv"
         fi
+
+        # Clang also needs -fwrapv
+        if test "$CC" = "clang" ; then
+            WRAP="-fwrapv"
+        fi
+
 	case $ac_cv_prog_cc_g in
 	yes)
 	    if test "$Py_DEBUG" = 'true' ; then
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -926,6 +926,12 @@
         if "$CC" -v --help 2>/dev/null |grep -- -fwrapv > /dev/null; then
            WRAP="-fwrapv"
         fi
+
+        # Clang also needs -fwrapv
+        if test "$CC" = "clang" ; then
+            WRAP="-fwrapv"
+        fi
+
 	case $ac_cv_prog_cc_g in
 	yes)
 	    if test "$Py_DEBUG" = 'true' ; then

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 14 16:17:35 2011
From: python-checkins at python.org (stefan.krah)
Date: Wed, 14 Sep 2011 16:17:35 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_fix_for_issue_=2311149=2E?=
Message-ID: <E1R3qHP-0006fz-Av@dinsdale.python.org>

http://hg.python.org/cpython/rev/637c67b34a1a
changeset:   72378:637c67b34a1a
parent:      72376:33f7044b5682
parent:      72377:0f1e8c246a7b
user:        Stefan Krah <skrah at bytereef.org>
date:        Wed Sep 14 15:17:12 2011 +0200
summary:
  Merge fix for issue #11149.

files:
  configure    |  6 ++++++
  configure.in |  6 ++++++
  2 files changed, 12 insertions(+), 0 deletions(-)


diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -5450,6 +5450,12 @@
         if "$CC" -v --help 2>/dev/null |grep -- -fwrapv > /dev/null; then
            WRAP="-fwrapv"
         fi
+
+        # Clang also needs -fwrapv
+        if test "$CC" = "clang" ; then
+            WRAP="-fwrapv"
+        fi
+
 	case $ac_cv_prog_cc_g in
 	yes)
 	    if test "$Py_DEBUG" = 'true' ; then
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -905,6 +905,12 @@
         if "$CC" -v --help 2>/dev/null |grep -- -fwrapv > /dev/null; then
            WRAP="-fwrapv"
         fi
+
+        # Clang also needs -fwrapv
+        if test "$CC" = "clang" ; then
+            WRAP="-fwrapv"
+        fi
+
 	case $ac_cv_prog_cc_g in
 	yes)
 	    if test "$Py_DEBUG" = 'true' ; then

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 14 16:17:36 2011
From: python-checkins at python.org (stefan.krah)
Date: Wed, 14 Sep 2011 16:17:36 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Backport_fix_fo?=
	=?utf8?q?r_issue_=2311149=2E?=
Message-ID: <E1R3qHQ-0006g5-D5@dinsdale.python.org>

http://hg.python.org/cpython/rev/feed6d2097b1
changeset:   72379:feed6d2097b1
branch:      2.7
parent:      72372:e8d8eb9e05fd
user:        Stefan Krah <skrah at bytereef.org>
date:        Wed Sep 14 15:19:42 2011 +0200
summary:
  Backport fix for issue #11149.

files:
  configure    |  6 ++++++
  configure.in |  6 ++++++
  2 files changed, 12 insertions(+), 0 deletions(-)


diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -5411,6 +5411,12 @@
         if "$CC" -v --help 2>/dev/null |grep -- -fwrapv > /dev/null; then
            WRAP="-fwrapv"
         fi
+
+        # Clang also needs -fwrapv
+        if test "$CC" = "clang" ; then
+            WRAP="-fwrapv"
+        fi
+
 	case $ac_cv_prog_cc_g in
 	yes)
 	    if test "$Py_DEBUG" = 'true' ; then
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -932,6 +932,12 @@
         if "$CC" -v --help 2>/dev/null |grep -- -fwrapv > /dev/null; then
            WRAP="-fwrapv"
         fi
+
+        # Clang also needs -fwrapv
+        if test "$CC" = "clang" ; then
+            WRAP="-fwrapv"
+        fi
+
 	case $ac_cv_prog_cc_g in
 	yes)
 	    if test "$Py_DEBUG" = 'true' ; then

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 14 17:46:21 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Wed, 14 Sep 2011 17:46:21 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_only_compile_xattrs_on_glib?=
	=?utf8?q?c_=28closes_=2312720=29?=
Message-ID: <E1R3rfJ-0001Bq-RN@dinsdale.python.org>

http://hg.python.org/cpython/rev/f325439d7f84
changeset:   72380:f325439d7f84
parent:      72376:33f7044b5682
user:        Benjamin Peterson <benjamin at python.org>
date:        Wed Sep 14 11:45:52 2011 -0400
summary:
  only compile xattrs on glibc (closes #12720)

files:
  Modules/posixmodule.c |  14 +++++++++-----
  1 files changed, 9 insertions(+), 5 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -107,7 +107,11 @@
 #include <sched.h>
 #endif
 
-#ifdef HAVE_SYS_XATTR_H
+#if defined(HAVE_SYS_XATTR_H) && defined(__GLIBC__)
+#define USE_XATTRS
+#endif
+
+#ifdef USE_XATTRS
 #include <sys/xattr.h>
 #endif
 
@@ -10032,7 +10036,7 @@
 }
 #endif
 
-#ifdef HAVE_SYS_XATTR_H
+#ifdef USE_XATTRS
 
 static int
 try_getxattr(const char *path, const char *name,
@@ -10408,7 +10412,7 @@
     return listxattr_common((const char *)(Py_uintptr_t)fd, wrap_flistxattr);
 }
 
-#endif /* HAVE_SYS_XATTR_H */
+#endif /* USE_XATTRS */
 
 static PyMethodDef posix_methods[] = {
     {"access",          posix_access, METH_VARARGS, posix_access__doc__},
@@ -10861,7 +10865,7 @@
 #ifdef HAVE_MKFIFOAT
     {"mkfifoat",        posix_mkfifoat, METH_VARARGS, posix_mkfifoat__doc__},
 #endif
-#ifdef HAVE_SYS_XATTR_H
+#ifdef USE_XATTRS
     {"setxattr", posix_setxattr, METH_VARARGS, posix_setxattr__doc__},
     {"lsetxattr", posix_lsetxattr, METH_VARARGS, posix_lsetxattr__doc__},
     {"fsetxattr", posix_fsetxattr, METH_VARARGS, posix_fsetxattr__doc__},
@@ -11336,7 +11340,7 @@
 #endif
 #endif
 
-#ifdef HAVE_SYS_XATTR_H
+#ifdef USE_XATTRS
     if (ins(d, "XATTR_CREATE", (long)XATTR_CREATE)) return -1;
     if (ins(d, "XATTR_REPLACE", (long)XATTR_REPLACE)) return -1;
     if (ins(d, "XATTR_SIZE_MAX", (long)XATTR_SIZE_MAX)) return -1;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 14 17:46:22 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Wed, 14 Sep 2011 17:46:22 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_merge_heads?=
Message-ID: <E1R3rfK-0001Bv-Jl@dinsdale.python.org>

http://hg.python.org/cpython/rev/63bf3bae20ef
changeset:   72381:63bf3bae20ef
parent:      72380:f325439d7f84
parent:      72378:637c67b34a1a
user:        Benjamin Peterson <benjamin at python.org>
date:        Wed Sep 14 11:46:17 2011 -0400
summary:
  merge heads

files:
  configure    |  6 ++++++
  configure.in |  6 ++++++
  2 files changed, 12 insertions(+), 0 deletions(-)


diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -5450,6 +5450,12 @@
         if "$CC" -v --help 2>/dev/null |grep -- -fwrapv > /dev/null; then
            WRAP="-fwrapv"
         fi
+
+        # Clang also needs -fwrapv
+        if test "$CC" = "clang" ; then
+            WRAP="-fwrapv"
+        fi
+
 	case $ac_cv_prog_cc_g in
 	yes)
 	    if test "$Py_DEBUG" = 'true' ; then
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -905,6 +905,12 @@
         if "$CC" -v --help 2>/dev/null |grep -- -fwrapv > /dev/null; then
            WRAP="-fwrapv"
         fi
+
+        # Clang also needs -fwrapv
+        if test "$CC" = "clang" ; then
+            WRAP="-fwrapv"
+        fi
+
 	case $ac_cv_prog_cc_g in
 	yes)
 	    if test "$Py_DEBUG" = 'true' ; then

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 14 23:57:30 2011
From: python-checkins at python.org (ned.deily)
Date: Wed, 14 Sep 2011 23:57:30 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzk4NzE6?=
 =?utf8?q?_Prevent_IDLE_3_crash_when_given_byte_stings?=
Message-ID: <E1R3xSU-0004Hx-2z@dinsdale.python.org>

http://hg.python.org/cpython/rev/e74860883a9c
changeset:   72382:e74860883a9c
branch:      3.2
parent:      72377:0f1e8c246a7b
user:        Ned Deily <nad at acm.org>
date:        Wed Sep 14 14:49:14 2011 -0700
summary:
  Issue #9871: Prevent IDLE 3 crash when given byte stings
with invalid hex escape sequences, like b'\x0'.
(Original patch by Claudiu Popa.)

files:
  Lib/idlelib/PyShell.py       |  6 +++---
  Lib/idlelib/ScriptBinding.py |  8 ++++----
  Misc/ACKS                    |  1 +
  Misc/NEWS                    |  4 ++++
  4 files changed, 12 insertions(+), 7 deletions(-)


diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py
--- a/Lib/idlelib/PyShell.py
+++ b/Lib/idlelib/PyShell.py
@@ -643,9 +643,9 @@
         text = tkconsole.text
         text.tag_remove("ERROR", "1.0", "end")
         type, value, tb = sys.exc_info()
-        msg = value.msg or "<no detail available>"
-        lineno = value.lineno or 1
-        offset = value.offset or 0
+        msg = getattr(value, 'msg', '') or value or "<no detail available>"
+        lineno = getattr(value, 'lineno', '') or 1
+        offset = getattr(value, 'offset', '') or 0
         if offset == 0:
             lineno += 1 #mark end of offending line
         if lineno == 1:
diff --git a/Lib/idlelib/ScriptBinding.py b/Lib/idlelib/ScriptBinding.py
--- a/Lib/idlelib/ScriptBinding.py
+++ b/Lib/idlelib/ScriptBinding.py
@@ -101,10 +101,10 @@
         try:
             # If successful, return the compiled code
             return compile(source, filename, "exec")
-        except (SyntaxError, OverflowError) as value:
-            msg = value.msg or "<no detail available>"
-            lineno = value.lineno or 1
-            offset = value.offset or 0
+        except (SyntaxError, OverflowError, ValueError) as value:
+            msg = getattr(value, 'msg', '') or value or "<no detail available>"
+            lineno = getattr(value, 'lineno', '') or 1
+            offset = getattr(value, 'offset', '') or 0
             if offset == 0:
                 lineno += 1  #mark end of offending line
             pos = "0.0 + %d lines + %d chars" % (lineno-1, offset-1)
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -705,6 +705,7 @@
 Guilherme Polo
 Michael Pomraning
 Iustin Pop
+Claudiu Popa
 John Popplewell
 Amrit Prem
 Paul Prescod
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -25,6 +25,10 @@
 Library
 -------
 
+- Issue #9871: Prevent IDLE 3 crash when given byte stings
+  with invalid hex escape sequences, like b'\x0'.
+  (Original patch by Claudiu Popa.)
+
 - Issue #8933: distutils' PKG-INFO files will now correctly report
   Metadata-Version: 1.1 instead of 1.0 if a Classifier or Download-URL field is
   present.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 14 23:57:31 2011
From: python-checkins at python.org (ned.deily)
Date: Wed, 14 Sep 2011 23:57:31 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=239871=3A_Prevent_IDLE_3_crash_when_given_byte_stings?=
Message-ID: <E1R3xSV-0004I3-CN@dinsdale.python.org>

http://hg.python.org/cpython/rev/fe6c7771c25c
changeset:   72383:fe6c7771c25c
parent:      72381:63bf3bae20ef
parent:      72382:e74860883a9c
user:        Ned Deily <nad at acm.org>
date:        Wed Sep 14 14:56:32 2011 -0700
summary:
  Issue #9871: Prevent IDLE 3 crash when given byte stings
with invalid hex escape sequences, like b'\x0'.
(Original patch by Claudiu Popa.)

files:
  Lib/idlelib/PyShell.py       |  6 +++---
  Lib/idlelib/ScriptBinding.py |  8 ++++----
  Misc/ACKS                    |  1 +
  Misc/NEWS                    |  4 ++++
  4 files changed, 12 insertions(+), 7 deletions(-)


diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py
--- a/Lib/idlelib/PyShell.py
+++ b/Lib/idlelib/PyShell.py
@@ -643,9 +643,9 @@
         text = tkconsole.text
         text.tag_remove("ERROR", "1.0", "end")
         type, value, tb = sys.exc_info()
-        msg = value.msg or "<no detail available>"
-        lineno = value.lineno or 1
-        offset = value.offset or 0
+        msg = getattr(value, 'msg', '') or value or "<no detail available>"
+        lineno = getattr(value, 'lineno', '') or 1
+        offset = getattr(value, 'offset', '') or 0
         if offset == 0:
             lineno += 1 #mark end of offending line
         if lineno == 1:
diff --git a/Lib/idlelib/ScriptBinding.py b/Lib/idlelib/ScriptBinding.py
--- a/Lib/idlelib/ScriptBinding.py
+++ b/Lib/idlelib/ScriptBinding.py
@@ -101,10 +101,10 @@
         try:
             # If successful, return the compiled code
             return compile(source, filename, "exec")
-        except (SyntaxError, OverflowError) as value:
-            msg = value.msg or "<no detail available>"
-            lineno = value.lineno or 1
-            offset = value.offset or 0
+        except (SyntaxError, OverflowError, ValueError) as value:
+            msg = getattr(value, 'msg', '') or value or "<no detail available>"
+            lineno = getattr(value, 'lineno', '') or 1
+            offset = getattr(value, 'offset', '') or 0
             if offset == 0:
                 lineno += 1  #mark end of offending line
             pos = "0.0 + %d lines + %d chars" % (lineno-1, offset-1)
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -753,6 +753,7 @@
 Guilherme Polo
 Michael Pomraning
 Iustin Pop
+Claudiu Popa
 John Popplewell
 Amrit Prem
 Paul Prescod
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -274,6 +274,10 @@
 Library
 -------
 
+- Issue #9871: Prevent IDLE 3 crash when given byte stings
+  with invalid hex escape sequences, like b'\x0'.
+  (Original patch by Claudiu Popa.)
+
 - Issue #12306: Expose the runtime version of the zlib C library as a constant,
   ZLIB_RUNTIME_VERSION, in the zlib module. Patch by Torsten Landschoff.
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Thu Sep 15 05:18:27 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Thu, 15 Sep 2011 05:18:27 +0200
Subject: [Python-checkins] Daily reference leaks (fe6c7771c25c): sum=0
Message-ID: <E1R42T5-0004Ds-2a@ap.vmr.nerim.net>

results for fe6c7771c25c on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogW2yTvq', '-x']

From python-checkins at python.org  Thu Sep 15 17:29:46 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Thu, 15 Sep 2011 17:29:46 +0200
Subject: [Python-checkins] =?utf8?q?peps=3A_Add_porting_guidelines=2E?=
Message-ID: <E1R4Dso-00050E-QH@dinsdale.python.org>

http://hg.python.org/peps/rev/61680adf0f5f
changeset:   3943:61680adf0f5f
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Thu Sep 15 17:29:41 2011 +0200
summary:
  Add porting guidelines.

files:
  pep-0393.txt |  58 ++++++++++++++++++++++++++++++++++++++++
  1 files changed, 58 insertions(+), 0 deletions(-)


diff --git a/pep-0393.txt b/pep-0393.txt
--- a/pep-0393.txt
+++ b/pep-0393.txt
@@ -272,6 +272,64 @@
 slowdowns of 1% to 30%; for specific benchmarks, speedups may
 happen as may happen significantly larger slowdowns.
 
+Porting Guidelines
+==================
+
+Only a small fraction of C code is affected by this PEP, namely code
+that needs to look "inside" unicode strings.  That code doesn't
+necessarily need to be ported to this API, as the existing API will
+continue to work correctly. In particular, modules that need to
+support both Python 2 and Python 3 might get too complicated when
+simultaneously supporting this new API and the old Unicode API.
+
+In order to port modules to the new API, try to eliminate
+the use of these API elements:
+
+- the Py_UNICODE type,
+- PyUnicode_AS_UNICODE and PyUnicode_AsUnicode,
+- PyUnicode_GET_LENGTH and PyUnicode_GetSize, and
+- PyUnicode_FromUnicode.
+
+When iterating over an existing string, or looking at specific
+characters, use indexing operations rather than pointer arithmetic;
+indexing works well for PyUnicode_READ(_CHAR) and PyUnicode_WRITE. Use
+void* as the buffer type for characters to let the compiler detect
+invalid dereferencing operations. If you do want to use pointer
+arithmentics (e.g. when converting existing code), use (unsigned)
+char* as the buffer type, and keep the element size (1, 2, or 4) in a
+variable. Notice that (1<<(kind-1)) will produce the element size
+given a buffer kind.
+
+When creating new strings, it was common in Python to start of with a
+heuristical buffer size, and then grow or shrink if the heuristics
+failed. With this PEP, this is now less practical, as you need not
+only a heuristics for the length of the string, but also for the
+maximum character.
+
+In order to avoid heuristics, you need to make two passes over the
+input: once to determine the output length, and the maximum character;
+then allocate the target string with PyUnicode_New and iterate over
+the input a second time to produce the final output. While this may
+sound expensive, it could actually be cheaper than having to copy the
+result again as in the following approach.
+
+If you take the heuristical route, avoid allocating a string meant to
+be resized, as resizing strings won't work for their canonical
+representation.  Instead, allocate a separate buffer to collect the
+characters, and then construct a unicode object from that using
+PyUnicode_FromKindAndData. One option is to use Py_UCS4 as the buffer
+element, assuming for the worst case in character ordinals. This will
+allow for pointer arithmetics, but may require a lot of memory.
+Alternatively, start with a 1-byte buffer, and increase the element
+size as you encounter larger characters. In any case,
+PyUnicode_FromKindAndData will scan over the buffer to verify the
+maximum character.
+
+For common tasks, direct access to the string representation may not
+be necessary: PyUnicode_Find, PyUnicode_FindChar, PyUnicode_Ord, and
+PyUnicode_CopyCharacters help in analyzing and creating string
+objects, operating on indices instead of data pointers.
+
 Copyright
 =========
 

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Thu Sep 15 18:19:41 2011
From: python-checkins at python.org (eric.araujo)
Date: Thu, 15 Sep 2011 18:19:41 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_packaging=2Edatabase=2E?=
 =?utf8?q?Distribution=2Elist=5Fdistinfo=5Ffiles_=28=2312785=29=2E?=
Message-ID: <E1R4Ef7-0002IS-DJ@dinsdale.python.org>

http://hg.python.org/cpython/rev/c6d52971dd2a
changeset:   72384:c6d52971dd2a
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 15 18:18:51 2011 +0200
summary:
  Fix packaging.database.Distribution.list_distinfo_files (#12785).

This method was supposed to return only the file under the dist-info
directory, but it actually returned all installed files.

The tests didn?t catch this because they were flawed; I updated them.
Thanks to Nadeem Vawda and Jeremy Kloth for testing.

As a bonus, the removal of os.path.relpath use should also fix the
Windows buildbots.

files:
  Lib/packaging/database.py            |   4 +-
  Lib/packaging/tests/test_database.py |  46 ++++++++-------
  2 files changed, 27 insertions(+), 23 deletions(-)


diff --git a/Lib/packaging/database.py b/Lib/packaging/database.py
--- a/Lib/packaging/database.py
+++ b/Lib/packaging/database.py
@@ -263,7 +263,9 @@
         :returns: iterator of paths
         """
         for path, checksum, size in self._get_records(local):
-            yield path
+            # XXX add separator or use real relpath algo
+            if path.startswith(self.path):
+                yield path
 
     def __eq__(self, other):
         return isinstance(other, Distribution) and self.path == other.path
diff --git a/Lib/packaging/tests/test_database.py b/Lib/packaging/tests/test_database.py
--- a/Lib/packaging/tests/test_database.py
+++ b/Lib/packaging/tests/test_database.py
@@ -4,7 +4,6 @@
 import sys
 import shutil
 import tempfile
-from os.path import relpath  # separate import for backport concerns
 from hashlib import md5
 from textwrap import dedent
 
@@ -32,11 +31,11 @@
     return checksum.hexdigest()
 
 
-def record_pieces(file):
-    path = relpath(file, sys.prefix)
-    digest = get_hexdigest(file)
-    size = os.path.getsize(file)
-    return [path, digest, size]
+def record_pieces(path):
+    path = os.path.join(*path)
+    digest = get_hexdigest(path)
+    size = os.path.getsize(path)
+    return path, digest, size
 
 
 class FakeDistsMixin:
@@ -141,12 +140,10 @@
 
                 for path, dirs, files in os.walk(dist_location):
                     for f in files:
-                        record_writer.writerow(record_pieces(
-                                               os.path.join(path, f)))
+                        record_writer.writerow(record_pieces((path, f)))
                 for file in ('INSTALLER', 'METADATA', 'REQUESTED'):
-                    record_writer.writerow(record_pieces(
-                                           os.path.join(distinfo_dir, file)))
-                record_writer.writerow([relpath(record_file, sys.prefix)])
+                    record_writer.writerow(record_pieces((distinfo_dir, file)))
+                record_writer.writerow([record_file])
 
             with open(record_file) as file:
                 record_reader = csv.reader(file, lineterminator='\n')
@@ -171,15 +168,17 @@
                                     distinfo_name + '.dist-info')
         true_path = [self.fake_dists_path, distinfo_name,
                      'grammar', 'utils.py']
-        true_path = relpath(os.path.join(*true_path), sys.prefix)
+        true_path = os.path.join(*true_path)
         false_path = [self.fake_dists_path, 'towel_stuff-0.1', 'towel_stuff',
                       '__init__.py']
-        false_path = relpath(os.path.join(*false_path), sys.prefix)
+        false_path = os.path.join(*false_path)
 
         # Test if the distribution uses the file in question
         dist = Distribution(distinfo_dir)
-        self.assertTrue(dist.uses(true_path))
-        self.assertFalse(dist.uses(false_path))
+        self.assertTrue(dist.uses(true_path), 'dist %r is supposed to use %r' %
+                        (dist, true_path))
+        self.assertFalse(dist.uses(false_path), 'dist %r is not supposed to '
+                         'use %r' % (dist, true_path))
 
     def test_get_distinfo_file(self):
         # Test the retrieval of dist-info file objects.
@@ -215,20 +214,23 @@
                           'MAGICFILE')
 
     def test_list_distinfo_files(self):
-        # Test for the iteration of RECORD path entries.
         distinfo_name = 'towel_stuff-0.1'
         distinfo_dir = os.path.join(self.fake_dists_path,
                                     distinfo_name + '.dist-info')
         dist = Distribution(distinfo_dir)
         # Test for the iteration of the raw path
-        distinfo_record_paths = self.records[distinfo_dir].keys()
+        distinfo_files = [os.path.join(distinfo_dir, filename) for filename in
+                          os.listdir(distinfo_dir)]
         found = dist.list_distinfo_files()
-        self.assertEqual(sorted(found), sorted(distinfo_record_paths))
+        self.assertEqual(sorted(found), sorted(distinfo_files))
         # Test for the iteration of local absolute paths
-        distinfo_record_paths = [os.path.join(sys.prefix, path)
-            for path in self.records[distinfo_dir]]
-        found = dist.list_distinfo_files(local=True)
-        self.assertEqual(sorted(found), sorted(distinfo_record_paths))
+        distinfo_files = [os.path.join(sys.prefix, distinfo_dir, path) for
+                          path in distinfo_files]
+        found = sorted(dist.list_distinfo_files(local=True))
+        if os.sep != '/':
+            self.assertNotIn('/', found[0])
+            self.assertIn(os.sep, found[0])
+        self.assertEqual(found, sorted(distinfo_files))
 
     def test_get_resources_path(self):
         distinfo_name = 'babar-0.1'

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 15 19:47:06 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 15 Sep 2011 19:47:06 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_the_import_?=
 =?utf8?q?machinery_if_there_is_an_error_on_sys=2Epath_or_sys=2Emeta=5Fpat?=
 =?utf8?q?h?=
Message-ID: <E1R4G1i-0002Nm-Jh@dinsdale.python.org>

http://hg.python.org/cpython/rev/4247f5e221c6
changeset:   72385:4247f5e221c6
branch:      3.2
parent:      72382:e74860883a9c
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 15 19:28:05 2011 +0200
summary:
  Fix the import machinery if there is an error on sys.path or sys.meta_path

find_module() now raises a RuntimeError, instead of ImportError, on an error on
sys.path or sys.meta_path because load_package() and import_submodule() returns
None and clear the exception if a ImportError occurred.

files:
  Python/import.c |  8 ++++----
  1 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -1591,7 +1591,7 @@
 
         meta_path = PySys_GetObject("meta_path");
         if (meta_path == NULL || !PyList_Check(meta_path)) {
-            PyErr_SetString(PyExc_ImportError,
+            PyErr_SetString(PyExc_RuntimeError,
                             "sys.meta_path must be a list of "
                             "import hooks");
             return NULL;
@@ -1641,14 +1641,14 @@
     }
 
     if (path == NULL || !PyList_Check(path)) {
-        PyErr_SetString(PyExc_ImportError,
+        PyErr_SetString(PyExc_RuntimeError,
                         "sys.path must be a list of directory names");
         return NULL;
     }
 
     path_hooks = PySys_GetObject("path_hooks");
     if (path_hooks == NULL || !PyList_Check(path_hooks)) {
-        PyErr_SetString(PyExc_ImportError,
+        PyErr_SetString(PyExc_RuntimeError,
                         "sys.path_hooks must be a list of "
                         "import hooks");
         return NULL;
@@ -1656,7 +1656,7 @@
     path_importer_cache = PySys_GetObject("path_importer_cache");
     if (path_importer_cache == NULL ||
         !PyDict_Check(path_importer_cache)) {
-        PyErr_SetString(PyExc_ImportError,
+        PyErr_SetString(PyExc_RuntimeError,
                         "sys.path_importer_cache must be a dict");
         return NULL;
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 15 19:47:07 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 15 Sep 2011 19:47:07 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_3=2E2=3A_Fix_the_import_machinery_if_there_is_an_error?=
 =?utf8?q?_on_sys=2Epath_or?=
Message-ID: <E1R4G1j-0002Nw-Bd@dinsdale.python.org>

http://hg.python.org/cpython/rev/3ff9a2d5f5e5
changeset:   72386:3ff9a2d5f5e5
parent:      72384:c6d52971dd2a
parent:      72385:4247f5e221c6
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 15 19:38:54 2011 +0200
summary:
  Merge 3.2: Fix the import machinery if there is an error on sys.path or sys.meta_path

find_module() now raises a RuntimeError, instead of ImportError, on an error on
sys.path or sys.meta_path because load_package() and import_submodule() returns
None and clear the exception if a ImportError occurred.

files:
  Python/import.c |  8 ++++----
  1 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -1992,7 +1992,7 @@
 
         meta_path = PySys_GetObject("meta_path");
         if (meta_path == NULL || !PyList_Check(meta_path)) {
-            PyErr_SetString(PyExc_ImportError,
+            PyErr_SetString(PyExc_RuntimeError,
                             "sys.meta_path must be a list of "
                             "import hooks");
             return NULL;
@@ -2044,14 +2044,14 @@
     }
 
     if (search_path_list == NULL || !PyList_Check(search_path_list)) {
-        PyErr_SetString(PyExc_ImportError,
+        PyErr_SetString(PyExc_RuntimeError,
                         "sys.path must be a list of directory names");
         return NULL;
     }
 
     path_hooks = PySys_GetObject("path_hooks");
     if (path_hooks == NULL || !PyList_Check(path_hooks)) {
-        PyErr_SetString(PyExc_ImportError,
+        PyErr_SetString(PyExc_RuntimeError,
                         "sys.path_hooks must be a list of "
                         "import hooks");
         return NULL;
@@ -2059,7 +2059,7 @@
     path_importer_cache = PySys_GetObject("path_importer_cache");
     if (path_importer_cache == NULL ||
         !PyDict_Check(path_importer_cache)) {
-        PyErr_SetString(PyExc_ImportError,
+        PyErr_SetString(PyExc_RuntimeError,
                         "sys.path_importer_cache must be a dict");
         return NULL;
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 15 19:47:07 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 15 Sep 2011 19:47:07 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_the_import_?=
 =?utf8?q?machinery_if_there_is_an_error_on_sys=2Epath_or_sys=2Emeta=5Fpat?=
 =?utf8?q?h?=
Message-ID: <E1R4G1j-0002O0-Ty@dinsdale.python.org>

http://hg.python.org/cpython/rev/c9bb16b7d148
changeset:   72387:c9bb16b7d148
branch:      2.7
parent:      72379:feed6d2097b1
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 15 19:45:53 2011 +0200
summary:
  Fix the import machinery if there is an error on sys.path or sys.meta_path

find_module() now raises a RuntimeError, instead of ImportError, on an error on
sys.path or sys.meta_path because load_package() and import_submodule() returns
None and clear the exception if a ImportError occurred.

files:
  Python/import.c |  8 ++++----
  1 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -1235,7 +1235,7 @@
 
         meta_path = PySys_GetObject("meta_path");
         if (meta_path == NULL || !PyList_Check(meta_path)) {
-            PyErr_SetString(PyExc_ImportError,
+            PyErr_SetString(PyExc_RuntimeError,
                             "sys.meta_path must be a list of "
                             "import hooks");
             return NULL;
@@ -1304,14 +1304,14 @@
         path = PySys_GetObject("path");
     }
     if (path == NULL || !PyList_Check(path)) {
-        PyErr_SetString(PyExc_ImportError,
+        PyErr_SetString(PyExc_RuntimeError,
                         "sys.path must be a list of directory names");
         return NULL;
     }
 
     path_hooks = PySys_GetObject("path_hooks");
     if (path_hooks == NULL || !PyList_Check(path_hooks)) {
-        PyErr_SetString(PyExc_ImportError,
+        PyErr_SetString(PyExc_RuntimeError,
                         "sys.path_hooks must be a list of "
                         "import hooks");
         return NULL;
@@ -1319,7 +1319,7 @@
     path_importer_cache = PySys_GetObject("path_importer_cache");
     if (path_importer_cache == NULL ||
         !PyDict_Check(path_importer_cache)) {
-        PyErr_SetString(PyExc_ImportError,
+        PyErr_SetString(PyExc_RuntimeError,
                         "sys.path_importer_cache must be a dict");
         return NULL;
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 15 19:52:43 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 15 Sep 2011 19:52:43 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_import=2Ec=3A_remove_now_us?=
	=?utf8?q?eless_arbitrary_limit?=
Message-ID: <E1R4G79-0002s7-Lw@dinsdale.python.org>

http://hg.python.org/cpython/rev/03843a8d3244
changeset:   72388:03843a8d3244
parent:      72386:3ff9a2d5f5e5
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 15 19:50:01 2011 +0200
summary:
  import.c: remove now useless arbitrary limit

files:
  Python/import.c |  6 ------
  1 files changed, 0 insertions(+), 6 deletions(-)


diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -1980,12 +1980,6 @@
     if (p_loader != NULL)
         *p_loader = NULL;
 
-    if (PyUnicode_GET_SIZE(name) > MAXPATHLEN) {
-        PyErr_SetString(PyExc_OverflowError,
-                        "module name is too long");
-        return NULL;
-    }
-
     /* sys.meta_path import hook */
     if (p_loader != NULL) {
         PyObject *meta_path;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 15 22:57:05 2011
From: python-checkins at python.org (stefan.krah)
Date: Thu, 15 Sep 2011 22:57:05 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Use_bitwise_ins?=
 =?utf8?q?tead_of_logical_or_for_flags=2E?=
Message-ID: <E1R4IzZ-0003z5-3E@dinsdale.python.org>

http://hg.python.org/cpython/rev/4b4b61dd24e1
changeset:   72389:4b4b61dd24e1
branch:      2.7
parent:      72387:c9bb16b7d148
user:        Stefan Krah <skrah at bytereef.org>
date:        Thu Sep 15 22:56:00 2011 +0200
summary:
  Use bitwise instead of logical or for flags.

files:
  Modules/_bsddb.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Modules/_bsddb.c b/Modules/_bsddb.c
--- a/Modules/_bsddb.c
+++ b/Modules/_bsddb.c
@@ -8795,7 +8795,7 @@
     {"txn_recover",     (PyCFunction)DBEnv_txn_recover,     METH_NOARGS},
 #if (DBVER < 48)
     {"set_rpc_server",  (PyCFunction)DBEnv_set_rpc_server,
-        METH_VARARGS||METH_KEYWORDS},
+        METH_VARARGS|METH_KEYWORDS},
 #endif
 #if (DBVER >= 43)
     {"set_mp_max_openfd", (PyCFunction)DBEnv_set_mp_max_openfd, METH_VARARGS},

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 15 23:08:14 2011
From: python-checkins at python.org (ned.deily)
Date: Thu, 15 Sep 2011 23:08:14 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyOTM1?=
 =?utf8?q?=3A_Correct_typo_in_findertools=2E?=
Message-ID: <E1R4JAM-0004yZ-KY@dinsdale.python.org>

http://hg.python.org/cpython/rev/b5f4c4085ae6
changeset:   72390:b5f4c4085ae6
branch:      2.7
user:        Ned Deily <nad at acm.org>
date:        Thu Sep 15 14:07:31 2011 -0700
summary:
  Issue #12935: Correct typo in findertools.

files:
  Lib/plat-mac/findertools.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/plat-mac/findertools.py b/Lib/plat-mac/findertools.py
--- a/Lib/plat-mac/findertools.py
+++ b/Lib/plat-mac/findertools.py
@@ -128,7 +128,7 @@
 def comment(object, comment=None):
     """comment: get or set the Finder-comment of the item, displayed in the 'Get Info' window."""
     object = Carbon.File.FSRef(object)
-    object_alias = object.FSNewAliasMonimal()
+    object_alias = object.FSNewAliasMinimal()
     if comment is None:
         return _getcomment(object_alias)
     else:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep 16 00:10:30 2011
From: python-checkins at python.org (ned.deily)
Date: Fri, 16 Sep 2011 00:10:30 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312765=3A_Fix_packa?=
 =?utf8?q?ging=2Etest=2Etest=5Fdatabase_failures_on_OS_X_due?=
Message-ID: <E1R4K8c-0001uN-PP@dinsdale.python.org>

http://hg.python.org/cpython/rev/d3e072083ff3
changeset:   72391:d3e072083ff3
parent:      72388:03843a8d3244
user:        Ned Deily <nad at acm.org>
date:        Thu Sep 15 15:09:23 2011 -0700
summary:
  Issue #12765: Fix packaging.test.test_database failures on OS X due
to unwarranted assumption about absolute paths: on OS X /var is a
symlink to /private/var.  (Also true for /etc and /tmp).

files:
  Lib/packaging/tests/test_database.py |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Lib/packaging/tests/test_database.py b/Lib/packaging/tests/test_database.py
--- a/Lib/packaging/tests/test_database.py
+++ b/Lib/packaging/tests/test_database.py
@@ -50,6 +50,7 @@
         tmpdir = tempfile.mkdtemp()
         self.addCleanup(shutil.rmtree, tmpdir)
         self.fake_dists_path = os.path.join(tmpdir, 'fake_dists')
+        self.fake_dists_path = os.path.realpath(self.fake_dists_path)
         fake_dists_src = os.path.abspath(
             os.path.join(os.path.dirname(__file__), 'fake_dists'))
         shutil.copytree(fake_dists_src, self.fake_dists_path)

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Fri Sep 16 05:18:25 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Fri, 16 Sep 2011 05:18:25 +0200
Subject: [Python-checkins] Daily reference leaks (d3e072083ff3): sum=0
Message-ID: <E1R4Owb-00043S-Rh@ap.vmr.nerim.net>

results for d3e072083ff3 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogdyv4GV', '-x']

From solipsis at pitrou.net  Sat Sep 17 05:20:33 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sat, 17 Sep 2011 05:20:33 +0200
Subject: [Python-checkins] Daily reference leaks (d3e072083ff3): sum=0
Message-ID: <E1R4lSD-00046e-J3@ap.vmr.nerim.net>

results for d3e072083ff3 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogRL66af', '-x']

From python-checkins at python.org  Sat Sep 17 20:39:15 2011
From: python-checkins at python.org (georg.brandl)
Date: Sat, 17 Sep 2011 20:39:15 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogRml4IHR5cG8u?=
Message-ID: <E1R4znH-000836-A1@dinsdale.python.org>

http://hg.python.org/cpython/rev/79ee8447a0da
changeset:   72392:79ee8447a0da
branch:      3.2
parent:      72385:4247f5e221c6
user:        Georg Brandl <georg at python.org>
date:        Sat Sep 17 20:20:04 2011 +0200
summary:
  Fix typo.

files:
  Doc/library/datetime.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst
--- a/Doc/library/datetime.rst
+++ b/Doc/library/datetime.rst
@@ -30,7 +30,7 @@
 :class:`tzinfo` objects capture information about the offset from UTC time, the
 time zone name, and whether Daylight Saving Time is in effect.  Note that only
 one concrete :class:`tzinfo` class, the :class:`timezone` class, is supplied by the
-:mod:`datetime` module.  The :class:`timezone` class can reprsent simple
+:mod:`datetime` module.  The :class:`timezone` class can represent simple
 timezones with fixed offset from UTC such as UTC itself or North American EST and
 EDT timezones.  Supporting timezones at whatever level of detail is
 required is up to the application.  The rules for time adjustment across the

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 17 20:39:16 2011
From: python-checkins at python.org (georg.brandl)
Date: Sat, 17 Sep 2011 20:39:16 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Add_info_from_t?=
 =?utf8?q?he_docstring_for_random=2Egammavariate=28=29_to_the_docs=2E?=
Message-ID: <E1R4znI-00083G-17@dinsdale.python.org>

http://hg.python.org/cpython/rev/e6407b2d2f0c
changeset:   72393:e6407b2d2f0c
branch:      3.2
user:        Georg Brandl <georg at python.org>
date:        Sat Sep 17 20:36:28 2011 +0200
summary:
  Add info from the docstring for random.gammavariate() to the docs.

files:
  Doc/library/random.rst |  7 +++++++
  1 files changed, 7 insertions(+), 0 deletions(-)


diff --git a/Doc/library/random.rst b/Doc/library/random.rst
--- a/Doc/library/random.rst
+++ b/Doc/library/random.rst
@@ -163,6 +163,7 @@
    The end-point value ``b`` may or may not be included in the range
    depending on floating-point rounding in the equation ``a + (b-a) * random()``.
 
+
 .. function:: triangular(low, high, mode)
 
    Return a random floating point number *N* such that ``low <= N <= high`` and
@@ -191,6 +192,12 @@
    Gamma distribution.  (*Not* the gamma function!)  Conditions on the
    parameters are ``alpha > 0`` and ``beta > 0``.
 
+   The probability distribution function is::
+
+                 x ** (alpha - 1) * math.exp(-x / beta)
+       pdf(x) =  --------------------------------------
+                   math.gamma(alpha) * beta ** alpha
+
 
 .. function:: gauss(mu, sigma)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 17 20:39:16 2011
From: python-checkins at python.org (georg.brandl)
Date: Sat, 17 Sep 2011 20:39:16 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_merge_doc_fixes_from_3=2E2?=
Message-ID: <E1R4znI-00083K-Np@dinsdale.python.org>

http://hg.python.org/cpython/rev/23e4b73937f7
changeset:   72394:23e4b73937f7
parent:      72391:d3e072083ff3
parent:      72393:e6407b2d2f0c
user:        Georg Brandl <georg at python.org>
date:        Sat Sep 17 20:40:35 2011 +0200
summary:
  merge doc fixes from 3.2

files:
  Doc/library/datetime.rst |  2 +-
  Doc/library/random.rst   |  7 +++++++
  2 files changed, 8 insertions(+), 1 deletions(-)


diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst
--- a/Doc/library/datetime.rst
+++ b/Doc/library/datetime.rst
@@ -30,7 +30,7 @@
 :class:`tzinfo` objects capture information about the offset from UTC time, the
 time zone name, and whether Daylight Saving Time is in effect.  Note that only
 one concrete :class:`tzinfo` class, the :class:`timezone` class, is supplied by the
-:mod:`datetime` module.  The :class:`timezone` class can reprsent simple
+:mod:`datetime` module.  The :class:`timezone` class can represent simple
 timezones with fixed offset from UTC such as UTC itself or North American EST and
 EDT timezones.  Supporting timezones at whatever level of detail is
 required is up to the application.  The rules for time adjustment across the
diff --git a/Doc/library/random.rst b/Doc/library/random.rst
--- a/Doc/library/random.rst
+++ b/Doc/library/random.rst
@@ -169,6 +169,7 @@
    The end-point value ``b`` may or may not be included in the range
    depending on floating-point rounding in the equation ``a + (b-a) * random()``.
 
+
 .. function:: triangular(low, high, mode)
 
    Return a random floating point number *N* such that ``low <= N <= high`` and
@@ -197,6 +198,12 @@
    Gamma distribution.  (*Not* the gamma function!)  Conditions on the
    parameters are ``alpha > 0`` and ``beta > 0``.
 
+   The probability distribution function is::
+
+                 x ** (alpha - 1) * math.exp(-x / beta)
+       pdf(x) =  --------------------------------------
+                   math.gamma(alpha) * beta ** alpha
+
 
 .. function:: gauss(mu, sigma)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 17 20:40:48 2011
From: python-checkins at python.org (georg.brandl)
Date: Sat, 17 Sep 2011 20:40:48 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Add_info_from_t?=
 =?utf8?q?he_docstring_for_random=2Egammavariate=28=29_to_the_docs=2E?=
Message-ID: <E1R4zom-0008Ew-TR@dinsdale.python.org>

http://hg.python.org/cpython/rev/169e07315ba5
changeset:   72395:169e07315ba5
branch:      2.7
parent:      72390:b5f4c4085ae6
user:        Georg Brandl <georg at python.org>
date:        Sat Sep 17 20:36:28 2011 +0200
summary:
  Add info from the docstring for random.gammavariate() to the docs.

files:
  Doc/library/random.rst |  7 +++++++
  1 files changed, 7 insertions(+), 0 deletions(-)


diff --git a/Doc/library/random.rst b/Doc/library/random.rst
--- a/Doc/library/random.rst
+++ b/Doc/library/random.rst
@@ -196,6 +196,7 @@
    The end-point value ``b`` may or may not be included in the range
    depending on floating-point rounding in the equation ``a + (b-a) * random()``.
 
+
 .. function:: triangular(low, high, mode)
 
    Return a random floating point number *N* such that ``low <= N <= high`` and
@@ -226,6 +227,12 @@
    Gamma distribution.  (*Not* the gamma function!)  Conditions on the
    parameters are ``alpha > 0`` and ``beta > 0``.
 
+   The probability distribution function is::
+
+                 x ** (alpha - 1) * math.exp(-x / beta)
+       pdf(x) =  --------------------------------------
+                   math.gamma(alpha) * beta ** alpha
+
 
 .. function:: gauss(mu, sigma)
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Sun Sep 18 05:20:14 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sun, 18 Sep 2011 05:20:14 +0200
Subject: [Python-checkins] Daily reference leaks (23e4b73937f7): sum=0
Message-ID: <E1R57vS-0004v8-B3@ap.vmr.nerim.net>

results for 23e4b73937f7 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogGJlgfI', '-x']

From python-checkins at python.org  Sun Sep 18 07:37:55 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 18 Sep 2011 07:37:55 +0200
Subject: [Python-checkins] =?utf8?q?devguide=3A_Add_Meador=2E?=
Message-ID: <E1R5A4h-0005Qk-4S@dinsdale.python.org>

http://hg.python.org/devguide/rev/9216932e781e
changeset:   451:9216932e781e
user:        Georg Brandl <georg at python.org>
date:        Sun Sep 18 07:39:15 2011 +0200
summary:
  Add Meador.

files:
  developers.rst |  3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)


diff --git a/developers.rst b/developers.rst
--- a/developers.rst
+++ b/developers.rst
@@ -24,6 +24,9 @@
 Permissions History
 -------------------
 
+- Meador Inge was given push privileges on Sep 19 2011 by GFB, for
+  general contributions, on recommendation by Mark Dickinson.
+
 - Sandro Tosi was given push privileges on Aug 1 2011 by Antoine Pitrou,
   for documentation and other contributions, on recommendation by Ezio
   Melotti, R. David Murray and others.

-- 
Repository URL: http://hg.python.org/devguide

From python-checkins at python.org  Sun Sep 18 07:39:11 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 18 Sep 2011 07:39:11 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_bug_in_heap?=
 =?utf8?q?q_priority_queue_example=2E?=
Message-ID: <E1R5A5v-0005Uc-F0@dinsdale.python.org>

http://hg.python.org/cpython/rev/5deecc04b7a2
changeset:   72396:5deecc04b7a2
branch:      3.2
parent:      72393:e6407b2d2f0c
user:        Georg Brandl <georg at python.org>
date:        Sun Sep 18 07:40:05 2011 +0200
summary:
  Fix bug in heapq priority queue example.

files:
  Doc/library/heapq.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/heapq.rst b/Doc/library/heapq.rst
--- a/Doc/library/heapq.rst
+++ b/Doc/library/heapq.rst
@@ -191,8 +191,8 @@
     def get_top_priority():
         while True:
             priority, count, task = heappop(pq)
-            del task_finder[task]
             if count is not INVALID:
+                del task_finder[task]
                 return task
 
     def delete_task(task):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep 18 07:39:12 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 18 Sep 2011 07:39:12 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_example_fix_from_3=2E2=2E?=
Message-ID: <E1R5A5w-0005Ug-5o@dinsdale.python.org>

http://hg.python.org/cpython/rev/fcd29041c991
changeset:   72397:fcd29041c991
parent:      72394:23e4b73937f7
parent:      72396:5deecc04b7a2
user:        Georg Brandl <georg at python.org>
date:        Sun Sep 18 07:40:20 2011 +0200
summary:
  Merge example fix from 3.2.

files:
  Doc/library/heapq.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/heapq.rst b/Doc/library/heapq.rst
--- a/Doc/library/heapq.rst
+++ b/Doc/library/heapq.rst
@@ -191,8 +191,8 @@
     def get_top_priority():
         while True:
             priority, count, task = heappop(pq)
-            del task_finder[task]
             if count is not INVALID:
+                del task_finder[task]
                 return task
 
     def delete_task(task):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep 18 07:39:12 2011
From: python-checkins at python.org (georg.brandl)
Date: Sun, 18 Sep 2011 07:39:12 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_bug_in_heap?=
 =?utf8?q?q_priority_queue_example=2E?=
Message-ID: <E1R5A5w-0005Uk-Si@dinsdale.python.org>

http://hg.python.org/cpython/rev/9f3595fb6c92
changeset:   72398:9f3595fb6c92
branch:      2.7
parent:      72395:169e07315ba5
user:        Georg Brandl <georg at python.org>
date:        Sun Sep 18 07:40:05 2011 +0200
summary:
  Fix bug in heapq priority queue example.

files:
  Doc/library/heapq.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/heapq.rst b/Doc/library/heapq.rst
--- a/Doc/library/heapq.rst
+++ b/Doc/library/heapq.rst
@@ -205,8 +205,8 @@
     def get_top_priority():
         while True:
             priority, count, task = heappop(pq)
-            del task_finder[task]
             if count is not INVALID:
+                del task_finder[task]
                 return task
 
     def delete_task(task):

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Mon Sep 19 05:18:46 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Mon, 19 Sep 2011 05:18:46 +0200
Subject: [Python-checkins] Daily reference leaks (fcd29041c991): sum=0
Message-ID: <E1R5UNa-0005Px-Jz@ap.vmr.nerim.net>

results for fcd29041c991 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogNRZJfu', '-x']

From python-checkins at python.org  Mon Sep 19 06:04:07 2011
From: python-checkins at python.org (meador.inge)
Date: Mon, 19 Sep 2011 06:04:07 +0200
Subject: [Python-checkins] =?utf8?q?devguide=3A_Adding_my_current_areas_to?=
	=?utf8?q?_the_experts_index=2E?=
Message-ID: <E1R5V5T-0000nc-Op@dinsdale.python.org>

http://hg.python.org/devguide/rev/437479b87672
changeset:   452:437479b87672
user:        Meador Inge <meadori at gmail.com>
date:        Sun Sep 18 23:03:48 2011 -0500
summary:
  Adding my current areas to the experts index.

files:
  experts.rst |  7 ++++---
  1 files changed, 4 insertions(+), 3 deletions(-)


diff --git a/experts.rst b/experts.rst
--- a/experts.rst
+++ b/experts.rst
@@ -87,7 +87,8 @@
 cProfile
 crypt                 jafo*
 csv                   skip.montanaro
-ctypes                theller (inactive), belopolsky, amaury.forgeotdarc
+ctypes                theller (inactive), belopolsky, amaury.forgeotdarc,
+                      meador.inge
 curses
 datetime              belopolsky
 dbm
@@ -202,7 +203,7 @@
 stat
 string                georg.brandl*
 stringprep
-struct                mark.dickinson
+struct                mark.dickinson, meador.inge
 subprocess            astrand (inactive)
 sunau
 symbol
@@ -222,7 +223,7 @@
 timeit                georg.brandl
 tkinter               gpolo
 token                 georg.brandl
-tokenize
+tokenize              meador.inge
 trace                 belopolsky
 traceback             georg.brandl*
 tty

-- 
Repository URL: http://hg.python.org/devguide

From python-checkins at python.org  Mon Sep 19 08:32:06 2011
From: python-checkins at python.org (ross.lagerwall)
Date: Mon, 19 Sep 2011 08:32:06 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312517=3A_Silence_w?=
 =?utf8?q?arning_on_windows_buildbots_=28from_7fd80c61ddaa=29=2E?=
Message-ID: <E1R5XOg-0004Hx-1R@dinsdale.python.org>

http://hg.python.org/cpython/rev/dc16b6ab8e71
changeset:   72399:dc16b6ab8e71
parent:      72397:fcd29041c991
user:        Ross Lagerwall <rosslagerwall at gmail.com>
date:        Mon Sep 19 08:30:43 2011 +0200
summary:
  Issue #12517: Silence warning on windows buildbots (from 7fd80c61ddaa).

files:
  Modules/posixmodule.c |  3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -375,6 +375,8 @@
 #endif
 #endif
 
+/* A helper used by a number of POSIX-only functions */
+#ifndef MS_WINDOWS
 static int
 _parse_off_t(PyObject* arg, void* addr)
 {
@@ -387,6 +389,7 @@
         return 0;
     return 1;
 }
+#endif
 
 #if defined _MSC_VER && _MSC_VER >= 1400
 /* Microsoft CRT in VS2005 and higher will verify that a filehandle is

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 12:26:45 2011
From: python-checkins at python.org (nick.coghlan)
Date: Mon, 19 Sep 2011 12:26:45 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Close_issue_12958_by_flaggi?=
 =?utf8?q?ng_expected_failures_in_test=5Fsocket_on_Mac_OS_X?=
Message-ID: <E1R5b3l-0008KS-I7@dinsdale.python.org>

http://hg.python.org/cpython/rev/a4e4facad164
changeset:   72400:a4e4facad164
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Mon Sep 19 20:26:31 2011 +1000
summary:
  Close issue 12958 by flagging expected failures in test_socket on Mac OS X

files:
  Doc/library/test.rst    |  11 +++++++++--
  Lib/test/support.py     |  12 ++++++++++++
  Lib/test/test_socket.py |  13 +++++++++++++
  3 files changed, 34 insertions(+), 2 deletions(-)


diff --git a/Doc/library/test.rst b/Doc/library/test.rst
--- a/Doc/library/test.rst
+++ b/Doc/library/test.rst
@@ -399,12 +399,19 @@
    otherwise.
 
 
-.. function:: skip_unless_symlink()
+.. decorator:: skip_unless_symlink()
 
    A decorator for running tests that require support for symbolic links.
 
 
-.. function:: run_with_locale(catstr, *locales)
+.. decorator:: anticipate_failure(condition)
+
+   A decorator to conditionally mark tests with
+   :func:`unittest.expectedFailure`. Any use of this decorator should
+   have an associated comment identifying the relevant tracker issue.
+
+
+.. decorator:: run_with_locale(catstr, *locales)
 
    A decorator for running a function in a different locale, correctly
    resetting it after it has finished.  *catstr* is the locale category as
diff --git a/Lib/test/support.py b/Lib/test/support.py
--- a/Lib/test/support.py
+++ b/Lib/test/support.py
@@ -57,6 +57,7 @@
     "get_attribute", "swap_item", "swap_attr", "requires_IEEE_754",
     "TestHandler", "Matcher", "can_symlink", "skip_unless_symlink",
     "import_fresh_module", "requires_zlib", "PIPE_MAX_SIZE", "failfast",
+    "anticipate_failure"
     ]
 
 class Error(Exception):
@@ -127,6 +128,17 @@
     return saved
 
 
+def anticipate_failure(condition):
+    """Decorator to mark a test that is known to be broken in some cases
+
+       Any use of this decorator should have a comment identifying the
+       associated tracker issue.
+    """
+    if condition:
+        return unittest.expectedFailure
+    return lambda f: f
+
+
 def import_fresh_module(name, fresh=(), blocked=(), deprecated=False):
     """Imports and returns a module, deliberately bypassing the sys.modules cache
     and importing a fresh copy of the module. Once the import is complete,
diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -167,6 +167,9 @@
             raise TypeError("test_func must be a callable function")
         try:
             test_func()
+        except unittest._ExpectedFailure:
+            # We deliberately ignore expected failures
+            pass
         except BaseException as e:
             self.queue.put(e)
         finally:
@@ -2090,6 +2093,8 @@
     def _testFDPassCMSG_LEN(self):
         self.createAndSendFDs(1)
 
+    # Issue #12958: The following test has problems on Mac OS X
+    @support.anticipate_failure(sys.platform == "darwin")
     @requireAttrs(socket, "CMSG_SPACE")
     def testFDPassSeparate(self):
         # Pass two FDs in two separate arrays.  Arrays may be combined
@@ -2099,6 +2104,7 @@
                              maxcmsgs=2)
 
     @testFDPassSeparate.client_skip
+    @support.anticipate_failure(sys.platform == "darwin")
     def _testFDPassSeparate(self):
         fd0, fd1 = self.newFDs(2)
         self.assertEqual(
@@ -2110,6 +2116,8 @@
                                           array.array("i", [fd1]))]),
             len(MSG))
 
+    # Issue #12958: The following test has problems on Mac OS X
+    @support.anticipate_failure(sys.platform == "darwin")
     @requireAttrs(socket, "CMSG_SPACE")
     def testFDPassSeparateMinSpace(self):
         # Pass two FDs in two separate arrays, receiving them into the
@@ -2121,6 +2129,7 @@
                              maxcmsgs=2, ignoreflags=socket.MSG_CTRUNC)
 
     @testFDPassSeparateMinSpace.client_skip
+    @support.anticipate_failure(sys.platform == "darwin")
     def _testFDPassSeparateMinSpace(self):
         fd0, fd1 = self.newFDs(2)
         self.assertEqual(
@@ -3024,9 +3033,12 @@
         self.assertNotIsInstance(cm.exception, socket.timeout)
         self.assertEqual(cm.exception.errno, errno.EINTR)
 
+    # Issue #12958: The following tests have problems on Mac OS X
+    @support.anticipate_failure(sys.platform == "darwin")
     def testInterruptedSendTimeout(self):
         self.checkInterruptedSend(self.serv_conn.send, b"a"*512)
 
+    @support.anticipate_failure(sys.platform == "darwin")
     def testInterruptedSendtoTimeout(self):
         # Passing an actual address here as Python's wrapper for
         # sendto() doesn't allow passing a zero-length one; POSIX
@@ -3035,6 +3047,7 @@
         self.checkInterruptedSend(self.serv_conn.sendto, b"a"*512,
                                   self.serv_addr)
 
+    @support.anticipate_failure(sys.platform == "darwin")
     @requireAttrs(socket.socket, "sendmsg")
     def testInterruptedSendmsgTimeout(self):
         self.checkInterruptedSend(self.serv_conn.sendmsg, [b"a"*512])

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Update_some_setup=2Ecfg_?=
	=?utf8?q?fields=2E?=
Message-ID: <E1R5deJ-0007kw-12@dinsdale.python.org>

http://hg.python.org/distutils2/rev/6b80061ac34a
changeset:   1147:6b80061ac34a
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 15 14:53:42 2011 +0200
summary:
  Update some setup.cfg fields.

- PSF license ? Python license
- Tarek is not the sole author, let?s direct people to the fellowship ML
(the same author is used for the documentation)

files:
  setup.cfg |  8 +++++---
  1 files changed, 5 insertions(+), 3 deletions(-)


diff --git a/setup.cfg b/setup.cfg
--- a/setup.cfg
+++ b/setup.cfg
@@ -4,9 +4,11 @@
 summary = Python Distribution Utilities
 description-file = README.txt
 home-page = http://bitbucket.org/tarek/distutils2/wiki/Home
-author = Tarek Ziade
-author-email = tarek at ziade.org
-license = PSF
+author = The Fellowship of the Packaging
+author-email = the-fellowship-of-the-packaging at googlegroups.com
+# we set a custom license field in addition to the classifier below
+# because the Python license is not just the PSF License
+license = Python license
 keywords = packaging, distutils
 classifier =
     Development Status :: 3 - Alpha

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Minor=3A_reorganize_sect?=
 =?utf8?q?ions_and_fields_in_setup=2Ecfg?=
Message-ID: <E1R5deJ-0007kn-0E@dinsdale.python.org>

http://hg.python.org/distutils2/rev/bb57064b93b6
changeset:   1146:bb57064b93b6
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 15 14:51:31 2011 +0200
summary:
  Minor: reorganize sections and fields in setup.cfg

files:
  setup.cfg |  22 ++++++++++++----------
  1 files changed, 12 insertions(+), 10 deletions(-)


diff --git a/setup.cfg b/setup.cfg
--- a/setup.cfg
+++ b/setup.cfg
@@ -1,10 +1,3 @@
-[build_ext]
-# needed so that tests work without mucking with sys.path
-inplace = 1
-
-[upload_docs]
-upload-dir = docs/build/html
-
 [metadata]
 name = Distutils2
 version = 1.0a3
@@ -13,6 +6,8 @@
 home-page = http://bitbucket.org/tarek/distutils2/wiki/Home
 author = Tarek Ziade
 author-email = tarek at ziade.org
+license = PSF
+keywords = packaging, distutils
 classifier =
     Development Status :: 3 - Alpha
     Intended Audience :: Developers
@@ -23,8 +18,6 @@
     Topic :: System :: Archiving :: Packaging
     Topic :: System :: Systems Administration
     Topic :: Utilities
-license = PSF
-keywords = packaging,distutils
 
 [files]
 packages =
@@ -40,4 +33,13 @@
     distutils2._backport = sysconfig.cfg
     distutils2.command = wininst*.exe
 scripts =
-    pysetup
\ No newline at end of file
+    pysetup
+
+# TODO build hashlib for Python < 2.4
+
+[build_ext]
+# needed so that tests work without mucking with sys.path
+inplace = 1
+
+[upload_docs]
+upload-dir = docs/build/html

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_database=2EDistribut?=
 =?utf8?b?aW9uLmxpc3RfZGlzdGluZm9fZmlsZXMgKCMxMjc4NSku?=
Message-ID: <E1R5deJ-0007l8-89@dinsdale.python.org>

http://hg.python.org/distutils2/rev/5e49c9bc43eb
changeset:   1150:5e49c9bc43eb
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Sep 18 21:12:17 2011 +0200
summary:
  Fix database.Distribution.list_distinfo_files (#12785).

This method was supposed to return only the file under the dist-info
directory, but it actually returned all installed files.

The tests didn?t catch this because they were flawed; I updated them.

files:
  distutils2/database.py            |   4 +-
  distutils2/tests/test_database.py |  52 +++++++++---------
  2 files changed, 29 insertions(+), 27 deletions(-)


diff --git a/distutils2/database.py b/distutils2/database.py
--- a/distutils2/database.py
+++ b/distutils2/database.py
@@ -273,7 +273,9 @@
         :returns: iterator of paths
         """
         for path, checksum, size in self._get_records(local):
-            yield path
+            # XXX add separator or use real relpath algo
+            if path.startswith(self.path):
+                yield path
 
     def __eq__(self, other):
         return isinstance(other, Distribution) and self.path == other.path
diff --git a/distutils2/tests/test_database.py b/distutils2/tests/test_database.py
--- a/distutils2/tests/test_database.py
+++ b/distutils2/tests/test_database.py
@@ -4,10 +4,6 @@
 import shutil
 import tempfile
 try:
-    from os.path import relpath  # separate import for backport concerns
-except:
-    from distutils2._backport.path import relpath
-try:
     from hashlib import md5
 except ImportError:
     from distutils2._backport.hashlib import md5
@@ -40,11 +36,11 @@
     return checksum.hexdigest()
 
 
-def record_pieces(file):
-    path = relpath(file, sys.prefix)
-    digest = get_hexdigest(file)
-    size = os.path.getsize(file)
-    return [path, digest, size]
+def record_pieces(path):
+    path = os.path.join(*path)
+    digest = get_hexdigest(path)
+    size = os.path.getsize(path)
+    return path, digest, size
 
 
 class FakeDistsMixin(object):
@@ -58,7 +54,8 @@
         # distributions
         tmpdir = tempfile.mkdtemp()
         self.addCleanup(shutil.rmtree, tmpdir)
-        self.fake_dists_path = os.path.join(tmpdir, 'fake_dists')
+        self.fake_dists_path = os.path.realpath(
+            os.path.join(tmpdir, 'fake_dists'))
         fake_dists_src = os.path.abspath(
             os.path.join(os.path.dirname(__file__), 'fake_dists'))
         shutil.copytree(fake_dists_src, self.fake_dists_path)
@@ -151,12 +148,10 @@
 
                 for path, dirs, files in os.walk(dist_location):
                     for f in files:
-                        record_writer.writerow(record_pieces(
-                                               os.path.join(path, f)))
+                        record_writer.writerow(record_pieces((path, f)))
                 for file in ('INSTALLER', 'METADATA', 'REQUESTED'):
-                    record_writer.writerow(record_pieces(
-                                           os.path.join(distinfo_dir, file)))
-                record_writer.writerow([relpath(record_file, sys.prefix)])
+                    record_writer.writerow(record_pieces((distinfo_dir, file)))
+                record_writer.writerow([record_file])
             finally:
                 fp.close()
 
@@ -186,15 +181,17 @@
                                     distinfo_name + '.dist-info')
         true_path = [self.fake_dists_path, distinfo_name,
                      'grammar', 'utils.py']
-        true_path = relpath(os.path.join(*true_path), sys.prefix)
+        true_path = os.path.join(*true_path)
         false_path = [self.fake_dists_path, 'towel_stuff-0.1', 'towel_stuff',
                       '__init__.py']
-        false_path = relpath(os.path.join(*false_path), sys.prefix)
+        false_path = os.path.join(*false_path)
 
         # Test if the distribution uses the file in question
         dist = Distribution(distinfo_dir)
-        self.assertTrue(dist.uses(true_path))
-        self.assertFalse(dist.uses(false_path))
+        self.assertTrue(dist.uses(true_path), 'dist %r is supposed to use %r' %
+                        (dist, true_path))
+        self.assertFalse(dist.uses(false_path), 'dist %r is not supposed to '
+                         'use %r' % (dist, true_path))
 
     def test_get_distinfo_file(self):
         # Test the retrieval of dist-info file objects.
@@ -233,20 +230,23 @@
                           'MAGICFILE')
 
     def test_list_distinfo_files(self):
-        # Test for the iteration of RECORD path entries.
         distinfo_name = 'towel_stuff-0.1'
         distinfo_dir = os.path.join(self.fake_dists_path,
                                     distinfo_name + '.dist-info')
         dist = Distribution(distinfo_dir)
         # Test for the iteration of the raw path
-        distinfo_record_paths = self.records[distinfo_dir].keys()
+        distinfo_files = [os.path.join(distinfo_dir, filename) for filename in
+                          os.listdir(distinfo_dir)]
         found = dist.list_distinfo_files()
-        self.assertEqual(sorted(found), sorted(distinfo_record_paths))
+        self.assertEqual(sorted(found), sorted(distinfo_files))
         # Test for the iteration of local absolute paths
-        distinfo_record_paths = [os.path.join(sys.prefix, path)
-            for path in self.records[distinfo_dir]]
-        found = dist.list_distinfo_files(local=True)
-        self.assertEqual(sorted(found), sorted(distinfo_record_paths))
+        distinfo_files = [os.path.join(sys.prefix, distinfo_dir, path) for
+                          path in distinfo_files]
+        found = sorted(dist.list_distinfo_files(local=True))
+        if os.sep != '/':
+            self.assertNotIn('/', found[0])
+            self.assertIn(os.sep, found[0])
+        self.assertEqual(found, sorted(distinfo_files))
 
     def test_get_resources_path(self):
         distinfo_name = 'babar-0.1'

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_the_backport_fixes?=
	=?utf8?q?=2E?=
Message-ID: <E1R5deJ-0007l1-4O@dinsdale.python.org>

http://hg.python.org/distutils2/rev/12f3ba3cbac2
changeset:   1148:12f3ba3cbac2
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Sep 18 20:20:13 2011 +0200
summary:
  Fix the backport fixes.

Backports:
- sysconfig is now always imported from our backports
- when hashlib is not found, our backport is used instead of the md5
  module (debatable; we could just drop hashlib)

Version-dependent features:
- PEP 370 features are only enabled for 2.6+
- the check for sys.dont_write_bytecode was fixed to use getattr
  with a default value instead of hasattr

Idioms/syntax:
- octal literals lost their extra 0
- misused try/except blocks have been changed back to try/finally
  (it?s legal in 2.4 too, it?s only try/except/finally that isn?t)
- exception catching uses the regular 2.x idiom instead of sys.exc_info
- file objects are closed within finally blocks (this causes much
  whitespace changes but actually makes diff with packaging easier)

Renamed modules:
- some missed renamings (_thread, Queue, isAlive, urllib.urlsplit, etc.)
  were fixed

Other:
- a few false positive replacements of ?packaging? by ?distutils2? in
  comments or docstrings were reverted
- util.is_packaging regained its name
- assorted whitespace/comment/import changes to match packaging

files:
  distutils2/__init__.py                       |    2 +-
  distutils2/_backport/tests/test_sysconfig.py |    3 +-
  distutils2/command/__init__.py               |    5 +-
  distutils2/command/bdist.py                  |   12 +-
  distutils2/command/bdist_dumb.py             |   15 +-
  distutils2/command/bdist_msi.py              |   32 +-
  distutils2/command/bdist_wininst.py          |   83 ++-
  distutils2/command/build_clib.py             |    3 +-
  distutils2/command/build_ext.py              |   30 +-
  distutils2/command/build_py.py               |    4 +-
  distutils2/command/build_scripts.py          |   10 +-
  distutils2/command/cmd.py                    |    7 +-
  distutils2/command/config.py                 |   43 +-
  distutils2/command/install_data.py           |    5 +-
  distutils2/command/install_dist.py           |   12 +-
  distutils2/command/install_distinfo.py       |    4 +-
  distutils2/command/install_lib.py            |    4 +-
  distutils2/command/install_scripts.py        |    2 +-
  distutils2/command/register.py               |   12 +-
  distutils2/command/sdist.py                  |    8 +-
  distutils2/command/upload.py                 |   23 +-
  distutils2/command/upload_docs.py            |   27 +-
  distutils2/compiler/bcppcompiler.py          |   20 +-
  distutils2/compiler/ccompiler.py             |   14 +-
  distutils2/compiler/cygwinccompiler.py       |   23 +-
  distutils2/compiler/msvc9compiler.py         |   26 +-
  distutils2/compiler/msvccompiler.py          |   22 +-
  distutils2/compiler/unixccompiler.py         |   18 +-
  distutils2/config.py                         |   24 +-
  distutils2/create.py                         |  128 +++--
  distutils2/database.py                       |   61 +-
  distutils2/depgraph.py                       |    9 +-
  distutils2/dist.py                           |   57 +-
  distutils2/fancy_getopt.py                   |    6 +-
  distutils2/install.py                        |   37 +-
  distutils2/manifest.py                       |   21 +-
  distutils2/markers.py                        |   10 +-
  distutils2/metadata.py                       |   20 +-
  distutils2/pypi/dist.py                      |   10 +-
  distutils2/pypi/simple.py                    |   45 +-
  distutils2/pypi/wrapper.py                   |    4 +-
  distutils2/pypi/xmlrpc.py                    |    5 +-
  distutils2/run.py                            |   35 +-
  distutils2/tests/__main__.py                 |    8 +-
  distutils2/tests/pypi_server.py              |   22 +-
  distutils2/tests/test_command_build.py       |    2 +-
  distutils2/tests/test_command_build_ext.py   |    9 +-
  distutils2/tests/test_command_upload_docs.py |    5 +-
  distutils2/tests/test_install.py             |    3 +-
  distutils2/tests/test_mixin2to3.py           |   10 +-
  distutils2/tests/test_pypi_simple.py         |    4 +-
  distutils2/util.py                           |  203 +++++----
  distutils2/version.py                        |    1 -
  53 files changed, 646 insertions(+), 562 deletions(-)


diff --git a/distutils2/__init__.py b/distutils2/__init__.py
--- a/distutils2/__init__.py
+++ b/distutils2/__init__.py
@@ -1,4 +1,4 @@
-"""Support for distutils2, distribution and installation of Python projects.
+"""Support for packaging, distribution and installation of Python projects.
 
 Third-party tools can use parts of distutils2 as building blocks
 without causing the other modules to be imported:
diff --git a/distutils2/_backport/tests/test_sysconfig.py b/distutils2/_backport/tests/test_sysconfig.py
--- a/distutils2/_backport/tests/test_sysconfig.py
+++ b/distutils2/_backport/tests/test_sysconfig.py
@@ -254,7 +254,8 @@
         # Issue 7880
         def get(python):
             cmd = [python, '-c',
-                   'import sysconfig; print sysconfig.get_platform()']
+                   'from distutils2._backport import sysconfig; '
+                   'print sysconfig.get_platform()']
             p = subprocess.Popen(cmd, stdout=subprocess.PIPE, env=os.environ)
             return p.communicate()
         real = os.path.realpath(sys.executable)
diff --git a/distutils2/command/__init__.py b/distutils2/command/__init__.py
--- a/distutils2/command/__init__.py
+++ b/distutils2/command/__init__.py
@@ -28,8 +28,11 @@
     'bdist_wininst': 'distutils2.command.bdist_wininst.bdist_wininst',
     'register': 'distutils2.command.register.register',
     'upload': 'distutils2.command.upload.upload',
-    'upload_docs': 'distutils2.command.upload_docs.upload_docs'}
+    'upload_docs': 'distutils2.command.upload_docs.upload_docs',
+}
 
+# XXX use OrderedDict to preserve the grouping (build-related, install-related,
+# distribution-related)
 STANDARD_COMMANDS = set(_COMMANDS)
 
 
diff --git a/distutils2/command/bdist.py b/distutils2/command/bdist.py
--- a/distutils2/command/bdist.py
+++ b/distutils2/command/bdist.py
@@ -58,7 +58,7 @@
     # This is of course very simplistic.  The various UNIX family operating
     # systems have their specific formats, but they are out of scope for us;
     # bdist_dumb is, well, dumb; it's more a building block for other
-    # distutils2 tools than a real end-user binary format.
+    # packaging tools than a real end-user binary format.
     default_format = {'posix': 'gztar',
                       'nt': 'zip',
                       'os2': 'zip'}
@@ -75,9 +75,8 @@
                       'wininst': ('bdist_wininst',
                                   "Windows executable installer"),
                       'zip':   ('bdist_dumb', "ZIP file"),
-                      'msi':   ('bdist_msi',  "Microsoft Installer")
-                      }
-
+                      'msi':   ('bdist_msi',  "Microsoft Installer"),
+                     }
 
     def initialize_options(self):
         self.bdist_base = None
@@ -109,8 +108,9 @@
             try:
                 self.formats = [self.default_format[os.name]]
             except KeyError:
-                raise PackagingPlatformError("don't know how to create built distributions " + \
-                      "on platform %s" % os.name)
+                raise PackagingPlatformError(
+                    "don't know how to create built distributions "
+                    "on platform %s" % os.name)
 
         if self.dist_dir is None:
             self.dist_dir = "dist"
diff --git a/distutils2/command/bdist_dumb.py b/distutils2/command/bdist_dumb.py
--- a/distutils2/command/bdist_dumb.py
+++ b/distutils2/command/bdist_dumb.py
@@ -7,11 +7,12 @@
 import os
 
 from shutil import rmtree
-from sysconfig import get_python_version
 from distutils2.util import get_platform
 from distutils2.command.cmd import Command
 from distutils2.errors import PackagingPlatformError
 from distutils2 import logger
+from distutils2._backport.sysconfig import get_python_version
+
 
 class bdist_dumb(Command):
 
@@ -44,10 +45,9 @@
 
     boolean_options = ['keep-temp', 'skip-build', 'relative']
 
-    default_format = { 'posix': 'gztar',
-                       'nt': 'zip',
-                       'os2': 'zip' }
-
+    default_format = {'posix': 'gztar',
+                      'nt': 'zip',
+                      'os2': 'zip'}
 
     def initialize_options(self):
         self.bdist_dir = None
@@ -69,8 +69,9 @@
             try:
                 self.format = self.default_format[os.name]
             except KeyError:
-                raise PackagingPlatformError(("don't know how to create dumb built distributions " +
-                       "on platform %s") % os.name)
+                raise PackagingPlatformError(
+                    "don't know how to create dumb built distributions "
+                    "on platform %s" % os.name)
 
         self.set_undefined_options('bdist', 'dist_dir', 'plat_name')
 
diff --git a/distutils2/command/bdist_msi.py b/distutils2/command/bdist_msi.py
--- a/distutils2/command/bdist_msi.py
+++ b/distutils2/command/bdist_msi.py
@@ -8,7 +8,7 @@
 import msilib
 
 
-from sysconfig import get_python_version
+from distutils2._backport.sysconfig import get_python_version
 from shutil import rmtree
 from distutils2.command.cmd import Command
 from distutils2.version import NormalizedVersion
@@ -391,19 +391,23 @@
         if self.pre_install_script:
             scriptfn = os.path.join(self.bdist_dir, "preinstall.bat")
             f = open(scriptfn, "w")
-            # The batch file will be executed with [PYTHON], so that %1
-            # is the path to the Python interpreter; %0 will be the path
-            # of the batch file.
-            # rem ="""
-            # %1 %0
-            # exit
-            # """
-            # <actual script>
-            f.write('rem ="""\n%1 %0\nexit\n"""\n')
-            fp = open(self.pre_install_script)
-            f.write(fp.read())
-            fp.close()
-            f.close()
+            try:
+                # The batch file will be executed with [PYTHON], so that %1
+                # is the path to the Python interpreter; %0 will be the path
+                # of the batch file.
+                # rem ="""
+                # %1 %0
+                # exit
+                # """
+                # <actual script>
+                f.write('rem ="""\n%1 %0\nexit\n"""\n')
+                fp = open(self.pre_install_script)
+                try:
+                    f.write(fp.read())
+                finally:
+                    fp.close()
+            finally:
+                f.close()
             add_data(self.db, "Binary",
                      [("PreInstall", msilib.Binary(scriptfn)),
                      ])
diff --git a/distutils2/command/bdist_wininst.py b/distutils2/command/bdist_wininst.py
--- a/distutils2/command/bdist_wininst.py
+++ b/distutils2/command/bdist_wininst.py
@@ -6,11 +6,11 @@
 import os
 
 from shutil import rmtree
-from sysconfig import get_python_version
 from distutils2.command.cmd import Command
 from distutils2.errors import PackagingOptionError, PackagingPlatformError
 from distutils2 import logger
 from distutils2.util import get_platform
+from distutils2._backport.sysconfig import get_python_version
 
 
 class bdist_wininst(Command):
@@ -246,49 +246,56 @@
 
         if bitmap:
             fp = open(bitmap, "rb")
-            bitmapdata = fp.read()
-            fp.close()
+            try:
+                bitmapdata = fp.read()
+            finally:
+                fp.close()
             bitmaplen = len(bitmapdata)
         else:
             bitmaplen = 0
 
         file = open(installer_name, "wb")
-        file.write(self.get_exe_bytes())
-        if bitmap:
-            file.write(bitmapdata)
+        try:
+            file.write(self.get_exe_bytes())
+            if bitmap:
+                file.write(bitmapdata)
 
-        # Convert cfgdata from unicode to ascii, mbcs encoded
-        if isinstance(cfgdata, unicode):
-            cfgdata = cfgdata.encode("mbcs")
+            # Convert cfgdata from unicode to ascii, mbcs encoded
+            if isinstance(cfgdata, unicode):
+                cfgdata = cfgdata.encode("mbcs")
 
-        # Append the pre-install script
-        cfgdata = cfgdata + "\0"
-        if self.pre_install_script:
-            fp = open(self.pre_install_script)
-            script_data = fp.read()
-            fp.close()
-            cfgdata = cfgdata + script_data + "\n\0"
-        else:
-            # empty pre-install script
+            # Append the pre-install script
             cfgdata = cfgdata + "\0"
-        file.write(cfgdata)
+            if self.pre_install_script:
+                fp = open(self.pre_install_script)
+                try:
+                    script_data = fp.read()
+                finally:
+                    fp.close()
+                cfgdata = cfgdata + script_data + "\n\0"
+            else:
+                # empty pre-install script
+                cfgdata = cfgdata + "\0"
+            file.write(cfgdata)
 
-        # The 'magic number' 0x1234567B is used to make sure that the
-        # binary layout of 'cfgdata' is what the wininst.exe binary
-        # expects.  If the layout changes, increment that number, make
-        # the corresponding changes to the wininst.exe sources, and
-        # recompile them.
-        header = struct.pack("<iii",
-                             0x1234567B,       # tag
-                             len(cfgdata),     # length
-                             bitmaplen,        # number of bytes in bitmap
-                             )
-        file.write(header)
-        file.close()
-
-        fp = open(arcname, "rb")
-        file.write(fp.read())
-        fp.close()
+            # The 'magic number' 0x1234567B is used to make sure that the
+            # binary layout of 'cfgdata' is what the wininst.exe binary
+            # expects.  If the layout changes, increment that number, make
+            # the corresponding changes to the wininst.exe sources, and
+            # recompile them.
+            header = struct.pack("<iii",
+                                 0x1234567B,       # tag
+                                 len(cfgdata),     # length
+                                 bitmaplen,        # number of bytes in bitmap
+                                 )
+            file.write(header)
+            fp = open(arcname, "rb")
+            try:
+                file.write(fp.read())
+            finally:
+                fp.close()
+        finally:
+            file.close()
 
     def get_installer_filename(self, fullname):
         # Factored out to allow overriding in subclasses
@@ -344,6 +351,8 @@
 
         filename = os.path.join(directory, "wininst-%.1f%s.exe" % (bv, sfix))
         fp = open(filename, "rb")
-        content = fp.read()
-        fp.close()
+        try:
+            content = fp.read()
+        finally:
+            fp.close()
         return content
diff --git a/distutils2/command/build_clib.py b/distutils2/command/build_clib.py
--- a/distutils2/command/build_clib.py
+++ b/distutils2/command/build_clib.py
@@ -16,7 +16,7 @@
 import os
 from distutils2.command.cmd import Command
 from distutils2.errors import PackagingSetupError
-from distutils2.compiler import customize_compiler
+from distutils2.compiler import customize_compiler, new_compiler
 from distutils2 import logger
 
 
@@ -93,7 +93,6 @@
             return
 
         # Yech -- this is cut 'n pasted from build_ext.py!
-        from distutils2.compiler import new_compiler
         self.compiler = new_compiler(compiler=self.compiler,
                                      dry_run=self.dry_run,
                                      force=self.force)
diff --git a/distutils2/command/build_ext.py b/distutils2/command/build_ext.py
--- a/distutils2/command/build_ext.py
+++ b/distutils2/command/build_ext.py
@@ -1,9 +1,5 @@
 """Build extension modules."""
 
-# FIXME Is this module limited to C extensions or do C++ extensions work too?
-# The docstring of this module said that C++ was not supported, but other
-# comments contradict that.
-
 import os
 import re
 import sys
@@ -20,7 +16,10 @@
 from distutils2 import logger
 
 import site
-HAS_USER_SITE = True
+if sys.version_info[:2] >= (2, 6):
+    HAS_USER_SITE = True
+else:
+    HAS_USER_SITE = False
 
 if os.name == 'nt':
     from distutils2.compiler.msvccompiler import get_build_version
@@ -363,12 +362,11 @@
         for ext in self.extensions:
             try:
                 self.build_extension(ext)
-            except (CCompilerError, PackagingError, CompileError):
+            except (CCompilerError, PackagingError, CompileError), e:
                 if not ext.optional:
                     raise
                 logger.warning('%s: building extension %r failed: %s',
-                               self.get_command_name(), ext.name,
-                               sys.exc_info()[1])
+                               self.get_command_name(), ext.name, e)
 
     def build_extension(self, ext):
         sources = ext.sources
@@ -608,8 +606,7 @@
                 template = "python%d%d"
                 if self.debug:
                     template = template + '_d'
-                pythonlib = (template %
-                       (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff))
+                pythonlib = template % sys.version_info[:2]
                 # don't extend ext.libraries, it may be shared with other
                 # extensions, it is a reference to the original list
                 return ext.libraries + [pythonlib]
@@ -623,22 +620,19 @@
             # not at this time - AIM Apr01
             #if self.debug:
             #    template = template + '_d'
-            pythonlib = (template %
-                   (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff))
+            pythonlib = template % sys.version_info[:2]
             # don't extend ext.libraries, it may be shared with other
             # extensions, it is a reference to the original list
             return ext.libraries + [pythonlib]
         elif sys.platform[:6] == "cygwin":
             template = "python%d.%d"
-            pythonlib = (template %
-                   (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff))
+            pythonlib = template % sys.version_info[:2]
             # don't extend ext.libraries, it may be shared with other
             # extensions, it is a reference to the original list
             return ext.libraries + [pythonlib]
         elif sys.platform[:6] == "atheos":
             template = "python%d.%d"
-            pythonlib = (template %
-                   (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff))
+            pythonlib = template % sys.version_info[:2]
             # Get SHLIBS from Makefile
             extra = []
             for lib in sysconfig.get_config_var('SHLIBS').split():
@@ -656,8 +650,8 @@
 
         else:
             if sysconfig.get_config_var('Py_ENABLE_SHARED'):
-                pythonlib = 'python%s.%s' % (
-                    sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff)
+                template = 'python%d.%d' + sys.abiflags
+                pythonlib = template % sys.version_info[:2]
                 return ext.libraries + [pythonlib]
             else:
                 return ext.libraries
diff --git a/distutils2/command/build_py.py b/distutils2/command/build_py.py
--- a/distutils2/command/build_py.py
+++ b/distutils2/command/build_py.py
@@ -388,12 +388,12 @@
                 self.build_module(module, module_file, package)
 
     def byte_compile(self, files):
-        if hasattr(sys, 'dont_write_bytecode') and sys.dont_write_bytecode:
+        if getattr(sys, 'dont_write_bytecode', False):
             logger.warning('%s: byte-compiling is disabled, skipping.',
                            self.get_command_name())
             return
 
-        from distutils2.util import byte_compile
+        from distutils2.util import byte_compile  # FIXME use compileall
         prefix = self.build_lib
         if prefix[-1] != os.sep:
             prefix = prefix + os.sep
diff --git a/distutils2/command/build_scripts.py b/distutils2/command/build_scripts.py
--- a/distutils2/command/build_scripts.py
+++ b/distutils2/command/build_scripts.py
@@ -130,9 +130,11 @@
                             "from the script encoding (%s)" % (
                                 shebang, encoding))
                     outf = open(outfile, "wb")
-                    outf.write(shebang)
-                    outf.writelines(f.readlines())
-                    outf.close()
+                    try:
+                        outf.write(shebang)
+                        outf.writelines(f.readlines())
+                    finally:
+                        outf.close()
                 if f:
                     f.close()
             else:
@@ -146,7 +148,7 @@
                     logger.info("changing mode of %s", file)
                 else:
                     oldmode = os.stat(file).st_mode & 07777
-                    newmode = (oldmode | 00555) & 07777
+                    newmode = (oldmode | 0555) & 07777
                     if newmode != oldmode:
                         logger.info("changing mode of %s from %o to %o",
                                  file, oldmode, newmode)
diff --git a/distutils2/command/cmd.py b/distutils2/command/cmd.py
--- a/distutils2/command/cmd.py
+++ b/distutils2/command/cmd.py
@@ -5,6 +5,7 @@
 from shutil import copyfile, move
 from distutils2 import util
 from distutils2 import logger
+from distutils2.util import make_archive
 from distutils2.errors import PackagingOptionError
 
 
@@ -403,9 +404,9 @@
 
     def make_archive(self, base_name, format, root_dir=None, base_dir=None,
                      owner=None, group=None):
-        return util.make_archive(base_name, format, root_dir,
-                                 base_dir, dry_run=self.dry_run,
-                                 owner=owner, group=group)
+        return make_archive(base_name, format, root_dir,
+                            base_dir, dry_run=self.dry_run,
+                            owner=owner, group=group)
 
     def make_file(self, infiles, outfile, func, args,
                   exec_msg=None, skip_msg=None, level=1):
diff --git a/distutils2/command/config.py b/distutils2/command/config.py
--- a/distutils2/command/config.py
+++ b/distutils2/command/config.py
@@ -111,14 +111,16 @@
     def _gen_temp_sourcefile(self, body, headers, lang):
         filename = "_configtest" + LANG_EXT[lang]
         file = open(filename, "w")
-        if headers:
-            for header in headers:
-                file.write("#include <%s>\n" % header)
-            file.write("\n")
-        file.write(body)
-        if body[-1] != "\n":
-            file.write("\n")
-        file.close()
+        try:
+            if headers:
+                for header in headers:
+                    file.write("#include <%s>\n" % header)
+                file.write("\n")
+            file.write(body)
+            if body[-1] != "\n":
+                file.write("\n")
+        finally:
+            file.close()
         return filename
 
     def _preprocess(self, body, headers, include_dirs, lang):
@@ -208,14 +210,17 @@
             pattern = re.compile(pattern)
 
         file = open(out)
-        match = False
-        while True:
-            line = file.readline()
-            if line == '':
-                break
-            if pattern.search(line):
-                match = True
-                break
+        try:
+            match = False
+            while True:
+                line = file.readline()
+                if line == '':
+                    break
+                if pattern.search(line):
+                    match = True
+                    break
+        finally:
+            file.close()
 
         self._clean()
         return match
@@ -347,5 +352,7 @@
     else:
         logger.info(head)
     file = open(filename)
-    logger.info(file.read())
-    file.close()
+    try:
+        logger.info(file.read())
+    finally:
+        file.close()
diff --git a/distutils2/command/install_data.py b/distutils2/command/install_data.py
--- a/distutils2/command/install_data.py
+++ b/distutils2/command/install_data.py
@@ -2,7 +2,7 @@
 
 # Contributed by Bastian Kleineidam
 
-import os, sys
+import os
 from shutil import Error
 from distutils2 import logger
 from distutils2.util import convert_path
@@ -48,8 +48,7 @@
             self.mkpath(dir_dest)
             try:
                 out = self.copy_file(_file[0], dir_dest)[0]
-            except Error:
-                e = sys.exc_info()[1]
+            except Error, e:
                 logger.warning('%s: %s', self.get_command_name(), e)
                 out = destination
 
diff --git a/distutils2/command/install_dist.py b/distutils2/command/install_dist.py
--- a/distutils2/command/install_dist.py
+++ b/distutils2/command/install_dist.py
@@ -295,9 +295,9 @@
 
         self.dump_dirs("post-expand_dirs()")
 
-        # Create directories in the home dir:
+        # Create directories under USERBASE
         if HAS_USER_SITE and self.user:
-            self.create_home_path()
+            self.create_user_dirs()
 
         # Pick the actual directory to install all modules to: either
         # install_purelib or install_platlib, depending on whether this
@@ -494,14 +494,12 @@
             attr = "install_" + name
             setattr(self, attr, change_root(self.root, getattr(self, attr)))
 
-    def create_home_path(self):
-        """Create directories under ~."""
-        if HAS_USER_SITE and not self.user:
-            return
+    def create_user_dirs(self):
+        """Create directories under USERBASE as needed."""
         home = convert_path(os.path.expanduser("~"))
         for name, path in self.config_vars.items():
             if path.startswith(home) and not os.path.isdir(path):
-                os.makedirs(path, 00700)
+                os.makedirs(path, 0700)
 
     # -- Command execution methods -------------------------------------
 
diff --git a/distutils2/command/install_distinfo.py b/distutils2/command/install_distinfo.py
--- a/distutils2/command/install_distinfo.py
+++ b/distutils2/command/install_distinfo.py
@@ -8,7 +8,7 @@
 import re
 try:
     import hashlib
-except ImportError: #<2.5
+except ImportError:
     from distutils2._backport import hashlib
 
 from distutils2.command.cmd import Command
@@ -32,7 +32,7 @@
         ('no-record', None,
          "do not generate a RECORD file"),
         ('no-resources', None,
-         "do not generate a RESSOURCES list installed file")
+         "do not generate a RESSOURCES list installed file"),
     ]
 
     boolean_options = ['requested', 'no-record', 'no-resources']
diff --git a/distutils2/command/install_lib.py b/distutils2/command/install_lib.py
--- a/distutils2/command/install_lib.py
+++ b/distutils2/command/install_lib.py
@@ -114,7 +114,7 @@
         return outfiles
 
     def byte_compile(self, files):
-        if hasattr(sys, 'dont_write_bytecode'):
+        if getattr(sys, 'dont_write_bytecode', False):
             # XXX do we want this?  because a Python runs without bytecode
             # doesn't mean that the *dists should not contain bytecode
             #--or does it?
@@ -122,7 +122,7 @@
                            self.get_command_name())
             return
 
-        from distutils2.util import byte_compile
+        from distutils2.util import byte_compile  # FIXME use compileall
 
         # Get the "--root" directory supplied to the "install_dist" command,
         # and use it as a prefix to strip off the purported filename
diff --git a/distutils2/command/install_scripts.py b/distutils2/command/install_scripts.py
--- a/distutils2/command/install_scripts.py
+++ b/distutils2/command/install_scripts.py
@@ -48,7 +48,7 @@
                 if self.dry_run:
                     logger.info("changing mode of %s", file)
                 else:
-                    mode = (os.stat(file).st_mode | 00555) & 07777
+                    mode = (os.stat(file).st_mode | 0555) & 07777
                     logger.info("changing mode of %s to %o", file, mode)
                     os.chmod(file, mode)
 
diff --git a/distutils2/command/register.py b/distutils2/command/register.py
--- a/distutils2/command/register.py
+++ b/distutils2/command/register.py
@@ -2,14 +2,13 @@
 
 # Contributed by Richard Jones
 
-import sys
 import getpass
+import urllib2
 import urlparse
-import urllib2
 
 from distutils2 import logger
 from distutils2.util import (read_pypirc, generate_pypirc, DEFAULT_REPOSITORY,
-                            DEFAULT_REALM, get_pypirc_path, encode_multipart)
+                             DEFAULT_REALM, get_pypirc_path, encode_multipart)
 from distutils2.command.cmd import Command
 
 class register(Command):
@@ -246,13 +245,12 @@
         data = ''
         try:
             result = opener.open(req)
-        except urllib2.HTTPError:
-            e = sys.exc_info()[1]
+        except urllib2.HTTPError, e:
             if self.show_response:
                 data = e.fp.read()
             result = e.code, e.msg
-        except urllib2.URLError:
-            result = 500, str(sys.exc_info()[1])
+        except urllib2.URLError, e:
+            result = 500, str(e)
         else:
             if self.show_response:
                 data = result.read()
diff --git a/distutils2/command/sdist.py b/distutils2/command/sdist.py
--- a/distutils2/command/sdist.py
+++ b/distutils2/command/sdist.py
@@ -9,7 +9,7 @@
 from distutils2 import logger
 from distutils2.util import resolve_name, get_archive_formats
 from distutils2.errors import (PackagingPlatformError, PackagingOptionError,
-                              PackagingModuleError, PackagingFileError)
+                               PackagingModuleError, PackagingFileError)
 from distutils2.command import get_command_names
 from distutils2.command.cmd import Command
 from distutils2.manifest import Manifest
@@ -143,8 +143,8 @@
                     continue
                 try:
                     builder = resolve_name(builder)
-                except ImportError:
-                    raise PackagingModuleError(sys.exc_info()[1])
+                except ImportError, e:
+                    raise PackagingModuleError(e)
 
                 builders.append(builder)
 
@@ -337,7 +337,7 @@
         """
         return self.archive_files
 
-    def create_tree(self, base_dir, files, mode=00777, verbose=1,
+    def create_tree(self, base_dir, files, mode=0777, verbose=1,
                     dry_run=False):
         need_dir = set()
         for file in files:
diff --git a/distutils2/command/upload.py b/distutils2/command/upload.py
--- a/distutils2/command/upload.py
+++ b/distutils2/command/upload.py
@@ -1,6 +1,6 @@
 """Upload a distribution to a project index."""
 
-import os, sys
+import os
 import socket
 import logging
 import platform
@@ -16,7 +16,7 @@
 from distutils2 import logger
 from distutils2.errors import PackagingOptionError
 from distutils2.util import (spawn, read_pypirc, DEFAULT_REPOSITORY,
-                            DEFAULT_REALM, encode_multipart)
+                             DEFAULT_REALM, encode_multipart)
 from distutils2.command.cmd import Command
 
 
@@ -105,8 +105,10 @@
         # Fill in the data - send all the metadata in case we need to
         # register a new release
         f = open(filename, 'rb')
-        content = f.read()
-        f.close()
+        try:
+            content = f.read()
+        finally:
+            f.close()
 
         data = self.distribution.metadata.todict()
 
@@ -123,8 +125,10 @@
 
         if self.sign:
             fp = open(filename + '.asc')
-            sig = fp.read()
-            fp.close()
+            try:
+                sig = fp.read()
+            finally:
+                fp.close()
             data['gpg_signature'] = [
                 (os.path.basename(filename) + ".asc", sig)]
 
@@ -156,11 +160,10 @@
             result = urlopen(request)
             status = result.code
             reason = result.msg
-        except socket.error:
-            logger.error(sys.exc_info()[1])
+        except socket.error, e:
+            logger.error(e)
             return
-        except HTTPError:
-            e = sys.exc_info()[1]
+        except HTTPError, e:
             status = e.code
             reason = e.msg
 
diff --git a/distutils2/command/upload_docs.py b/distutils2/command/upload_docs.py
--- a/distutils2/command/upload_docs.py
+++ b/distutils2/command/upload_docs.py
@@ -1,6 +1,6 @@
 """Upload HTML documentation to a project index."""
 
-import os, sys
+import os
 import base64
 import socket
 import zipfile
@@ -11,7 +11,7 @@
 
 from distutils2 import logger
 from distutils2.util import (read_pypirc, DEFAULT_REPOSITORY, DEFAULT_REALM,
-                            encode_multipart)
+                             encode_multipart)
 from distutils2.errors import PackagingFileError
 from distutils2.command.cmd import Command
 
@@ -20,13 +20,15 @@
     """Compresses recursively contents of directory into a BytesIO object"""
     destination = StringIO()
     zip_file = zipfile.ZipFile(destination, "w")
-    for root, dirs, files in os.walk(directory):
-        for name in files:
-            full = os.path.join(root, name)
-            relative = root[len(directory):].lstrip(os.path.sep)
-            dest = os.path.join(relative, name)
-            zip_file.write(full, dest)
-    zip_file.close()
+    try:
+        for root, dirs, files in os.walk(directory):
+            for name in files:
+                full = os.path.join(root, name)
+                relative = root[len(directory):].lstrip(os.path.sep)
+                dest = os.path.join(relative, name)
+                zip_file.write(full, dest)
+    finally:
+        zip_file.close()
     return destination
 
 
@@ -88,7 +90,8 @@
         content_type, body = encode_multipart(fields, files)
 
         credentials = self.username + ':' + self.password
-        auth = "Basic " + base64.encodebytes(credentials.encode()).strip()
+        # FIXME should use explicit encoding
+        auth = "Basic " + base64.encodestring(credentials.encode()).strip()
 
         logger.info("Submitting documentation to %s", self.repository)
 
@@ -110,8 +113,8 @@
             conn.endheaders()
             conn.send(body)
 
-        except socket.error:
-            logger.error(sys.exc_info()[1])
+        except socket.error, e:
+            logger.error(e)
             return
 
         r = conn.getresponse()
diff --git a/distutils2/compiler/bcppcompiler.py b/distutils2/compiler/bcppcompiler.py
--- a/distutils2/compiler/bcppcompiler.py
+++ b/distutils2/compiler/bcppcompiler.py
@@ -7,10 +7,10 @@
 # someone should sit down and factor out the common code as
 # WindowsCCompiler!  --GPW
 
-import os, sys
+import os
 
 from distutils2.errors import (PackagingExecError, CompileError, LibError,
-                              LinkError, UnknownFileError)
+                               LinkError, UnknownFileError)
 from distutils2.compiler.ccompiler import CCompiler
 from distutils2.compiler import gen_preprocess_options
 from distutils2.file_util import write_file
@@ -104,8 +104,8 @@
                 # This needs to be compiled to a .res file -- do it now.
                 try:
                     self.spawn(["brcc32", "-fo", obj, src])
-                except PackagingExecError:
-                    raise CompileError(sys.exc_info()[1])
+                except PackagingExecError, msg:
+                    raise CompileError(msg)
                 continue # the 'for' loop
 
             # The next two are both for the real compiler.
@@ -128,8 +128,8 @@
                 self.spawn([self.cc] + compile_opts + pp_opts +
                            [input_opt, output_opt] +
                            extra_postargs + [src])
-            except PackagingExecError:
-                raise CompileError(sys.exc_info()[1])
+            except PackagingExecError, msg:
+                raise CompileError(msg)
 
         return objects
 
@@ -146,8 +146,8 @@
                 pass                    # XXX what goes here?
             try:
                 self.spawn([self.lib] + lib_args)
-            except PackagingExecError:
-                raise LibError(sys.exc_info()[1])
+            except PackagingExecError, msg:
+                raise LibError(msg)
         else:
             logger.debug("skipping %s (up-to-date)", output_filename)
 
@@ -268,8 +268,8 @@
             self.mkpath(os.path.dirname(output_filename))
             try:
                 self.spawn([self.linker] + ld_args)
-            except PackagingExecError:
-                raise LinkError(sys.exc_info()[1])
+            except PackagingExecError, msg:
+                raise LinkError(msg)
 
         else:
             logger.debug("skipping %s (up-to-date)", output_filename)
diff --git a/distutils2/compiler/ccompiler.py b/distutils2/compiler/ccompiler.py
--- a/distutils2/compiler/ccompiler.py
+++ b/distutils2/compiler/ccompiler.py
@@ -8,7 +8,7 @@
 from shutil import move
 from distutils2 import logger
 from distutils2.util import split_quoted, execute, newer_group, spawn
-from distutils2.errors import (CompileError, LinkError, UnknownFileError)
+from distutils2.errors import CompileError, LinkError, UnknownFileError
 from distutils2.compiler import gen_preprocess_options
 
 
@@ -728,14 +728,16 @@
             library_dirs = []
         fd, fname = tempfile.mkstemp(".c", funcname, text=True)
         f = os.fdopen(fd, "w")
-        for incl in includes:
-            f.write("""#include "%s"\n""" % incl)
-        f.write("""\
+        try:
+            for incl in includes:
+                f.write("""#include "%s"\n""" % incl)
+            f.write("""\
 main (int argc, char **argv) {
     %s();
 }
 """ % funcname)
-        f.close()
+        finally:
+            f.close()
         try:
             objects = self.compile([fname], include_dirs=include_dirs)
         except CompileError:
@@ -852,7 +854,7 @@
             return
         return move(src, dst)
 
-    def mkpath(self, name, mode=00777):
+    def mkpath(self, name, mode=0777):
         name = os.path.normpath(name)
         if os.path.isdir(name) or name == '':
             return
diff --git a/distutils2/compiler/cygwinccompiler.py b/distutils2/compiler/cygwinccompiler.py
--- a/distutils2/compiler/cygwinccompiler.py
+++ b/distutils2/compiler/cygwinccompiler.py
@@ -156,14 +156,14 @@
             # gcc needs '.res' and '.rc' compiled to object files !!!
             try:
                 self.spawn(["windres", "-i", src, "-o", obj])
-            except PackagingExecError:
-                raise CompileError(sys.exc_info()[1])
+            except PackagingExecError, msg:
+                raise CompileError(msg)
         else: # for other files use the C-compiler
             try:
                 self.spawn(self.compiler_so + cc_args + [src, '-o', obj] +
                            extra_postargs)
-            except PackagingExecError:
-                raise CompileError(sys.exc_info()[1])
+            except PackagingExecError, msg:
+                raise CompileError(msg)
 
     def link(self, target_desc, objects, output_filename, output_dir=None,
              libraries=None, library_dirs=None, runtime_library_dirs=None,
@@ -345,12 +345,13 @@
     fn = sysconfig.get_config_h_filename()
     try:
         config_h = open(fn)
-        if "__GNUC__" in config_h.read():
-            return CONFIG_H_OK, "'%s' mentions '__GNUC__'" % fn
-        else:
-            return CONFIG_H_NOTOK, "'%s' does not mention '__GNUC__'" % fn
-        config_h.close()
-    except IOError:
-        exc = sys.exc_info()[1]
+        try:
+            if "__GNUC__" in config_h.read():
+                return CONFIG_H_OK, "'%s' mentions '__GNUC__'" % fn
+            else:
+                return CONFIG_H_NOTOK, "'%s' does not mention '__GNUC__'" % fn
+        finally:
+            config_h.close()
+    except IOError, exc:
         return (CONFIG_H_UNCERTAIN,
                 "couldn't read '%s': %s" % (fn, exc.strerror))
diff --git a/distutils2/compiler/msvc9compiler.py b/distutils2/compiler/msvc9compiler.py
--- a/distutils2/compiler/msvc9compiler.py
+++ b/distutils2/compiler/msvc9compiler.py
@@ -14,7 +14,7 @@
 import re
 
 from distutils2.errors import (PackagingExecError, PackagingPlatformError,
-                              CompileError, LibError, LinkError)
+                               CompileError, LibError, LinkError)
 from distutils2.compiler.ccompiler import CCompiler
 from distutils2.compiler import gen_lib_options
 from distutils2 import logger
@@ -477,8 +477,8 @@
                 try:
                     self.spawn([self.rc] + pp_opts +
                                [output_opt] + [input_opt])
-                except PackagingExecError:
-                    raise CompileError(sys.exc_info()[1])
+                except PackagingExecError, msg:
+                    raise CompileError(msg)
                 continue
             elif ext in self._mc_extensions:
                 # Compile .MC to .RC file to .RES file.
@@ -504,8 +504,8 @@
                     self.spawn([self.rc] +
                                ["/fo" + obj] + [rc_file])
 
-                except PackagingExecError:
-                    raise CompileError(sys.exc_info()[1])
+                except PackagingExecError, msg:
+                    raise CompileError(msg)
                 continue
             else:
                 # how to handle this file?
@@ -517,8 +517,8 @@
                 self.spawn([self.cc] + compile_opts + pp_opts +
                            [input_opt, output_opt] +
                            extra_postargs)
-            except PackagingExecError:
-                raise CompileError(sys.exc_info()[1])
+            except PackagingExecError, msg:
+                raise CompileError(msg)
 
         return objects
 
@@ -542,8 +542,8 @@
                 pass # XXX what goes here?
             try:
                 self.spawn([self.lib] + lib_args)
-            except PackagingExecError:
-                raise LibError(sys.exc_info()[1])
+            except PackagingExecError, msg:
+                raise LibError(msg)
         else:
             logger.debug("skipping %s (up-to-date)", output_filename)
 
@@ -620,8 +620,8 @@
             self.mkpath(os.path.dirname(output_filename))
             try:
                 self.spawn([self.linker] + ld_args)
-            except PackagingExecError:
-                raise LinkError(sys.exc_info()[1])
+            except PackagingExecError, msg:
+                raise LinkError(msg)
 
             # embed the manifest
             # XXX - this is somewhat fragile - if mt.exe fails, distutils
@@ -637,8 +637,8 @@
             try:
                 self.spawn(['mt.exe', '-nologo', '-manifest',
                             temp_manifest, out_arg])
-            except PackagingExecError:
-                raise LinkError(sys.exc_info()[1])
+            except PackagingExecError, msg:
+                raise LinkError(msg)
         else:
             logger.debug("skipping %s (up-to-date)", output_filename)
 
diff --git a/distutils2/compiler/msvccompiler.py b/distutils2/compiler/msvccompiler.py
--- a/distutils2/compiler/msvccompiler.py
+++ b/distutils2/compiler/msvccompiler.py
@@ -12,7 +12,7 @@
 import os
 
 from distutils2.errors import (PackagingExecError, PackagingPlatformError,
-                              CompileError, LibError, LinkError)
+                               CompileError, LibError, LinkError)
 from distutils2.compiler.ccompiler import CCompiler
 from distutils2.compiler import gen_lib_options
 from distutils2 import logger
@@ -386,8 +386,8 @@
                 try:
                     self.spawn([self.rc] + pp_opts +
                                [output_opt] + [input_opt])
-                except PackagingExecError:
-                    raise CompileError(sys.exc_info()[1])
+                except PackagingExecError, msg:
+                    raise CompileError(msg)
                 continue
             elif ext in self._mc_extensions:
 
@@ -415,8 +415,8 @@
                     self.spawn([self.rc] +
                                 ["/fo" + obj] + [rc_file])
 
-                except PackagingExecError:
-                    raise CompileError(sys.exc_info()[1])
+                except PackagingExecError, msg:
+                    raise CompileError(msg)
                 continue
             else:
                 # how to handle this file?
@@ -429,8 +429,8 @@
                 self.spawn([self.cc] + compile_opts + pp_opts +
                            [input_opt, output_opt] +
                            extra_postargs)
-            except PackagingExecError:
-                raise CompileError(sys.exc_info()[1])
+            except PackagingExecError, msg:
+                raise CompileError(msg)
 
         return objects
 
@@ -448,8 +448,8 @@
                 pass                    # XXX what goes here?
             try:
                 self.spawn([self.lib] + lib_args)
-            except PackagingExecError:
-                raise LibError(sys.exc_info()[1])
+            except PackagingExecError, msg:
+                raise LibError(msg)
 
         else:
             logger.debug("skipping %s (up-to-date)", output_filename)
@@ -515,8 +515,8 @@
             self.mkpath(os.path.dirname(output_filename))
             try:
                 self.spawn([self.linker] + ld_args)
-            except PackagingExecError:
-                raise LinkError(sys.exc_info()[1])
+            except PackagingExecError, msg:
+                raise LinkError(msg)
 
         else:
             logger.debug("skipping %s (up-to-date)", output_filename)
diff --git a/distutils2/compiler/unixccompiler.py b/distutils2/compiler/unixccompiler.py
--- a/distutils2/compiler/unixccompiler.py
+++ b/distutils2/compiler/unixccompiler.py
@@ -127,7 +127,7 @@
         executables['ranlib'] = ["ranlib"]
 
     # Needed for the filename generation methods provided by the base
-    # class, CCompiler.  NB. whoever instantiates/uses a particular
+    # class, CCompiler.  XXX whoever instantiates/uses a particular
     # UnixCCompiler instance should set 'shared_lib_ext' -- we set a
     # reasonable common default here, but it's not necessarily used on all
     # Unices!
@@ -165,8 +165,8 @@
                 self.mkpath(os.path.dirname(output_file))
             try:
                 self.spawn(pp_args)
-            except PackagingExecError:
-                raise CompileError(sys.exc_info()[1])
+            except PackagingExecError, msg:
+                raise CompileError(msg)
 
     def _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts):
         compiler_so = self.compiler_so
@@ -175,8 +175,8 @@
         try:
             self.spawn(compiler_so + cc_args + [src, '-o', obj] +
                        extra_postargs)
-        except PackagingExecError:
-            raise CompileError(sys.exc_info()[1])
+        except PackagingExecError, msg:
+            raise CompileError(msg)
 
     def create_static_lib(self, objects, output_libname,
                           output_dir=None, debug=False, target_lang=None):
@@ -199,8 +199,8 @@
             if self.ranlib:
                 try:
                     self.spawn(self.ranlib + [output_filename])
-                except PackagingExecError:
-                    raise LibError(sys.exc_info()[1])
+                except PackagingExecError, msg:
+                    raise LibError(msg)
         else:
             logger.debug("skipping %s (up-to-date)", output_filename)
 
@@ -253,8 +253,8 @@
                     linker = _darwin_compiler_fixup(linker, ld_args)
 
                 self.spawn(linker + ld_args)
-            except PackagingExecError:
-                raise LinkError(sys.exc_info()[1])
+            except PackagingExecError, msg:
+                raise LinkError(msg)
         else:
             logger.debug("skipping %s (up-to-date)", output_filename)
 
diff --git a/distutils2/config.py b/distutils2/config.py
--- a/distutils2/config.py
+++ b/distutils2/config.py
@@ -1,8 +1,8 @@
 """Utilities to find and read config files used by distutils2."""
 
-import codecs
 import os
 import sys
+import codecs
 import logging
 
 from shlex import split
@@ -11,7 +11,7 @@
 from distutils2.errors import PackagingOptionError
 from distutils2.compiler.extension import Extension
 from distutils2.util import (check_environ, iglob, resolve_name, strtobool,
-                            split_multiline)
+                             split_multiline)
 from distutils2.compiler import set_compiler
 from distutils2.command import set_command
 from distutils2.markers import interpret
@@ -142,9 +142,9 @@
                     for line in setup_hooks:
                         try:
                             hook = resolve_name(line)
-                        except ImportError:
+                        except ImportError, e:
                             logger.warning('cannot find setup hook: %s',
-                                           sys.exc_info()[1].args[0])
+                                           e.args[0])
                         else:
                             self.setup_hooks.append(hook)
                     self.run_hooks(content)
@@ -178,8 +178,10 @@
                     for filename in filenames:
                         # will raise if file not found
                         description_file = open(filename)
-                        value.append(description_file.read().strip())
-                        description_file.close()
+                        try:
+                            value.append(description_file.read().strip())
+                        finally:
+                            description_file.close()
                         # add filename as a required file
                         if filename not in metadata.requires_files:
                             metadata.requires_files.append(filename)
@@ -290,8 +292,10 @@
         for filename in filenames:
             logger.debug("  reading %s", filename)
             f = codecs.open(filename, 'r', encoding='utf-8')
-            parser.readfp(f)
-            f.close()
+            try:
+                parser.readfp(f)
+            finally:
+                f.close()
 
             if os.path.split(filename)[-1] == 'setup.cfg':
                 self._read_setup_cfg(parser, filename)
@@ -348,8 +352,8 @@
                         setattr(self.dist, opt, strtobool(val))
                     else:
                         setattr(self.dist, opt, val)
-                except ValueError:
-                    raise PackagingOptionError(sys.exc_info()[1])
+                except ValueError, msg:
+                    raise PackagingOptionError(msg)
 
     def _load_compilers(self, compilers):
         compilers = split_multiline(compilers)
diff --git a/distutils2/create.py b/distutils2/create.py
--- a/distutils2/create.py
+++ b/distutils2/create.py
@@ -18,30 +18,33 @@
 #  Ask for a description
 #  Detect scripts (not sure how.  #! outside of package?)
 
-import codecs
 import os
 import re
 import imp
 import sys
 import glob
+import codecs
 import shutil
-from distutils2._backport import sysconfig
-if 'any' not in dir(__builtins__):
-    from distutils2._backport import any
-try:
-    from hashlib import md5
-except ImportError: #<2.5
-    from md5 import md5
 from textwrap import dedent
+from ConfigParser import RawConfigParser
 from distutils2.util import cmp_to_key, detect_encoding
-from ConfigParser import RawConfigParser
 # importing this with an underscore as it should be replaced by the
 # dict form or another structures for all purposes
 from distutils2._trove import all_classifiers as _CLASSIFIERS_LIST
 from distutils2.version import is_valid_version
+from distutils2._backport import sysconfig
+try:
+    any
+except NameError:
+    from distutils2._backport import any
+try:
+    from hashlib import md5
+except ImportError:
+    from distutils2._backport.hashlib import md5
+
 
 _FILENAME = 'setup.cfg'
-_DEFAULT_CFG = '.pypkgcreate'
+_DEFAULT_CFG = '.pypkgcreate'  # FIXME use a section in user .pydistutils.cfg
 
 _helptext = {
     'name': '''
@@ -117,11 +120,16 @@
     been loaded before, because we are monkey patching its setup function with
     a particular one"""
     f = open("setup.py", "rb")
-    encoding, lines = detect_encoding(f.readline)
-    f.close()
+    try:
+        encoding, lines = detect_encoding(f.readline)
+    finally:
+        f.close()
     f = open("setup.py")
-    imp.load_module("setup", f, "setup.py", (".py", "r", imp.PY_SOURCE))
-    f.close()
+    try:
+        imp.load_module("setup", f, "setup.py", (".py", "r", imp.PY_SOURCE))
+    finally:
+        f.close()
+
 
 def ask_yn(question, default=None, helptext=None):
     question += ' (y/n)'
@@ -133,6 +141,10 @@
         print '\nERROR: You must select "Y" or "N".\n'
 
 
+# XXX use util.ask
+# FIXME: if prompt ends with '?', don't add ':'
+
+
 def ask(question, default=None, helptext=None, required=True,
         lengthy=False, multiline=False):
     prompt = u'%s: ' % (question,)
@@ -280,50 +292,52 @@
             shutil.move(_FILENAME, '%s.old' % _FILENAME)
 
         fp = codecs.open(_FILENAME, 'w', encoding='utf-8')
-        fp.write(u'[metadata]\n')
-        # TODO use metadata module instead of hard-coding field-specific
-        # behavior here
+        try:
+            fp.write(u'[metadata]\n')
+            # TODO use metadata module instead of hard-coding field-specific
+            # behavior here
 
-        # simple string entries
-        for name in ('name', 'version', 'summary', 'download_url'):
-            fp.write(u'%s = %s\n' % (name, self.data.get(name, 'UNKNOWN')))
+            # simple string entries
+            for name in ('name', 'version', 'summary', 'download_url'):
+                fp.write(u'%s = %s\n' % (name, self.data.get(name, 'UNKNOWN')))
 
-        # optional string entries
-        if 'keywords' in self.data and self.data['keywords']:
-            fp.write(u'keywords = %s\n' % ' '.join(self.data['keywords']))
-        for name in ('home_page', 'author', 'author_email',
-                     'maintainer', 'maintainer_email', 'description-file'):
-            if name in self.data and self.data[name]:
-                fp.write(u'%s = %s\n' % (name.decode('utf-8'),
-                    self.data[name].decode('utf-8')))
-        if 'description' in self.data:
-            fp.write(
-                u'description = %s\n'
-                % u'\n       |'.join(self.data['description'].split('\n')))
+            # optional string entries
+            if 'keywords' in self.data and self.data['keywords']:
+                fp.write(u'keywords = %s\n' % ' '.join(self.data['keywords']))
+            for name in ('home_page', 'author', 'author_email',
+                         'maintainer', 'maintainer_email', 'description-file'):
+                if name in self.data and self.data[name]:
+                    fp.write(u'%s = %s\n' % (name.decode('utf-8'),
+                                             self.data[name].decode('utf-8')))
+            if 'description' in self.data:
+                fp.write(
+                    u'description = %s\n'
+                    % u'\n       |'.join(self.data['description'].split('\n')))
 
-        # multiple use string entries
-        for name in ('platform', 'supported-platform', 'classifier',
-                     'requires-dist', 'provides-dist', 'obsoletes-dist',
-                     'requires-external'):
-            if not(name in self.data and self.data[name]):
-                continue
-            fp.write(u'%s = ' % name)
-            fp.write(u''.join('    %s\n' % val
-                             for val in self.data[name]).lstrip())
-        fp.write(u'\n[files]\n')
-        for name in ('packages', 'modules', 'scripts',
-                     'package_data', 'extra_files'):
-            if not(name in self.data and self.data[name]):
-                continue
-            fp.write(u'%s = %s\n'
-                     % (name, u'\n    '.join(self.data[name]).strip()))
-        fp.write(u'\nresources =\n')
-        for src, dest in self.data['resources']:
-            fp.write(u'    %s = %s\n' % (src, dest))
-        fp.write(u'\n')
-        fp.close()
+            # multiple use string entries
+            for name in ('platform', 'supported-platform', 'classifier',
+                         'requires-dist', 'provides-dist', 'obsoletes-dist',
+                         'requires-external'):
+                if not(name in self.data and self.data[name]):
+                    continue
+                fp.write(u'%s = ' % name)
+                fp.write(u''.join('    %s\n' % val
+                                 for val in self.data[name]).lstrip())
+            fp.write(u'\n[files]\n')
+            for name in ('packages', 'modules', 'scripts',
+                         'package_data', 'extra_files'):
+                if not(name in self.data and self.data[name]):
+                    continue
+                fp.write(u'%s = %s\n'
+                         % (name, u'\n    '.join(self.data[name]).strip()))
+            fp.write(u'\nresources =\n')
+            for src, dest in self.data['resources']:
+                fp.write(u'    %s = %s\n' % (src, dest))
+            fp.write(u'\n')
+        finally:
+            fp.close()
 
-        os.chmod(_FILENAME, 00644)
+        os.chmod(_FILENAME, 0644)
         print 'Wrote %r.' % _FILENAME
 
     def convert_py_to_cfg(self):
@@ -418,8 +432,10 @@
                 ref = ref.digest()
                 for readme in glob.glob('README*'):
                     fp = codecs.open(readme, encoding='utf-8')
-                    contents = fp.read()
-                    fp.close()
+                    try:
+                        contents = fp.read()
+                    finally:
+                        fp.close()
                     contents = re.sub('\s', '', contents.lower()).encode()
                     val = md5(contents).digest()
                     if val == ref:
diff --git a/distutils2/database.py b/distutils2/database.py
--- a/distutils2/database.py
+++ b/distutils2/database.py
@@ -1,15 +1,16 @@
 """PEP 376 implementation."""
 
-from StringIO import StringIO
 import os
 import re
 import csv
 import sys
 import zipimport
+from StringIO import StringIO
 try:
     from hashlib import md5
-except ImportError: #<2.5
-    from md5 import md5
+except ImportError:
+    from distutils2._backport.hashlib import md5
+
 from distutils2 import logger
 from distutils2.errors import PackagingError
 from distutils2.version import suggest_normalized_version, VersionPredicate
@@ -162,26 +163,30 @@
     def _get_records(self, local=False):
         results = []
         record = self.get_distinfo_file('RECORD')
-        record_reader = csv.reader(record, delimiter=',',
-                                   lineterminator='\n')
-        for row in record_reader:
-            missing = [None for i in range(len(row), 3)]
-            path, checksum, size = row + missing
-            if local:
-                path = path.replace('/', os.sep)
-                path = os.path.join(sys.prefix, path)
-            results.append((path, checksum, size))
-        record.close()
+        try:
+            record_reader = csv.reader(record, delimiter=',',
+                                       lineterminator='\n')
+            for row in record_reader:
+                missing = [None for i in range(len(row), 3)]
+                path, checksum, size = row + missing
+                if local:
+                    path = path.replace('/', os.sep)
+                    path = os.path.join(sys.prefix, path)
+                results.append((path, checksum, size))
+        finally:
+            record.close()
         return results
 
     def get_resource_path(self, relative_path):
         resources_file = self.get_distinfo_file('RESOURCES')
-        resources_reader = csv.reader(resources_file, delimiter=',',
-                                       lineterminator='\n')
-        for relative, destination in resources_reader:
-            if relative == relative_path:
-                return destination
-        resources_file.close()
+        try:
+            resources_reader = csv.reader(resources_file, delimiter=',',
+                                          lineterminator='\n')
+            for relative, destination in resources_reader:
+                if relative == relative_path:
+                    return destination
+        finally:
+            resources_file.close()
         raise KeyError(
             'no resource file with relative path %r is installed' %
             relative_path)
@@ -331,8 +336,10 @@
                 try:
                     req_path = os.path.join(path, 'EGG-INFO', 'requires.txt')
                     fp = open(req_path, 'r')
-                    requires = fp.read()
-                    fp.close()
+                    try:
+                        requires = fp.read()
+                    finally:
+                        fp.close()
                 except IOError:
                     requires = None
             else:
@@ -353,8 +360,10 @@
                 path = os.path.join(path, 'PKG-INFO')
                 try:
                     fp = open(os.path.join(path, 'requires.txt'), 'r')
-                    requires = fp.read()
-                    fp.close()
+                    try:
+                        requires = fp.read()
+                    finally:
+                        fp.close()
                 except IOError:
                     requires = None
             self.metadata = Metadata(path=path)
@@ -417,8 +426,10 @@
 
         def _md5(path):
             f = open(path, 'rb')
-            content = f.read()
-            f.close()
+            try:
+                content = f.read()
+            finally:
+                f.close()
             return md5(content).hexdigest()
 
         def _size(path):
diff --git a/distutils2/depgraph.py b/distutils2/depgraph.py
--- a/distutils2/depgraph.py
+++ b/distutils2/depgraph.py
@@ -234,8 +234,7 @@
             graph = generate_graph(dists)
         finally:
             sys.stderr = old
-    except Exception:
-        e = sys.exc_info()[1]
+    except Exception, e:
         tempout.seek(0)
         tempout = tempout.read()
         print 'Could not generate the graph'
@@ -259,8 +258,10 @@
             filename = 'depgraph.dot'
 
         f = open(filename, 'w')
-        graph_to_dot(graph, f, True)
-        f.close()
+        try:
+            graph_to_dot(graph, f, True)
+        finally:
+            f.close()
         tempout.seek(0)
         tempout = tempout.read()
         print tempout
diff --git a/distutils2/dist.py b/distutils2/dist.py
--- a/distutils2/dist.py
+++ b/distutils2/dist.py
@@ -1,21 +1,21 @@
-"""Class representing the distribution being built/installed/etc."""
+"""Class representing the project being built/installed/etc."""
 
 import os
 import re
-import sys
 
+from distutils2 import logger
+from distutils2.util import strtobool, resolve_name
 from distutils2.errors import (PackagingOptionError, PackagingArgError,
-                              PackagingModuleError, PackagingClassError)
-from distutils2.fancy_getopt import FancyGetopt
-from distutils2.util import strtobool, resolve_name
-from distutils2 import logger
-from distutils2.metadata import Metadata
+                               PackagingModuleError, PackagingClassError)
 from distutils2.config import Config
 from distutils2.command import get_command_class, STANDARD_COMMANDS
+from distutils2.command.cmd import Command
+from distutils2.metadata import Metadata
+from distutils2.fancy_getopt import FancyGetopt
 
 # Regex to define acceptable Packaging command names.  This is not *quite*
-# the same as a Python NAME -- I don't allow leading underscores.  The fact
-# that they're very similar is no coincidence; the default naming scheme is
+# the same as a Python name -- leading underscores are not allowed.  The fact
+# that they're very similar is no coincidence: the default naming scheme is
 # to look for a Python module named after the command.
 command_re = re.compile(r'^[a-zA-Z]([a-zA-Z0-9_]*)$')
 
@@ -33,17 +33,11 @@
 
 
 class Distribution(object):
-    """The core of the Packaging.  Most of the work hiding behind 'setup'
-    is really done within a Distribution instance, which farms the work out
-    to the Packaging commands specified on the command line.
+    """Class used to represent a project and work with it.
 
-    Setup scripts will almost never instantiate Distribution directly,
-    unless the 'setup()' function is totally inadequate to their needs.
-    However, it is conceivable that a setup script might wish to subclass
-    Distribution for some specialized purpose, and then pass the subclass
-    to 'setup()' as the 'distclass' keyword argument.  If so, it is
-    necessary to respect the expectations that 'setup' has of Distribution.
-    See the code for 'setup()', in run.py, for details.
+    Most of the work hiding behind 'pysetup run' is really done within a
+    Distribution instance, which farms the work out to the commands
+    specified on the command line.
     """
 
     # 'global_options' describes the command-line options that may be
@@ -64,8 +58,8 @@
     common_usage = """\
 Common commands: (see '--help-commands' for more)
 
-  pysetup run build      will build the package underneath 'build/'
-  pysetup run install    will install the package
+  pysetup run build      will build the project underneath 'build/'
+  pysetup run install    will install the project
 """
 
     # options that are not propagated to the commands
@@ -373,7 +367,7 @@
                             commands=self.commands)
             return
 
-        return 1
+        return True
 
     def _get_toplevel_options(self):
         """Return the non-display options recognized at the top level.
@@ -403,8 +397,8 @@
         # it takes.
         try:
             cmd_class = get_command_class(command)
-        except PackagingModuleError:
-            raise PackagingArgError(sys.exc_info()[1])
+        except PackagingModuleError, msg:
+            raise PackagingArgError(msg)
 
         # XXX We want to push this in distutils2.command
         #
@@ -501,9 +495,6 @@
         lists per-command help for every command name or command class
         in 'commands'.
         """
-        # late import because of mutual dependence between these modules
-        from distutils2.command.cmd import Command
-
         if global_options:
             if display_options:
                 options = self._get_toplevel_options()
@@ -629,7 +620,7 @@
         """
         cmd_obj = self.command_obj.get(command)
         if not cmd_obj and create:
-            logger.debug("Distribution.get_command_obj(): " \
+            logger.debug("Distribution.get_command_obj(): "
                          "creating %r command object", command)
 
             cls = get_command_class(command)
@@ -685,8 +676,8 @@
                     raise PackagingOptionError(
                         "error in %s: command %r has no such option %r" %
                         (source, command_name, option))
-            except ValueError:
-                raise PackagingOptionError(sys.exc_info()[1])
+            except ValueError, msg:
+                raise PackagingOptionError(msg)
 
     def get_reinitialized_command(self, command, reinit_subcommands=False):
         """Reinitializes a command to the state it was in when first
@@ -707,7 +698,6 @@
 
         Returns the reinitialized command object.
         """
-        from distutils2.command.cmd import Command
         if not isinstance(command, Command):
             command_name = command
             command = self.get_command_obj(command_name)
@@ -716,6 +706,7 @@
 
         if not command.finalized:
             return command
+
         command.initialize_options()
         self.have_run[command_name] = 0
         command.finalized = False
@@ -780,8 +771,8 @@
             if isinstance(hook, basestring):
                 try:
                     hook_obj = resolve_name(hook)
-                except ImportError:
-                    raise PackagingModuleError(sys.exc_info()[1])
+                except ImportError, e:
+                    raise PackagingModuleError(e)
             else:
                 hook_obj = hook
 
diff --git a/distutils2/fancy_getopt.py b/distutils2/fancy_getopt.py
--- a/distutils2/fancy_getopt.py
+++ b/distutils2/fancy_getopt.py
@@ -237,8 +237,8 @@
 
         try:
             opts, args = getopt.getopt(args, short_opts, self.long_opts)
-        except getopt.error:
-            raise PackagingArgError(sys.exc_info()[1])
+        except getopt.error, msg:
+            raise PackagingArgError(msg)
 
         for opt, val in opts:
             if len(opt) == 2 and opt[0] == '-':   # it's a short option
@@ -368,7 +368,7 @@
         if file is None:
             file = sys.stdout
         for line in self.generate_help(header):
-            file.write(line + u"\n")
+            file.write(line + "\n")
 
 
 def fancy_getopt(options, negative_opt, object, args):
diff --git a/distutils2/install.py b/distutils2/install.py
--- a/distutils2/install.py
+++ b/distutils2/install.py
@@ -13,21 +13,22 @@
 import shutil
 import logging
 import tempfile
-from sysconfig import get_config_var, get_path, is_python_build
 
 from distutils2 import logger
 from distutils2.dist import Distribution
 from distutils2.util import (_is_archive_file, ask, get_install_method,
-                            egginfo_to_distinfo, unpack_archive)
+                             egginfo_to_distinfo, unpack_archive)
 from distutils2.pypi import wrapper
 from distutils2.version import get_version_predicate
 from distutils2.database import get_distributions, get_distribution
 from distutils2.depgraph import generate_graph
 
 from distutils2.errors import (PackagingError, InstallationException,
-                              InstallationConflict, CCompilerError)
+                               InstallationConflict, CCompilerError)
 from distutils2.pypi.errors import ProjectNotFound, ReleaseNotFound
 from distutils2 import database
+from distutils2._backport.sysconfig import (get_config_var, get_path,
+                                            is_python_build)
 
 
 __all__ = ['install_dists', 'install_from_infos', 'get_infos', 'remove',
@@ -50,8 +51,7 @@
         # try to make the paths.
         try:
             os.makedirs(os.path.dirname(new))
-        except OSError:
-            e = sys.exc_info()[1]
+        except OSError, e:
             if e.errno != errno.EEXIST:
                 raise
         os.rename(old, new)
@@ -88,8 +88,8 @@
         dist.run_command('install_dist')
         name = dist.metadata['Name']
         return database.get_distribution(name) is not None
-    except (IOError, os.error, PackagingError, CCompilerError):
-        raise ValueError("Failed to install, " + str(sys.exc_info()[1]))
+    except (IOError, os.error, PackagingError, CCompilerError), msg:
+        raise ValueError("Failed to install, " + str(msg))
 
 
 def _install_dist(dist, path):
@@ -160,9 +160,9 @@
         try:
             func(source_dir)
             return True
-        except ValueError:
+        except ValueError, err:
             # failed to install
-            logger.info(str(sys.exc_info()[1]))
+            logger.info(str(err))
             return False
     finally:
         os.chdir(old_dir)
@@ -187,8 +187,8 @@
         try:
             _install_dist(dist, path)
             installed_dists.append(dist)
-        except Exception:
-            logger.info('Failed: %s', sys.exc_info()[1])
+        except Exception, e:
+            logger.info('Failed: %s', e)
 
             # reverting
             for installed_dist in installed_dists:
@@ -396,8 +396,8 @@
     def _move_file(source, target):
         try:
             os.rename(source, target)
-        except OSError:
-            return sys.exc_info()[1]
+        except OSError, err:
+            return err
         return None
 
     success = True
@@ -496,9 +496,11 @@
     # trying to write a file there
     try:
         testfile = tempfile.NamedTemporaryFile(suffix=project,
-                dir=purelib_path)
-        testfile.write('test')
-        testfile.close()
+                                               dir=purelib_path)
+        try:
+            testfile.write('test')
+        finally:
+            testfile.close()
     except OSError:
         # FIXME this should check the errno, or be removed altogether (race
         # condition: the directory permissions could be changed between here
@@ -523,8 +525,7 @@
         install_from_infos(install_path,
                            info['install'], info['remove'], info['conflict'])
 
-    except InstallationConflict:
-        e = sys.exc_info()[1]
+    except InstallationConflict, e:
         if logger.isEnabledFor(logging.INFO):
             projects = ('%r %s' % (p.name, p.version) for p in e.args[0])
             logger.info('%r conflicts with %s', project, ','.join(projects))
diff --git a/distutils2/manifest.py b/distutils2/manifest.py
--- a/distutils2/manifest.py
+++ b/distutils2/manifest.py
@@ -8,13 +8,12 @@
 # XXX todo: document + add tests
 import re
 import os
-import sys
 import fnmatch
 
 from distutils2 import logger
 from distutils2.util import write_file, convert_path
 from distutils2.errors import (PackagingTemplateError,
-                              PackagingInternalError)
+                               PackagingInternalError)
 
 __all__ = ['Manifest']
 
@@ -90,8 +89,8 @@
                 continue
             try:
                 self._process_template_line(line)
-            except PackagingTemplateError:
-                logger.warning("%s, %s", path_or_file, sys.exc_info()[1])
+            except PackagingTemplateError, msg:
+                logger.warning("%s, %s", path_or_file, msg)
 
     def write(self, path):
         """Write the file list in 'self.filelist' (presumably as filled in
@@ -100,8 +99,10 @@
         """
         if os.path.isfile(path):
             fp = open(path)
-            first_line = fp.readline()
-            fp.close()
+            try:
+                first_line = fp.readline()
+            finally:
+                fp.close()
 
             if first_line != '# file GENERATED by distutils2, do NOT edit\n':
                 logger.info("not writing to manually maintained "
@@ -122,9 +123,11 @@
         """
         logger.info("reading manifest file %r", path)
         manifest = open(path)
-        for line in manifest.readlines():
-            self.append(line)
-        manifest.close()
+        try:
+            for line in manifest.readlines():
+                self.append(line)
+        finally:
+            manifest.close()
 
     def exclude_pattern(self, pattern, anchor=True, prefix=None,
                         is_regex=False):
diff --git a/distutils2/markers.py b/distutils2/markers.py
--- a/distutils2/markers.py
+++ b/distutils2/markers.py
@@ -6,6 +6,11 @@
 
 from tokenize import generate_tokens, NAME, OP, STRING, ENDMARKER
 from StringIO import StringIO as BytesIO
+try:
+    python_implementation = platform.python_implementation()
+except AttributeError:
+    # FIXME import from compat
+    python_implementation = 'CPython'
 
 __all__ = ['interpret']
 
@@ -24,10 +29,7 @@
 def _operate(operation, x, y):
     return _OPERATORS[operation](x, y)
 
-try:
-    python_implementation = platform.python_implementation()
-except AttributeError: #<2.6 - assume CPython?
-    python_implementation = 'CPython'
+
 # restricted set of variables
 _VARS = {'sys.platform': sys.platform,
          'python_version': sys.version[:3],
diff --git a/distutils2/metadata.py b/distutils2/metadata.py
--- a/distutils2/metadata.py
+++ b/distutils2/metadata.py
@@ -3,8 +3,8 @@
 Supports all metadata formats (1.0, 1.1, 1.2).
 """
 
+import re
 import codecs
-import re
 import logging
 
 from StringIO import StringIO
@@ -12,10 +12,10 @@
 from distutils2 import logger
 from distutils2.markers import interpret
 from distutils2.version import (is_valid_predicate, is_valid_version,
-                               is_valid_versions)
+                                is_valid_versions)
 from distutils2.errors import (MetadataMissingError,
-                              MetadataConflictError,
-                              MetadataUnrecognizedVersionError)
+                               MetadataConflictError,
+                               MetadataUnrecognizedVersionError)
 
 try:
     # docutils is installed
@@ -311,8 +311,10 @@
     def read(self, filepath):
         """Read the metadata values from a file path."""
         fp = codecs.open(filepath, 'r', encoding='utf-8')
-        self.read_file(fp)
-        fp.close()
+        try:
+            self.read_file(fp)
+        finally:
+            fp.close()
 
     def read_file(self, fileob):
         """Read the metadata values from a file object."""
@@ -335,8 +337,10 @@
     def write(self, filepath):
         """Write the metadata fields to filepath."""
         fp = codecs.open(filepath, 'w', encoding='utf-8')
-        self.write_file(fp)
-        fp.close()
+        try:
+            self.write_file(fp)
+        finally:
+            fp.close()
 
     def write_file(self, fileobject):
         """Write the PKG-INFO format data to a file object."""
diff --git a/distutils2/pypi/dist.py b/distutils2/pypi/dist.py
--- a/distutils2/pypi/dist.py
+++ b/distutils2/pypi/dist.py
@@ -10,7 +10,7 @@
 import re
 try:
     import hashlib
-except ImportError: #<2.5
+except ImportError:
     from distutils2._backport import hashlib
 import tempfile
 import urllib
@@ -328,9 +328,11 @@
         expected_hashval = self.url['hashval']
         if None not in (expected_hashval, hashname):
             f = open(filename, 'rb')
-            hashval = hashlib.new(hashname)
-            hashval.update(f.read())
-            f.close()
+            try:
+                hashval = hashlib.new(hashname)
+                hashval.update(f.read())
+            finally:
+                f.close()
 
             if hashval.hexdigest() != expected_hashval:
                 raise HashDoesNotMatch("got %s instead of %s"
diff --git a/distutils2/pypi/simple.py b/distutils2/pypi/simple.py
--- a/distutils2/pypi/simple.py
+++ b/distutils2/pypi/simple.py
@@ -161,16 +161,18 @@
         Return a list of names.
         """
         index = self._open_url(self.index_url)
-        if '*' in name:
-            name.replace('*', '.*')
-        else:
-            name = "%s%s%s" % ('*.?', name, '*.?')
-        name = name.replace('*', '[^<]*')  # avoid matching end tag
-        projectname = re.compile('<a[^>]*>(%s)</a>' % name, re.I)
-        matching_projects = []
+        try:
+            if '*' in name:
+                name.replace('*', '.*')
+            else:
+                name = "%s%s%s" % ('*.?', name, '*.?')
+            name = name.replace('*', '[^<]*')  # avoid matching end tag
+            projectname = re.compile('<a[^>]*>(%s)</a>' % name, re.I)
+            matching_projects = []
 
-        index_content = index.read()
-        index.close()
+            index_content = index.read()
+        finally:
+            index.close()
 
         # FIXME should use bytes I/O and regexes instead of decoding
         index_content = index_content.decode()
@@ -236,7 +238,9 @@
         self._mirrors_used.add(self.index_url)
         index_url = self._mirrors.pop()
         # XXX use urlparse for a real check of missing scheme part
-        if not index_url.startswith(("http://", "https://", "file://")):
+        if not (index_url.startswith("http://") or
+                index_url.startswith("https://") or
+                index_url.startswith("file://")):
             index_url = "http://%s" % index_url
 
         if not index_url.endswith("/simple"):
@@ -327,8 +331,7 @@
                             try:
                                 infos = get_infos_from_url(link, project_name,
                                             is_external=self.index_url not in url)
-                            except CantParseArchiveName:
-                                e = sys.exc_info()[1]
+                            except CantParseArchiveName, e:
                                 if self.verbose:
                                     logger.warning(
                                         "version has not been parsed: %s", e)
@@ -410,7 +413,7 @@
 
         # authentication stuff
         if scheme in ('http', 'https'):
-            auth, host = urlparse.splituser(netloc)
+            auth, host = urllib2.splituser(netloc)
         else:
             auth = None
 
@@ -432,21 +435,17 @@
         request.add_header('User-Agent', USER_AGENT)
         try:
             fp = urllib2.urlopen(request)
-        except (ValueError, httplib.InvalidURL):
-            v = sys.exc_info()[1]
+        except (ValueError, httplib.InvalidURL), v:
             msg = ' '.join([str(arg) for arg in v.args])
             raise PackagingPyPIError('%s %s' % (url, msg))
-        except urllib2.HTTPError:
-            return sys.exc_info()[1]
-        except urllib2.URLError:
-            v = sys.exc_info()[1]
+        except urllib2.HTTPError, v:
+            return v
+        except urllib2.URLError, v:
             raise DownloadError("Download error for %s: %s" % (url, v.reason))
-        except httplib.BadStatusLine:
-            v = sys.exc_info()[1]
+        except httplib.BadStatusLine, v:
             raise DownloadError('%s returned a bad status line. '
                 'The server might be down, %s' % (url, v.line))
-        except httplib.HTTPException:
-            v = sys.exc_info()[1]
+        except httplib.HTTPException, v:
             raise DownloadError("Download error for %s: %s" % (url, v))
         except socket.timeout:
             raise DownloadError("The server timeouted")
diff --git a/distutils2/pypi/wrapper.py b/distutils2/pypi/wrapper.py
--- a/distutils2/pypi/wrapper.py
+++ b/distutils2/pypi/wrapper.py
@@ -31,8 +31,8 @@
             try:
                 response = method(*args, **kwargs)
                 retry = False
-            except Exception:
-                exception = sys.exc_info()[1]
+            except Exception, e:
+                exception = e
             if not retry:
                 break
         if retry and exception:
diff --git a/distutils2/pypi/xmlrpc.py b/distutils2/pypi/xmlrpc.py
--- a/distutils2/pypi/xmlrpc.py
+++ b/distutils2/pypi/xmlrpc.py
@@ -13,7 +13,7 @@
 from distutils2.version import get_version_predicate
 from distutils2.pypi.base import BaseClient
 from distutils2.pypi.errors import (ProjectNotFound, InvalidSearchField,
-                                   ReleaseNotFound)
+                                    ReleaseNotFound)
 from distutils2.pypi.dist import ReleaseInfo
 
 __all__ = ['Client', 'DEFAULT_XMLRPC_INDEX_URL']
@@ -171,8 +171,7 @@
                 project.add_release(release=ReleaseInfo(p['name'],
                     p['version'], metadata={'summary': p['summary']},
                     index=self._index))
-            except IrrationalVersionError:
-                e = sys.exc_info()[1]
+            except IrrationalVersionError, e:
                 logger.warning("Irrational version error found: %s", e)
         return [self._projects[p['name'].lower()] for p in projects]
 
diff --git a/distutils2/run.py b/distutils2/run.py
--- a/distutils2/run.py
+++ b/distutils2/run.py
@@ -15,8 +15,8 @@
 from distutils2.depgraph import generate_graph
 from distutils2.fancy_getopt import FancyGetopt
 from distutils2.errors import (PackagingArgError, PackagingError,
-                              PackagingModuleError, PackagingClassError,
-                              CCompilerError)
+                               PackagingModuleError, PackagingClassError,
+                               CCompilerError)
 
 
 command_re = re.compile(r'^[a-zA-Z]([a-zA-Z0-9_]*)$')
@@ -473,8 +473,8 @@
         # it takes.
         try:
             cmd_class = get_command_class(command)
-        except PackagingModuleError:
-            raise PackagingArgError(sys.exc_info()[1])
+        except PackagingModuleError, msg:
+            raise PackagingArgError(msg)
 
         # XXX We want to push this in distutils2.command
         #
@@ -674,22 +674,21 @@
     old_level = logger.level
     old_handlers = list(logger.handlers)
     try:
-        dispatcher = Dispatcher(args)
-        if dispatcher.action is None:
-            return
-        return dispatcher()
-    except KeyboardInterrupt:
-        logger.info('interrupted')
-        return 1
-    except (IOError, os.error, PackagingError, CCompilerError):
-        logger.exception(sys.exc_info()[1])
-        return 1
-    except:
+        try:
+            dispatcher = Dispatcher(args)
+            if dispatcher.action is None:
+                return
+            return dispatcher()
+        except KeyboardInterrupt:
+            logger.info('interrupted')
+            return 1
+        except (IOError, os.error, PackagingError, CCompilerError), exc:
+            logger.exception(exc)
+            return 1
+    finally:
         logger.setLevel(old_level)
         logger.handlers[:] = old_handlers
-        raise
-    logger.setLevel(old_level)
-    logger.handlers[:] = old_handlers
+
 
 if __name__ == '__main__':
     sys.exit(main())
diff --git a/distutils2/tests/__main__.py b/distutils2/tests/__main__.py
--- a/distutils2/tests/__main__.py
+++ b/distutils2/tests/__main__.py
@@ -4,8 +4,8 @@
 
 import os
 import sys
-import unittest2
-from .support import run_unittest, reap_children, reap_threads
+from distutils2.tests import unittest
+from distutils2.tests.support import reap_children, reap_threads, run_unittest
 
 
 @reap_threads
@@ -13,7 +13,9 @@
     try:
         start_dir = os.path.dirname(__file__)
         top_dir = os.path.dirname(os.path.dirname(start_dir))
-        test_loader = unittest2.TestLoader()
+        test_loader = unittest.TestLoader()
+        # XXX find out how to use unittest.main, to get command-line options
+        # (failfast, catch, etc.)
         run_unittest(test_loader.discover(start_dir, top_level_dir=top_dir))
     finally:
         reap_children()
diff --git a/distutils2/tests/pypi_server.py b/distutils2/tests/pypi_server.py
--- a/distutils2/tests/pypi_server.py
+++ b/distutils2/tests/pypi_server.py
@@ -30,19 +30,21 @@
 """
 
 import os
-import queue
+import Queue
 import select
 import threading
-import socketserver
+import SocketServer
+from BaseHTTPServer import HTTPServer
+from SimpleHTTPServer import SimpleHTTPRequestHandler
+from SimpleXMLRPCServer import SimpleXMLRPCServer
+
+from distutils2.tests import unittest
+
 try:
     from functools import wraps
 except ImportError:
     from distutils2._backport.functools import wraps
 
-from http.server import HTTPServer, SimpleHTTPRequestHandler
-from xmlrpc.server import SimpleXMLRPCServer
-
-from distutils2.tests import unittest
 
 PYPI_DEFAULT_STATIC_PATH = os.path.join(
     os.path.dirname(os.path.abspath(__file__)), 'pypiserver')
@@ -117,7 +119,7 @@
             self.server = HTTPServer(('127.0.0.1', 0), PyPIRequestHandler)
             self.server.RequestHandlerClass.pypi_server = self
 
-            self.request_queue = queue.Queue()
+            self.request_queue = Queue.Queue()
             self._requests = []
             self.default_response_status = 404
             self.default_response_headers = [('Content-type', 'text/plain')]
@@ -154,7 +156,7 @@
     def stop(self):
         """self shutdown is not supported for python < 2.6"""
         self._run = False
-        if self.is_alive():
+        if self.isAlive():
             self.join()
         self.server.server_close()
 
@@ -171,7 +173,7 @@
         while True:
             try:
                 self._requests.append(self.request_queue.get_nowait())
-            except queue.Empty:
+            except Queue.Empty:
                 break
         return self._requests
 
@@ -275,7 +277,7 @@
 class PyPIXMLRPCServer(SimpleXMLRPCServer):
     def server_bind(self):
         """Override server_bind to store the server name."""
-        socketserver.TCPServer.server_bind(self)
+        SocketServer.TCPServer.server_bind(self)
         host, port = self.socket.getsockname()[:2]
         self.server_port = port
 
diff --git a/distutils2/tests/test_command_build.py b/distutils2/tests/test_command_build.py
--- a/distutils2/tests/test_command_build.py
+++ b/distutils2/tests/test_command_build.py
@@ -3,7 +3,7 @@
 import sys
 
 from distutils2.command.build import build
-from sysconfig import get_platform
+from distutils2._backport.sysconfig import get_platform
 from distutils2.tests import unittest, support
 
 
diff --git a/distutils2/tests/test_command_build_ext.py b/distutils2/tests/test_command_build_ext.py
--- a/distutils2/tests/test_command_build_ext.py
+++ b/distutils2/tests/test_command_build_ext.py
@@ -5,6 +5,7 @@
 import textwrap
 from StringIO import StringIO
 from distutils2._backport import sysconfig
+from distutils2._backport.sysconfig import _CONFIG_VARS
 from distutils2.dist import Distribution
 from distutils2.errors import (UnknownFileError, CompileError,
                               PackagingPlatformError)
@@ -36,9 +37,10 @@
         filename = _get_source_filename()
         if os.path.exists(filename):
             shutil.copy(filename, self.tmp_dir)
-        self.old_user_base = site.USER_BASE
-        site.USER_BASE = self.mkdtemp()
-        build_ext.USER_BASE = site.USER_BASE
+        if sys.version > "2.6":
+            self.old_user_base = site.USER_BASE
+            site.USER_BASE = self.mkdtemp()
+            build_ext.USER_BASE = site.USER_BASE
 
     def tearDown(self):
         # Get everything back to normal
@@ -122,7 +124,6 @@
         old = sys.platform
 
         sys.platform = 'sunos'  # fooling finalize_options
-        from sysconfig import _CONFIG_VARS
 
         old_var = _CONFIG_VARS.get('Py_ENABLE_SHARED')
         _CONFIG_VARS['Py_ENABLE_SHARED'] = 1
diff --git a/distutils2/tests/test_command_upload_docs.py b/distutils2/tests/test_command_upload_docs.py
--- a/distutils2/tests/test_command_upload_docs.py
+++ b/distutils2/tests/test_command_upload_docs.py
@@ -18,7 +18,7 @@
     from distutils2.tests.pypi_server import PyPIServerTestCase
 except ImportError:
     threading = None
-    PyPIServerTestCase = object
+    PyPIServerTestCase = unittest.TestCase
 
 
 PYPIRC = """\
@@ -32,8 +32,7 @@
 """
 
 
-class UploadDocsTestCase(unittest.TestCase,
-                         support.TempdirManager,
+class UploadDocsTestCase(support.TempdirManager,
                          support.EnvironRestorer,
                          support.LoggingCatcher,
                          PyPIServerTestCase):
diff --git a/distutils2/tests/test_install.py b/distutils2/tests/test_install.py
--- a/distutils2/tests/test_install.py
+++ b/distutils2/tests/test_install.py
@@ -1,7 +1,6 @@
 """Tests for the distutils2.install module."""
 import os
 import logging
-from sysconfig import is_python_build
 from tempfile import mkstemp
 
 from distutils2 import install
@@ -9,6 +8,8 @@
 from distutils2.metadata import Metadata
 from distutils2.tests.support import (LoggingCatcher, TempdirManager, unittest,
                                      fake_dec)
+from distutils2._backport.sysconfig import is_python_build
+
 try:
     import threading
     from distutils2.tests.pypi_server import use_xmlrpc_server
diff --git a/distutils2/tests/test_mixin2to3.py b/distutils2/tests/test_mixin2to3.py
--- a/distutils2/tests/test_mixin2to3.py
+++ b/distutils2/tests/test_mixin2to3.py
@@ -1,3 +1,4 @@
+import sys
 import textwrap
 
 from distutils2.tests import unittest, support
@@ -8,8 +9,7 @@
                         support.LoggingCatcher,
                         unittest.TestCase):
 
-    #@unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
-    @unittest.skipIf(True, 'Not needed for backport')
+    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_convert_code_only(self):
         # used to check if code gets converted properly.
         code = "print 'test'"
@@ -28,8 +28,7 @@
 
         self.assertEqual(expected, converted)
 
-    #@unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
-    @unittest.skipIf(True, 'Not needed for backport')
+    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_doctests_only(self):
         # used to check if doctests gets converted properly.
         doctest = textwrap.dedent('''\
@@ -62,8 +61,7 @@
 
         self.assertEqual(expected, converted)
 
-    #@unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
-    @unittest.skipIf(True, 'Not needed for backport')
+    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_additional_fixers(self):
         # used to check if use_2to3_fixers works
         code = 'type(x) is not T'
diff --git a/distutils2/tests/test_pypi_simple.py b/distutils2/tests/test_pypi_simple.py
--- a/distutils2/tests/test_pypi_simple.py
+++ b/distutils2/tests/test_pypi_simple.py
@@ -12,9 +12,9 @@
                                      fake_dec)
 
 try:
-    import _thread
+    import thread as _thread
     from distutils2.tests.pypi_server import (use_pypi_server, PyPIServer,
-                                             PYPI_DEFAULT_STATIC_PATH)
+                                              PYPI_DEFAULT_STATIC_PATH)
 except ImportError:
     _thread = None
     use_pypi_server = fake_dec
diff --git a/distutils2/util.py b/distutils2/util.py
--- a/distutils2/util.py
+++ b/distutils2/util.py
@@ -180,8 +180,8 @@
 
     try:
         return re.sub(r'\$([a-zA-Z_][a-zA-Z_0-9]*)', _subst, s)
-    except KeyError:
-        raise ValueError("invalid variable '$%s'" % sys.exc_info()[1])
+    except KeyError, e:
+        raise ValueError("invalid variable '$%s'" % e)
 
 
 # Needed by 'split_quoted()'
@@ -334,7 +334,7 @@
     """
     # nothing is done if sys.dont_write_bytecode is True
     # FIXME this should not raise an error
-    if hasattr(sys, 'dont_write_bytecode') and sys.dont_write_bytecode:
+    if getattr(sys, 'dont_write_bytecode', False):
         raise PackagingByteCompileError('byte-compiling is disabled.')
 
     # First, if the caller didn't force us into direct or indirect mode,
@@ -354,7 +354,7 @@
     # run it with the appropriate flags.
     if not direct:
         from tempfile import mkstemp
-        # XXX script_fd may leak, use something better than mkstemp
+        # XXX use something better than mkstemp
         script_fd, script_name = mkstemp(".py")
         os.close(script_fd)
         script_fd = None
@@ -365,33 +365,36 @@
             else:
                 script = codecs.open(script_name, "w", encoding='utf-8')
 
-            script.write("""\
+            try:
+                script.write("""\
 from distutils2.util import byte_compile
 files = [
 """)
 
-            # XXX would be nice to write absolute filenames, just for
-            # safety's sake (script should be more robust in the face of
-            # chdir'ing before running it).  But this requires abspath'ing
-            # 'prefix' as well, and that breaks the hack in build_lib's
-            # 'byte_compile()' method that carefully tacks on a trailing
-            # slash (os.sep really) to make sure the prefix here is "just
-            # right".  This whole prefix business is rather delicate -- the
-            # problem is that it's really a directory, but I'm treating it
-            # as a dumb string, so trailing slashes and so forth matter.
+                # XXX would be nice to write absolute filenames, just for
+                # safety's sake (script should be more robust in the face of
+                # chdir'ing before running it).  But this requires abspath'ing
+                # 'prefix' as well, and that breaks the hack in build_lib's
+                # 'byte_compile()' method that carefully tacks on a trailing
+                # slash (os.sep really) to make sure the prefix here is "just
+                # right".  This whole prefix business is rather delicate -- the
+                # problem is that it's really a directory, but I'm treating it
+                # as a dumb string, so trailing slashes and so forth matter.
 
-            #py_files = map(os.path.abspath, py_files)
-            #if prefix:
-            #    prefix = os.path.abspath(prefix)
+                #py_files = map(os.path.abspath, py_files)
+                #if prefix:
+                #    prefix = os.path.abspath(prefix)
 
-            script.write(",\n".join(map(repr, py_files)) + "]\n")
-            script.write("""
+                script.write(",\n".join(map(repr, py_files)) + "]\n")
+                script.write("""
 byte_compile(files, optimize=%r, force=%r,
              prefix=%r, base_dir=%r,
              verbose=%r, dry_run=False,
              direct=True)
 """ % (optimize, force, prefix, base_dir, verbose))
-        script.close()
+            finally:
+                script.close()
+
         cmd = [sys.executable, script_name]
         if optimize == 1:
             cmd.insert(1, "-O")
@@ -553,9 +556,11 @@
     *contents* is a sequence of strings without line terminators.
     """
     f = open(filename, "w")
-    for line in contents:
-        f.write(line + "\n")
-    f.close()
+    try:
+        for line in contents:
+            f.write(line + "\n")
+    finally:
+        f.close()
 
 def _is_package(path):
     return os.path.isdir(path) and os.path.isfile(
@@ -657,8 +662,8 @@
     for part in parts[1:]:
         try:
             ret = getattr(ret, part)
-        except AttributeError:
-            raise ImportError(sys.exc_info()[1])
+        except AttributeError, exc:
+            raise ImportError(exc)
 
     return ret
 
@@ -775,6 +780,7 @@
     _cfg_target = None
     _cfg_target_split = None
 
+
 def spawn(cmd, search_path=True, verbose=0, dry_run=False, env=None):
     """Run another program specified as a command list 'cmd' in a new process.
 
@@ -872,11 +878,12 @@
     """Create a default .pypirc file."""
     rc = get_pypirc_path()
     f = open(rc, 'w')
-    f.write(DEFAULT_PYPIRC % (username, password))
-    f.close()
-
     try:
-        os.chmod(rc, 00600)
+        f.write(DEFAULT_PYPIRC % (username, password))
+    finally:
+        f.close()
+    try:
+        os.chmod(rc, 0600)
     except OSError:
         # should do something better here
         pass
@@ -1084,8 +1091,10 @@
         raise PackagingFileError("file '%s' does not exist" %
                                  os.path.abspath(path))
     f = codecs.open(path, encoding='utf-8')
-    config.readfp(f)
-    f.close()
+    try:
+        config.readfp(f)
+    finally:
+        f.close()
 
     kwargs = {}
     for arg in D1_D2_SETUP_ARGS:
@@ -1108,8 +1117,10 @@
                     in_cfg_value = []
                     for filename in filenames:
                         fp = open(filename)
-                        in_cfg_value.append(fp.read())
-                        fp.close()
+                        try:
+                            in_cfg_value.append(fp.read())
+                        finally:
+                            fp.close()
                     in_cfg_value = '\n\n'.join(in_cfg_value)
             else:
                 continue
@@ -1147,8 +1158,10 @@
         raise PackagingFileError("a setup.py file already exists")
 
     fp = codecs.open("setup.py", "w", encoding='utf-8')
-    fp.write(_SETUP_TMPL % {'func': getsource(cfg_to_args)})
-    fp.close()
+    try:
+        fp.write(_SETUP_TMPL % {'func': getsource(cfg_to_args)})
+    finally:
+        fp.close()
 
 
 # Taken from the pip project
@@ -1168,26 +1181,29 @@
 def _parse_record_file(record_file):
     distinfo, extra_metadata, installed = ({}, [], [])
     rfile = open(record_file, 'r')
-    for path in rfile:
-        path = path.strip()
-        if path.endswith('egg-info') and os.path.isfile(path):
-            distinfo_dir = path.replace('egg-info', 'dist-info')
-            metadata = path
-            egginfo = path
-        elif path.endswith('egg-info') and os.path.isdir(path):
-            distinfo_dir = path.replace('egg-info', 'dist-info')
-            egginfo = path
-            for metadata_file in os.listdir(path):
-                metadata_fpath = os.path.join(path, metadata_file)
-                if metadata_file == 'PKG-INFO':
-                    metadata = metadata_fpath
-                else:
-                    extra_metadata.append(metadata_fpath)
-        elif 'egg-info' in path and os.path.isfile(path):
-            # skip extra metadata files
-            continue
-        else:
-            installed.append(path)
+    try:
+        for path in rfile:
+            path = path.strip()
+            if path.endswith('egg-info') and os.path.isfile(path):
+                distinfo_dir = path.replace('egg-info', 'dist-info')
+                metadata = path
+                egginfo = path
+            elif path.endswith('egg-info') and os.path.isdir(path):
+                distinfo_dir = path.replace('egg-info', 'dist-info')
+                egginfo = path
+                for metadata_file in os.listdir(path):
+                    metadata_fpath = os.path.join(path, metadata_file)
+                    if metadata_file == 'PKG-INFO':
+                        metadata = metadata_fpath
+                    else:
+                        extra_metadata.append(metadata_fpath)
+            elif 'egg-info' in path and os.path.isfile(path):
+                # skip extra metadata files
+                continue
+            else:
+                installed.append(path)
+    finally:
+        rfile.close()
 
     distinfo['egginfo'] = egginfo
     distinfo['metadata'] = metadata
@@ -1204,25 +1220,29 @@
 
 def _write_record_file(record_path, installed_files):
     f = codecs.open(record_path, 'w', encoding='utf-8')
-    writer = csv.writer(f, delimiter=',', lineterminator=os.linesep,
-                        quotechar='"')
+    try:
+        writer = csv.writer(f, delimiter=',', lineterminator=os.linesep,
+                            quotechar='"')
 
-    for fpath in installed_files:
-        if fpath.endswith('.pyc') or fpath.endswith('.pyo'):
-            # do not put size and md5 hash, as in PEP-376
-            writer.writerow((fpath, '', ''))
-        else:
-            hash = hashlib.md5()
-            fp = open(fpath, 'rb')
-            hash.update(fp.read())
-            fp.close()
-            md5sum = hash.hexdigest()
-            size = os.path.getsize(fpath)
-            writer.writerow((fpath, md5sum, size))
+        for fpath in installed_files:
+            if fpath.endswith('.pyc') or fpath.endswith('.pyo'):
+                # do not put size and md5 hash, as in PEP-376
+                writer.writerow((fpath, '', ''))
+            else:
+                hash = hashlib.md5()
+                fp = open(fpath, 'rb')
+                try:
+                    hash.update(fp.read())
+                finally:
+                    fp.close()
+                md5sum = hash.hexdigest()
+                size = os.path.getsize(fpath)
+                writer.writerow((fpath, md5sum, size))
 
-    # add the RECORD file itself
-    writer.writerow((record_path, '', ''))
-    f.close()
+        # add the RECORD file itself
+        writer.writerow((record_path, '', ''))
+    finally:
+        f.close()
     return record_path
 
 
@@ -1258,8 +1278,10 @@
     installer_path = distinfo['installer_path']
     logger.info('creating %s', installer_path)
     f = open(installer_path, 'w')
-    f.write(installer)
-    f.close()
+    try:
+        f.write(installer)
+    finally:
+        f.close()
 
     if requested:
         requested_path = distinfo['requested_path']
@@ -1301,13 +1323,15 @@
 
 def _has_text(setup_py, installer):
     installer_pattern = re.compile('import %s|from %s' % (
-        installer[0], installer[1]))
+        installer[0], installer[0]))
     setup = codecs.open(setup_py, 'r', encoding='utf-8')
-    for line in setup:
-        if re.search(installer_pattern, line):
-            logger.debug("Found %s text in setup.py.", installer)
-            return True
-    setup.close()
+    try:
+        for line in setup:
+            if re.search(installer_pattern, line):
+                logger.debug("Found %s text in setup.py.", installer)
+                return True
+    finally:
+        setup.close()
     logger.debug("No %s text found in setup.py.", installer)
     return False
 
@@ -1315,8 +1339,10 @@
 def _has_required_metadata(setup_cfg):
     config = RawConfigParser()
     f = codecs.open(setup_cfg, 'r', encoding='utf8')
-    config.readfp(f)
-    f.close()
+    try:
+        config.readfp(f)
+    finally:
+        f.close()
     return (config.has_section('metadata') and
             'name' in config.options('metadata') and
             'version' in config.options('metadata'))
@@ -1377,7 +1403,7 @@
                                       _has_distutils_text(setup_py))
 
 
-def is_distutils2(path):
+def is_packaging(path):
     """Check if the project is based on distutils2
 
     :param path: path to source directory containing a setup.cfg file.
@@ -1398,7 +1424,7 @@
 
     Returns a string representing the best install method to use.
     """
-    if is_distutils2(path):
+    if is_packaging(path):
         return "distutils2"
     elif is_setuptools(path):
         return "setuptools"
@@ -1419,8 +1445,8 @@
               "cannot copy tree '%s': not a directory" % src)
     try:
         names = os.listdir(src)
-    except os.error:
-        errstr = sys.exc_info()[1][1]
+    except os.error, e:
+        errstr = e[1]
         if dry_run:
             names = []
         else:
@@ -1465,7 +1491,7 @@
 # I don't use os.makedirs because a) it's new to Python 1.5.2, and
 # b) it blows up if the directory already exists (I want to silently
 # succeed in that case).
-def _mkpath(name, mode=00777, verbose=True, dry_run=False):
+def _mkpath(name, mode=0777, verbose=True, dry_run=False):
     # Detect a common bug -- name is None
     if not isinstance(name, basestring):
         raise PackagingInternalError(
@@ -1506,8 +1532,7 @@
         if not dry_run:
             try:
                 os.mkdir(head, mode)
-            except OSError:
-                exc = sys.exc_info()[1]
+            except OSError, exc:
                 if not (exc.errno == errno.EEXIST and os.path.isdir(head)):
                     raise PackagingFileError(
                           "could not create '%s': %s" % (head, exc.args[-1]))
diff --git a/distutils2/version.py b/distutils2/version.py
--- a/distutils2/version.py
+++ b/distutils2/version.py
@@ -182,7 +182,6 @@
         return "%s('%s')" % (self.__class__.__name__, self)
 
     def _cannot_compare(self, other):
-        import pdb; pdb.set_trace()
         raise TypeError("cannot compare %s and %s"
                 % (type(self).__name__, type(other).__name__))
 

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Let_=E2=80=9Cpython2=2E4?=
 =?utf8?q?_setup=2Epy_build=E2=80=9D_work_=28util_imports_hashlib=29?=
Message-ID: <E1R5deJ-0007lC-8h@dinsdale.python.org>

http://hg.python.org/distutils2/rev/f07fce09ada8
changeset:   1151:f07fce09ada8
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Sep 18 22:34:35 2011 +0200
summary:
  Let ?python2.4 setup.py build? work (util imports hashlib)

files:
  setup.py |  22 +++++++++++++---------
  1 files changed, 13 insertions(+), 9 deletions(-)


diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -7,17 +7,21 @@
 from distutils import sysconfig
 from distutils.core import setup, Extension
 from distutils.ccompiler import new_compiler
-try:
-    from configparser import RawConfigParser
-except ImportError: #<3.0
-    from ConfigParser import RawConfigParser
+from ConfigParser import RawConfigParser
+
+
+def split_multiline(value):
+    return [element for element in (line.strip() for line in value.split('\n'))
+            if element]
+
+def split_elements(value):
+    return [v.strip() for v in value.split(',')]
+
+def split_files(value):
+    return [str(v) for v in split_multiline(value)]
+
 
 def cfg_to_args(path='setup.cfg'):
-    from distutils2.util import split_multiline
-    def split_elements(value):
-        return [ v.strip() for v in value.split(',') ]
-    def split_files(value):
-        return [ str(v) for v in split_multiline(value) ]
     opts_to_args =  {
         'metadata': (
             ('name', 'name', None),

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_backport_changesets_?=
	=?utf8?q?part_2=3A_tests?=
Message-ID: <E1R5deJ-0007l5-6j@dinsdale.python.org>

http://hg.python.org/distutils2/rev/0439c5abecf2
changeset:   1149:0439c5abecf2
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Sep 18 20:23:12 2011 +0200
summary:
  Fix backport changesets part 2: tests

files:
  distutils2/_backport/tests/test_shutil.py         |    3 +-
  distutils2/_backport/tests/test_sysconfig.py      |    2 +-
  distutils2/tests/__init__.py                      |   18 +-
  distutils2/tests/__main__.py                      |    2 +-
  distutils2/tests/pypi_server.py                   |   20 +-
  distutils2/tests/support.py                       |   88 +------
  distutils2/tests/test_command_build_ext.py        |   27 +-
  distutils2/tests/test_command_build_py.py         |    3 +-
  distutils2/tests/test_command_build_scripts.py    |   14 +-
  distutils2/tests/test_command_config.py           |    6 +-
  distutils2/tests/test_command_install_data.py     |    1 +
  distutils2/tests/test_command_install_dist.py     |   17 +-
  distutils2/tests/test_command_install_distinfo.py |   41 ++-
  distutils2/tests/test_command_install_lib.py      |   12 +-
  distutils2/tests/test_command_install_scripts.py  |    6 +-
  distutils2/tests/test_command_register.py         |    9 +-
  distutils2/tests/test_command_sdist.py            |   44 ++-
  distutils2/tests/test_command_upload.py           |   26 +-
  distutils2/tests/test_command_upload_docs.py      |   63 ++---
  distutils2/tests/test_compiler.py                 |    2 +-
  distutils2/tests/test_config.py                   |   10 +-
  distutils2/tests/test_create.py                   |   51 ++-
  distutils2/tests/test_database.py                 |  107 +++++----
  distutils2/tests/test_depgraph.py                 |   22 +-
  distutils2/tests/test_dist.py                     |   33 +-
  distutils2/tests/test_install.py                  |    2 +-
  distutils2/tests/test_manifest.py                 |   12 +-
  distutils2/tests/test_markers.py                  |    2 +-
  distutils2/tests/test_metadata.py                 |   47 ++-
  distutils2/tests/test_mixin2to3.py                |   36 ++-
  distutils2/tests/test_msvc9compiler.py            |   14 +-
  distutils2/tests/test_pypi_server.py              |   37 +-
  distutils2/tests/test_pypi_simple.py              |   36 +-
  distutils2/tests/test_pypi_xmlrpc.py              |    3 +-
  distutils2/tests/test_run.py                      |   29 +--
  distutils2/tests/test_util.py                     |   58 ++--
  36 files changed, 452 insertions(+), 451 deletions(-)


diff --git a/distutils2/_backport/tests/test_shutil.py b/distutils2/_backport/tests/test_shutil.py
--- a/distutils2/_backport/tests/test_shutil.py
+++ b/distutils2/_backport/tests/test_shutil.py
@@ -14,7 +14,8 @@
     register_unpack_format, unregister_unpack_format, get_unpack_formats,
     Error, RegistryError)
 
-from distutils2.tests import unittest, support, TESTFN
+from distutils2.tests import unittest, support
+from test.test_support import TESTFN
 
 try:
     import bz2
diff --git a/distutils2/_backport/tests/test_sysconfig.py b/distutils2/_backport/tests/test_sysconfig.py
--- a/distutils2/_backport/tests/test_sysconfig.py
+++ b/distutils2/_backport/tests/test_sysconfig.py
@@ -14,7 +14,7 @@
         get_config_var, get_config_vars, get_path, get_paths, get_platform,
         get_scheme_names, _main, _SCHEMES)
 
-from distutils2.tests import unittest, TESTFN, unlink
+from distutils2.tests import unittest
 from distutils2.tests.support import EnvironRestorer
 from test.test_support import TESTFN, unlink
 
diff --git a/distutils2/tests/__init__.py b/distutils2/tests/__init__.py
--- a/distutils2/tests/__init__.py
+++ b/distutils2/tests/__init__.py
@@ -6,17 +6,15 @@
 to return an initialized unittest.TestSuite instance.
 
 Utility code is included in distutils2.tests.support.
+
+Always import unittest from this module, it will be the right version
+standard library for packaging tests and unittest2 for distutils2 tests.
 """
 
-# Put this text back for the backport
-#Always import unittest from this module, it will be the right version
-#(standard library unittest for 3.2 and higher, third-party unittest2
-#elease for older versions).
-
 import os
 import sys
 import unittest2 as unittest
-from distutils2.tests.support import TESTFN
+from StringIO import StringIO
 
 # XXX move helpers to support, add tests for them, remove things that
 # duplicate test.support (or keep them for the backport; needs thinking)
@@ -115,7 +113,6 @@
 
 
 def captured_stdout(func, *args, **kw):
-    from StringIO import StringIO
     orig_stdout = getattr(sys, 'stdout')
     setattr(sys, 'stdout', StringIO())
     try:
@@ -131,10 +128,3 @@
         del sys.modules[name]
     except KeyError:
         pass
-
-
-def unlink(filename):
-    try:
-        os.unlink(filename)
-    except OSError:
-        pass
diff --git a/distutils2/tests/__main__.py b/distutils2/tests/__main__.py
--- a/distutils2/tests/__main__.py
+++ b/distutils2/tests/__main__.py
@@ -1,4 +1,4 @@
-"""Packaging test suite runner."""
+"""Distutils2 test suite runner."""
 
 # Ripped from importlib tests, thanks Brett!
 
diff --git a/distutils2/tests/pypi_server.py b/distutils2/tests/pypi_server.py
--- a/distutils2/tests/pypi_server.py
+++ b/distutils2/tests/pypi_server.py
@@ -225,14 +225,18 @@
                         relative_path += "index.html"
 
                     if relative_path.endswith('.tar.gz'):
-                        fp = open(fs_path + relative_path, 'br')
-                        data = fp.read()
-                        fp.close()
+                        file = open(fs_path + relative_path, 'rb')
+                        try:
+                            data = file.read()
+                        finally:
+                            file.close()
                         headers = [('Content-type', 'application/x-gtar')]
                     else:
-                        fp = open(fs_path + relative_path)
-                        data = fp.read().encode()
-                        fp.close()
+                        file = open(fs_path + relative_path)
+                        try:
+                            data = file.read().encode()
+                        finally:
+                            file.close()
                         headers = [('Content-type', 'text/html')]
 
                     headers.append(('Content-Length', len(data)))
@@ -268,8 +272,8 @@
             self.send_header(header, value)
         self.end_headers()
 
-        if type(data) is str:
-            data = data.encode()
+        if isinstance(data, unicode):
+            data = data.encode('utf-8')
 
         self.wfile.write(data)
 
diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -1,5 +1,8 @@
 """Support code for distutils2 test cases.
 
+*This module should not be considered public: its content and API may
+change in incompatible ways.*
+
 A few helper classes are provided: LoggingCatcher, TempdirManager and
 EnvironRestorer. They are written to be used as mixins::
 
@@ -7,6 +10,7 @@
     from distutils2.tests.support import LoggingCatcher
 
     class SomeTestCase(LoggingCatcher, unittest.TestCase):
+        ...
 
 If you need to define a setUp method on your test class, you have to
 call the mixin class' setUp method or it won't work (same thing for
@@ -14,7 +18,7 @@
 
         def setUp(self):
             super(SomeTestCase, self).setUp()
-            ... # other setup code
+            ...  # other setup code
 
 Also provided is a DummyCommand class, useful to mock commands in the
 tests of another command that needs them, a create_distribution function
@@ -25,15 +29,16 @@
 and a skip_unless_symlink decorator.
 
 Each class or function has a docstring to explain its purpose and usage.
+Existing tests should also be used as examples.
 """
 
+import os
+import sys
 import codecs
-import os
 import shutil
 import logging
 import logging.handlers
 import subprocess
-import sys
 import weakref
 import tempfile
 try:
@@ -172,8 +177,10 @@
         if isinstance(path, (list, tuple)):
             path = os.path.join(*path)
         f = codecs.open(path, 'w', encoding=encoding)
-        f.write(content)
-        f.close()
+        try:
+            f.write(content)
+        finally:
+            f.close()
 
     def create_dist(self, **kw):
         """Create a stub distribution object and files.
@@ -244,6 +251,8 @@
     Useful for mocking one dependency command in the tests for another
     command, see e.g. the dummy build command in test_build_scripts.
     """
+    # XXX does not work with dist.get_reinitialized_command, which typechecks
+    # and wants a finalized attribute
 
     def __init__(self, **kwargs):
         for kw, val in kwargs.items():
@@ -284,74 +293,17 @@
     return _wrap
 
 
-#try:
-#    from test.support import skip_unless_symlink
-#except ImportError:
-#    skip_unless_symlink = unittest.skip(
-#        'requires test.support.skip_unless_symlink')
+try:
+    from test.test_support import skip_unless_symlink
+except ImportError:
+    skip_unless_symlink = unittest.skip(
+        'requires test.support.skip_unless_symlink')
 
-if os.name == 'java':
-    # Jython disallows @ in module names
-    TESTFN = '$test'
-else:
-    TESTFN = '@test'
-
-# Disambiguate TESTFN for parallel testing, while letting it remain a valid
-# module name.
-TESTFN = "%s_%s_tmp" % (TESTFN, os.getpid())
-
-
-# TESTFN_UNICODE is a non-ascii filename
-TESTFN_UNICODE = TESTFN + "-\xe0\xf2\u0258\u0141\u011f"
-if sys.platform == 'darwin':
-    # In Mac OS X's VFS API file names are, by definition, canonically
-    # decomposed Unicode, encoded using UTF-8. See QA1173:
-    # http://developer.apple.com/mac/library/qa/qa2001/qa1173.html
-    import unicodedata
-    TESTFN_UNICODE = unicodedata.normalize('NFD', TESTFN_UNICODE)
-TESTFN_ENCODING = sys.getfilesystemencoding()
-
-# TESTFN_UNENCODABLE is a filename (str type) that should *not* be able to be
-# encoded by the filesystem encoding (in strict mode). It can be None if we
-# cannot generate such filename.
-TESTFN_UNENCODABLE = None
-if os.name in ('nt', 'ce'):
-    # skip win32s (0) or Windows 9x/ME (1)
-    if sys.getwindowsversion().platform >= 2:
-        # Different kinds of characters from various languages to minimize the
-        # probability that the whole name is encodable to MBCS (issue #9819)
-        TESTFN_UNENCODABLE = TESTFN + "-\u5171\u0141\u2661\u0363\uDC80"
-        try:
-            TESTFN_UNENCODABLE.encode(TESTFN_ENCODING)
-        except UnicodeEncodeError:
-            pass
-        else:
-            print ('WARNING: The filename %r CAN be encoded by the filesystem encoding (%s). '
-                   'Unicode filename tests may not be effective'
-                   % (TESTFN_UNENCODABLE, TESTFN_ENCODING))
-            TESTFN_UNENCODABLE = None
-# Mac OS X denies unencodable filenames (invalid utf-8)
-elif sys.platform != 'darwin':
-    try:
-        # ascii and utf-8 cannot encode the byte 0xff
-        '\xff'.decode(TESTFN_ENCODING)
-    except UnicodeDecodeError:
-        # 0xff will be encoded using the surrogate character u+DCFF
-        try:
-            TESTFN_UNENCODABLE = TESTFN \
-                + '-\xff'.decode(TESTFN_ENCODING, 'surrogateescape')
-        except LookupError:
-            pass
-    else:
-        # File system encoding (eg. ISO-8859-* encodings) can encode
-        # the byte 0xff. Skip some unicode filename tests.
-        pass
 
 def unlink(filename):
     try:
         os.unlink(filename)
-    except OSError:
-        error = sys.exc_info()[1]
+    except OSError, error:
         # The filename need not exist.
         if error.errno not in (errno.ENOENT, errno.ENOTDIR):
             raise
diff --git a/distutils2/tests/test_command_build_ext.py b/distutils2/tests/test_command_build_ext.py
--- a/distutils2/tests/test_command_build_ext.py
+++ b/distutils2/tests/test_command_build_ext.py
@@ -8,7 +8,7 @@
 from distutils2._backport.sysconfig import _CONFIG_VARS
 from distutils2.dist import Distribution
 from distutils2.errors import (UnknownFileError, CompileError,
-                              PackagingPlatformError)
+                               PackagingPlatformError)
 from distutils2.command.build_ext import build_ext
 from distutils2.compiler.extension import Extension
 from distutils2.tests.support import assert_python_ok
@@ -30,8 +30,6 @@
                        support.LoggingCatcher,
                        unittest.TestCase):
     def setUp(self):
-        # Create a simple test environment
-        # Note that we're making changes to sys.path
         super(BuildExtTestCase, self).setUp()
         self.tmp_dir = self.mkdtemp()
         filename = _get_source_filename()
@@ -40,13 +38,10 @@
         if sys.version > "2.6":
             self.old_user_base = site.USER_BASE
             site.USER_BASE = self.mkdtemp()
-            build_ext.USER_BASE = site.USER_BASE
 
     def tearDown(self):
-        # Get everything back to normal
         if sys.version > "2.6":
             site.USER_BASE = self.old_user_base
-            build_ext.USER_BASE = self.old_user_base
 
         super(BuildExtTestCase, self).tearDown()
 
@@ -94,10 +89,8 @@
         try:
             cmd.ensure_finalized()
             cmd.run()
-        except:
+        finally:
             sys.stdout = old_stdout
-            raise
-        sys.stdout = old_stdout
 
         code = """if 1:
             import sys
@@ -127,18 +120,14 @@
 
         old_var = _CONFIG_VARS.get('Py_ENABLE_SHARED')
         _CONFIG_VARS['Py_ENABLE_SHARED'] = 1
-        def cleanup():
+        try:
+            cmd.ensure_finalized()
+        finally:
             sys.platform = old
             if old_var is None:
                 del _CONFIG_VARS['Py_ENABLE_SHARED']
             else:
                 _CONFIG_VARS['Py_ENABLE_SHARED'] = old_var
-        try:
-            cmd.ensure_finalized()
-        except:
-            cleanup()
-            raise
-        cleanup()
 
         # make sure we get some library dirs under solaris
         self.assertGreater(len(cmd.library_dirs), 0)
@@ -415,7 +404,8 @@
         deptarget_c = os.path.join(self.tmp_dir, 'deptargetmodule.c')
 
         fp = open(deptarget_c, 'w')
-        fp.write(textwrap.dedent('''\
+        try:
+            fp.write(textwrap.dedent('''\
                 #include <AvailabilityMacros.h>
 
                 int dummy;
@@ -426,7 +416,8 @@
                 #endif
 
             ''' % operator))
-        fp.close()
+        finally:
+            fp.close()
 
         # get the deployment target that the interpreter was built with
         target = sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET')
diff --git a/distutils2/tests/test_command_build_py.py b/distutils2/tests/test_command_build_py.py
--- a/distutils2/tests/test_command_build_py.py
+++ b/distutils2/tests/test_command_build_py.py
@@ -61,7 +61,7 @@
         self.assertIn("__init__.py", files)
         self.assertIn("README.txt", files)
         # XXX even with -O, distutils writes pyc, not pyo; bug?
-        if hasattr(sys , 'dont_write_bytecode'):
+        if getattr(sys , 'dont_write_bytecode', False):
             self.assertNotIn("__init__.pyc", files)
         else:
             self.assertIn("__init__.pyc", files)
@@ -81,6 +81,7 @@
 
         os.chdir(sources)
         old_stdout = sys.stdout
+        #sys.stdout = StringIO.StringIO()
 
         try:
             dist = Distribution({"packages": ["pkg"],
diff --git a/distutils2/tests/test_command_build_scripts.py b/distutils2/tests/test_command_build_scripts.py
--- a/distutils2/tests/test_command_build_scripts.py
+++ b/distutils2/tests/test_command_build_scripts.py
@@ -2,9 +2,10 @@
 
 import os
 import sys
-from distutils2._backport import sysconfig
 from distutils2.dist import Distribution
 from distutils2.command.build_scripts import build_scripts
+from distutils2._backport import sysconfig
+
 from distutils2.tests import unittest, support
 
 
@@ -71,8 +72,10 @@
 
     def write_script(self, dir, name, text):
         f = open(os.path.join(dir, name), "w")
-        f.write(text)
-        f.close()
+        try:
+            f.write(text)
+        finally:
+            f.close()
 
     def test_version_int(self):
         source = self.mkdtemp()
@@ -94,12 +97,9 @@
         sysconfig._CONFIG_VARS['VERSION'] = 4
         try:
             cmd.run()
-        except:
+        finally:
             if old is not None:
                 sysconfig._CONFIG_VARS['VERSION'] = old
-            raise
-        if old is not None:
-            sysconfig._CONFIG_VARS['VERSION'] = old
 
         built = os.listdir(target)
         for name in expected:
diff --git a/distutils2/tests/test_command_config.py b/distutils2/tests/test_command_config.py
--- a/distutils2/tests/test_command_config.py
+++ b/distutils2/tests/test_command_config.py
@@ -14,8 +14,10 @@
     def test_dump_file(self):
         this_file = __file__.rstrip('co')
         f = open(this_file)
-        numlines = len(f.readlines())
-        f.close()
+        try:
+            numlines = len(f.readlines())
+        finally:
+            f.close()
 
         dump_file(this_file, 'I am the header')
 
diff --git a/distutils2/tests/test_command_install_data.py b/distutils2/tests/test_command_install_data.py
--- a/distutils2/tests/test_command_install_data.py
+++ b/distutils2/tests/test_command_install_data.py
@@ -35,6 +35,7 @@
         two = os.path.join(pkg_dir, 'two')
         self.write_file(two, 'xxx')
 
+        # FIXME this creates a literal \{inst2\} directory!
         cmd.data_files = {one: '{inst}/one', two: '{inst2}/two'}
         self.assertItemsEqual(cmd.get_inputs(), [one, two])
 
diff --git a/distutils2/tests/test_command_install_dist.py b/distutils2/tests/test_command_install_dist.py
--- a/distutils2/tests/test_command_install_dist.py
+++ b/distutils2/tests/test_command_install_dist.py
@@ -43,12 +43,9 @@
             cmd = install_dist(dist)
             cmd.home = destination
             cmd.ensure_finalized()
-        except:
+        finally:
             _SCHEMES.set('posix_prefix', 'platinclude', old_posix_prefix)
             _SCHEMES.set('posix_home', 'platinclude', old_posix_home)
-            raise
-        _SCHEMES.set('posix_prefix', 'platinclude', old_posix_prefix)
-        _SCHEMES.set('posix_home', 'platinclude', old_posix_home)
 
         self.assertEqual(cmd.install_base, destination)
         self.assertEqual(cmd.install_platbase, destination)
@@ -88,17 +85,13 @@
         self.old_expand = os.path.expanduser
         os.path.expanduser = _expanduser
 
-        def cleanup():
+        try:
+            # this is the actual test
+            self._test_user_site()
+        finally:
             _CONFIG_VARS['userbase'] = self.old_user_base
             _SCHEMES.set(scheme, 'purelib', self.old_user_site)
             os.path.expanduser = self.old_expand
-        try:
-            # this is the actual test
-            self._test_user_site()
-        except:
-            cleanup()
-            raise
-        cleanup()
 
     def _test_user_site(self):
         schemes = get_scheme_names()
diff --git a/distutils2/tests/test_command_install_distinfo.py b/distutils2/tests/test_command_install_distinfo.py
--- a/distutils2/tests/test_command_install_distinfo.py
+++ b/distutils2/tests/test_command_install_distinfo.py
@@ -58,14 +58,16 @@
         self.checkLists(os.listdir(dist_info),
                         ['METADATA', 'RECORD', 'REQUESTED', 'INSTALLER'])
         fp = open(os.path.join(dist_info, 'INSTALLER'))
-        content = fp.read()
-        fp.close()
-        self.assertEqual(content, 'distutils')
+        try:
+            self.assertEqual(fp.read(), 'distutils')
+        finally:
+            fp.close()
 
         fp = open(os.path.join(dist_info, 'REQUESTED'))
-        content = fp.read()
-        fp.close()
-        self.assertEqual(content, '')
+        try:
+            self.assertEqual(fp.read(), '')
+        finally:
+            fp.close()
 
         meta_path = os.path.join(dist_info, 'METADATA')
         self.assertTrue(Metadata(path=meta_path).check())
@@ -89,9 +91,10 @@
 
         dist_info = os.path.join(install_dir, 'foo-1.0.dist-info')
         fp = open(os.path.join(dist_info, 'INSTALLER'))
-        content = fp.read()
-        fp.close()
-        self.assertEqual(content, 'bacon-python')
+        try:
+            self.assertEqual(fp.read(), 'bacon-python')
+        finally:
+            fp.close()
 
     def test_requested(self):
         pkg_dir, dist = self.create_dist(name='foo',
@@ -172,25 +175,29 @@
 
         expected = []
         for f in install.get_outputs():
-            if (f.endswith('.pyc',) or f.endswith('.pyo') or f == os.path.join(
+            if (f.endswith('.pyc') or f.endswith('.pyo') or f == os.path.join(
                 install_dir, 'foo-1.0.dist-info', 'RECORD')):
                 expected.append([f, '', ''])
             else:
                 size = os.path.getsize(f)
                 md5 = hashlib.md5()
                 fp = open(f, 'rb')
-                md5.update(fp.read())
-                fp.close()
+                try:
+                    md5.update(fp.read())
+                finally:
+                    fp.close()
                 hash = md5.hexdigest()
                 expected.append([f, hash, str(size)])
 
         parsed = []
         f = open(os.path.join(dist_info, 'RECORD'), 'r')
-        reader = csv.reader(f, delimiter=',',
-                               lineterminator=os.linesep,
-                               quotechar='"')
-        parsed = list(reader)
-        f.close()
+        try:
+            reader = csv.reader(f, delimiter=',',
+                                   lineterminator=os.linesep,
+                                   quotechar='"')
+            parsed = list(reader)
+        finally:
+            f.close()
 
         self.maxDiff = None
         self.checkLists(parsed, expected)
diff --git a/distutils2/tests/test_command_install_lib.py b/distutils2/tests/test_command_install_lib.py
--- a/distutils2/tests/test_command_install_lib.py
+++ b/distutils2/tests/test_command_install_lib.py
@@ -7,13 +7,6 @@
 from distutils2.compiler.extension import Extension
 from distutils2.errors import PackagingOptionError
 
-try:
-    no_bytecode = sys.dont_write_bytecode
-    bytecode_support = True
-except AttributeError:
-    no_bytecode = False
-    bytecode_support = False
-
 
 class InstallLibTestCase(support.TempdirManager,
                          support.LoggingCatcher,
@@ -40,7 +33,8 @@
         cmd.finalize_options()
         self.assertEqual(cmd.optimize, 2)
 
-    @unittest.skipIf(no_bytecode, 'byte-compile not supported')
+    @unittest.skipIf(getattr(sys, 'dont_write_bytecode', False),
+                     'byte-compile disabled')
     def test_byte_compile(self):
         pkg_dir, dist = self.create_dist()
         cmd = install_lib(dist)
@@ -89,7 +83,7 @@
         # get_input should return 2 elements
         self.assertEqual(len(cmd.get_inputs()), 2)
 
-    @unittest.skipUnless(bytecode_support,
+    @unittest.skipUnless(hasattr(sys, 'dont_write_bytecode'),
                          'sys.dont_write_bytecode not supported')
     def test_dont_write_bytecode(self):
         # makes sure byte_compile is not used
diff --git a/distutils2/tests/test_command_install_scripts.py b/distutils2/tests/test_command_install_scripts.py
--- a/distutils2/tests/test_command_install_scripts.py
+++ b/distutils2/tests/test_command_install_scripts.py
@@ -39,8 +39,10 @@
         def write_script(name, text):
             expected.append(name)
             f = open(os.path.join(source, name), "w")
-            f.write(text)
-            f.close()
+            try:
+                f.write(text)
+            finally:
+                f.close()
 
         write_script("script1.py", ("#! /usr/bin/env python2.3\n"
                                     "# bogus script w/ Python sh-bang\n"
diff --git a/distutils2/tests/test_command_register.py b/distutils2/tests/test_command_register.py
--- a/distutils2/tests/test_command_register.py
+++ b/distutils2/tests/test_command_register.py
@@ -1,3 +1,4 @@
+# encoding: utf-8
 """Tests for distutils2.command.register."""
 import os
 import getpass
@@ -129,8 +130,10 @@
 
         # with the content similar to WANTED_PYPIRC
         fp = open(self.rc)
-        content = fp.read()
-        fp.close()
+        try:
+            content = fp.read()
+        finally:
+            fp.close()
         self.assertEqual(content, WANTED_PYPIRC)
 
         # now let's make sure the .pypirc file generated
@@ -213,7 +216,7 @@
 
         # metadata is OK but long_description is broken
         metadata = {'home_page': 'xxx', 'author': 'xxx',
-                    'author_email': '\xc3x\xc3x\xc3',
+                    'author_email': '?x?x?',
                     'name': 'xxx', 'version': 'xxx',
                     'description': 'title\n==\n\ntext'}
 
diff --git a/distutils2/tests/test_command_sdist.py b/distutils2/tests/test_command_sdist.py
--- a/distutils2/tests/test_command_sdist.py
+++ b/distutils2/tests/test_command_sdist.py
@@ -214,8 +214,10 @@
 
         # Checking the MANIFEST
         fp = open(join(self.tmp_dir, 'MANIFEST'))
-        manifest = fp.read()
-        fp.close()
+        try:
+            manifest = fp.read()
+        finally:
+            fp.close()
         self.assertEqual(manifest, MANIFEST % {'sep': os.sep})
 
     @requires_zlib
@@ -332,9 +334,11 @@
         # Should produce four lines. Those lines are one comment, one default
         # (README) and two package files.
         f = open(cmd.manifest)
-        manifest = [line.strip() for line in f.read().split('\n')
-                    if line.strip() != '']
-        f.close()
+        try:
+            manifest = [line.strip() for line in f.read().split('\n')
+                        if line.strip() != '']
+        finally:
+            f.close()
         self.assertEqual(len(manifest), 3)
 
         # Adding a file
@@ -348,9 +352,11 @@
         cmd.run()
 
         f = open(cmd.manifest)
-        manifest2 = [line.strip() for line in f.read().split('\n')
-                     if line.strip() != '']
-        f.close()
+        try:
+            manifest2 = [line.strip() for line in f.read().split('\n')
+                         if line.strip() != '']
+        finally:
+            f.close()
 
         # Do we have the new file in MANIFEST?
         self.assertEqual(len(manifest2), 4)
@@ -364,9 +370,11 @@
         cmd.run()
 
         f = open(cmd.manifest)
-        manifest = [line.strip() for line in f.read().split('\n')
-                    if line.strip() != '']
-        f.close()
+        try:
+            manifest = [line.strip() for line in f.read().split('\n')
+                        if line.strip() != '']
+        finally:
+            f.close()
 
         self.assertEqual(manifest[0],
                          '# file GENERATED by distutils2, do NOT edit')
@@ -380,9 +388,11 @@
         cmd.run()
 
         f = open(cmd.manifest)
-        manifest = [line.strip() for line in f.read().split('\n')
-                    if line.strip() != '']
-        f.close()
+        try:
+            manifest = [line.strip() for line in f.read().split('\n')
+                        if line.strip() != '']
+        finally:
+            f.close()
 
         self.assertEqual(manifest, ['README.manual'])
 
@@ -394,8 +404,10 @@
         self.write_file((self.tmp_dir, 'yeah'), 'xxx')
         cmd.run()
         f = open(cmd.manifest)
-        content = f.read()
-        f.close()
+        try:
+            content = f.read()
+        finally:
+            f.close()
 
         self.assertIn('yeah', content)
 
diff --git a/distutils2/tests/test_command_upload.py b/distutils2/tests/test_command_upload.py
--- a/distutils2/tests/test_command_upload.py
+++ b/distutils2/tests/test_command_upload.py
@@ -1,3 +1,4 @@
+# encoding: utf-8
 """Tests for distutils2.command.upload."""
 import os
 
@@ -101,22 +102,23 @@
         command, pyversion, filename = 'xxx', '3.3', path
         dist_files = [(command, pyversion, filename)]
 
-        # lets run it
-        pkg_dir, dist = self.create_dist(dist_files=dist_files, author='d\xc3d\xc3')
+        # let's run it
+        dist = self.create_dist(dist_files=dist_files, author='d?d?')[1]
         cmd = upload(dist)
         cmd.ensure_finalized()
         cmd.repository = self.pypi.full_address
         cmd.run()
 
-        # what did we send ?
+        # what did we send?
         handler, request_data = self.pypi.requests[-1]
         headers = handler.headers
-        #self.assertIn('d\xc3d\xc3', str(request_data))
+        self.assertIn('d?d?', request_data)
         self.assertIn('xxx', request_data)
 
         self.assertEqual(int(headers['content-length']), len(request_data))
         self.assertLess(int(headers['content-length']), 2500)
-        self.assertTrue(headers['content-type'].startswith('multipart/form-data'))
+        self.assertTrue(headers['content-type'].startswith(
+            'multipart/form-data'))
         self.assertEqual(handler.command, 'POST')
         self.assertNotIn('\n', headers['authorization'])
 
@@ -130,20 +132,16 @@
         self.write_file(os.path.join(docs_path, "index.html"), "yellow")
         self.write_file(self.rc, PYPIRC)
 
-        # lets run it
-        pkg_dir, dist = self.create_dist(dist_files=dist_files, author='d\xc3d\xc3')
+        # let's run it
+        dist = self.create_dist(dist_files=dist_files, author='d?d?')[1]
 
         cmd = upload(dist)
         cmd.get_finalized_command("build").run()
         cmd.upload_docs = True
         cmd.ensure_finalized()
         cmd.repository = self.pypi.full_address
-        prev_dir = os.getcwd()
-        try:
-            os.chdir(self.tmp_dir)
-            cmd.run()
-        finally:
-            os.chdir(prev_dir)
+        os.chdir(self.tmp_dir)
+        cmd.run()
 
         handler, request_data = self.pypi.requests[-1]
         action, name, content = request_data.split(
@@ -156,6 +154,8 @@
 
 UploadTestCase = unittest.skipIf(threading is None, 'needs threading')(
         UploadTestCase)
+
+
 def test_suite():
     return unittest.makeSuite(UploadTestCase)
 
diff --git a/distutils2/tests/test_command_upload_docs.py b/distutils2/tests/test_command_upload_docs.py
--- a/distutils2/tests/test_command_upload_docs.py
+++ b/distutils2/tests/test_command_upload_docs.py
@@ -50,34 +50,27 @@
 
     def test_default_uploaddir(self):
         sandbox = self.mkdtemp()
-        previous = os.getcwd()
         os.chdir(sandbox)
-        try:
-            os.mkdir("build")
-            self.prepare_sample_dir("build")
-            self.cmd.ensure_finalized()
-            self.assertEqual(self.cmd.upload_dir, os.path.join("build", "docs"))
-        finally:
-            os.chdir(previous)
+        os.mkdir("build")
+        self.prepare_sample_dir("build")
+        self.cmd.ensure_finalized()
+        self.assertEqual(self.cmd.upload_dir, os.path.join("build", "docs"))
 
     def test_default_uploaddir_looks_for_doc_also(self):
         sandbox = self.mkdtemp()
-        previous = os.getcwd()
         os.chdir(sandbox)
-        try:
-            os.mkdir("build")
-            self.prepare_sample_dir("build")
-            os.rename(os.path.join("build", "docs"), os.path.join("build", "doc"))
-            self.cmd.ensure_finalized()
-            self.assertEqual(self.cmd.upload_dir, os.path.join("build", "doc"))
-        finally:
-            os.chdir(previous)
+        os.mkdir("build")
+        self.prepare_sample_dir("build")
+        os.rename(os.path.join("build", "docs"), os.path.join("build", "doc"))
+        self.cmd.ensure_finalized()
+        self.assertEqual(self.cmd.upload_dir, os.path.join("build", "doc"))
 
     def prepare_sample_dir(self, sample_dir=None):
         if sample_dir is None:
             sample_dir = self.mkdtemp()
         os.mkdir(os.path.join(sample_dir, "docs"))
-        self.write_file(os.path.join(sample_dir, "docs", "index.html"), "Ce mortel ennui")
+        self.write_file(os.path.join(sample_dir, "docs", "index.html"),
+                        "Ce mortel ennui")
         self.write_file(os.path.join(sample_dir, "index.html"), "Oh la la")
         return sample_dir
 
@@ -106,9 +99,8 @@
         self.assertTrue(handler.headers['content-type']
             .startswith('multipart/form-data;'))
 
-        action, name, version, content =\
-            request_data.split("----------------GHSKFJDLGDS7543FJKLFHRE75642756743254".encode())[1:5]
-
+        action, name, version, content = request_data.split(
+            '----------------GHSKFJDLGDS7543FJKLFHRE75642756743254')[1:5]
 
         # check that we picked the right chunks
         self.assertIn('name=":action"', action)
@@ -125,8 +117,9 @@
     @unittest.skipIf(_ssl is None, 'Needs SSL support')
     def test_https_connection(self):
         self.https_called = False
-
-        orig_https = upload_docs_mod.httplib.HTTPSConnection
+        self.addCleanup(
+            setattr, upload_docs_mod.httplib, 'HTTPSConnection',
+            upload_docs_mod.httplib.HTTPSConnection)
 
         def https_conn_wrapper(*args):
             self.https_called = True
@@ -134,16 +127,14 @@
             return upload_docs_mod.httplib.HTTPConnection(*args)
 
         upload_docs_mod.httplib.HTTPSConnection = https_conn_wrapper
-        try:
-            self.prepare_command()
-            self.cmd.run()
-            self.assertFalse(self.https_called)
 
-            self.cmd.repository = self.cmd.repository.replace("http", "https")
-            self.cmd.run()
-            self.assertTrue(self.https_called)
-        finally:
-            upload_docs_mod.httplib.HTTPSConnection = orig_https
+        self.prepare_command()
+        self.cmd.run()
+        self.assertFalse(self.https_called)
+
+        self.cmd.repository = self.cmd.repository.replace("http", "https")
+        self.cmd.run()
+        self.assertTrue(self.https_called)
 
     def test_handling_response(self):
         self.pypi.default_response_status = '403 Forbidden'
@@ -152,7 +143,8 @@
         self.assertIn('Upload failed (403): Forbidden', self.get_logs()[-1])
 
         self.pypi.default_response_status = '301 Moved Permanently'
-        self.pypi.default_response_headers.append(("Location", "brand_new_location"))
+        self.pypi.default_response_headers.append(
+            ("Location", "brand_new_location"))
         self.cmd.run()
         self.assertIn('brand_new_location', self.get_logs()[-1])
 
@@ -182,7 +174,10 @@
         self.assertTrue(record, "should report the response")
         self.assertIn(self.pypi.default_response_data, record)
 
-UploadDocsTestCase = unittest.skipIf(threading is None, "Needs threading")(UploadDocsTestCase)
+UploadDocsTestCase = unittest.skipIf(threading is None, "Needs threading")(
+    UploadDocsTestCase)
+
+
 def test_suite():
     return unittest.makeSuite(UploadDocsTestCase)
 
diff --git a/distutils2/tests/test_compiler.py b/distutils2/tests/test_compiler.py
--- a/distutils2/tests/test_compiler.py
+++ b/distutils2/tests/test_compiler.py
@@ -2,7 +2,7 @@
 import os
 
 from distutils2.compiler import (get_default_compiler, customize_compiler,
-                                gen_lib_options)
+                                 gen_lib_options)
 from distutils2.tests import unittest, support
 
 
diff --git a/distutils2/tests/test_config.py b/distutils2/tests/test_config.py
--- a/distutils2/tests/test_config.py
+++ b/distutils2/tests/test_config.py
@@ -1,3 +1,4 @@
+# encoding: utf-8
 """Tests for distutils2.config."""
 import os
 import sys
@@ -20,7 +21,7 @@
 version = 0.6.4
 author = Carl Meyer
 author_email = carl at oddbird.net
-maintainer = \xc3ric Araujo
+maintainer = ?ric Araujo
 maintainer_email = merwok at netwok.org
 summary = A sample project demonstrating distutils2
 description-file = %(description-file)s
@@ -441,9 +442,10 @@
         cmd.get_file_list()
         cmd.make_distribution()
         fp = open('MANIFEST')
-        content = fp.read()
-        fp.close()
-        self.assertIn('README\nREADME2\n', content)
+        try:
+            self.assertIn('README\nREADME2\n', fp.read())
+        finally:
+            fp.close()
 
     def test_sub_commands(self):
         self.write_setup()
diff --git a/distutils2/tests/test_create.py b/distutils2/tests/test_create.py
--- a/distutils2/tests/test_create.py
+++ b/distutils2/tests/test_create.py
@@ -1,11 +1,12 @@
+# encoding: utf-8
 """Tests for distutils2.create."""
-from StringIO import StringIO
-import codecs
 import os
 import sys
+import codecs
+from StringIO import StringIO
 from textwrap import dedent
+from distutils2.create import MainProgram, ask_yn, ask, main
 from distutils2._backport import sysconfig
-from distutils2.create import MainProgram, ask_yn, ask, main
 
 from distutils2.tests import support, unittest
 
@@ -40,13 +41,13 @@
         super(CreateTestCase, self).tearDown()
 
     def test_ask_yn(self):
-        sys.stdin.write(u'y\n')
+        sys.stdin.write('y\n')
         sys.stdin.seek(0)
         self.assertEqual('y', ask_yn('is this a test'))
 
     def test_ask(self):
-        sys.stdin.write(u'a\n')
-        sys.stdin.write(u'b\n')
+        sys.stdin.write('a\n')
+        sys.stdin.write('b\n')
         sys.stdin.seek(0)
         self.assertEqual('a', ask('is this a test'))
         self.assertEqual('b', ask(str(list(range(0, 70))), default='c',
@@ -54,7 +55,7 @@
 
     def test_set_multi(self):
         mainprogram = MainProgram()
-        sys.stdin.write(u'aaaaa\n')
+        sys.stdin.write('aaaaa\n')
         sys.stdin.seek(0)
         mainprogram.data['author'] = []
         mainprogram._set_multi('_set_multi test', 'author')
@@ -109,7 +110,7 @@
               version='0.2',
               description='Python bindings for the Xfoil engine',
               long_description=long_description,
-              maintainer='Andr\xc3 Espaze',
+              maintainer='Andr? Espaze',
               maintainer_email='andre.espaze at logilab.fr',
               url='http://www.python-science.org/project/pyxfoil',
               license='GPLv2',
@@ -136,10 +137,12 @@
         sys.stdin.seek(0)
         main()
 
-        fp = codecs.open(os.path.join(self.wdir, 'setup.cfg'),
-                encoding='utf-8')
-        contents = fp.read()
-        fp.close()
+        path = os.path.join(self.wdir, 'setup.cfg')
+        fp = codecs.open(path, encoding='utf-8')
+        try:
+            contents = fp.read()
+        finally:
+            fp.close()
 
         self.assertEqual(contents, dedent(u"""\
             [metadata]
@@ -148,7 +151,7 @@
             summary = Python bindings for the Xfoil engine
             download_url = UNKNOWN
             home_page = http://www.python-science.org/project/pyxfoil
-            maintainer = Andr\xc3 Espaze
+            maintainer = Andr? Espaze
             maintainer_email = andre.espaze at logilab.fr
             description = My super Death-scription
                    |barbar is now on the public domain,
@@ -185,14 +188,16 @@
         # coding: utf-8
         from distutils.core import setup
         fp = open('README.txt')
-        long_description = fp.read()
-        fp.close()
+        try:
+            long_description = fp.read()
+        finally:
+            fp.close()
 
         setup(name='pyxfoil',
               version='0.2',
               description='Python bindings for the Xfoil engine',
               long_description=long_description,
-              maintainer='Andr\xc3 Espaze',
+              maintainer='Andr? Espaze',
               maintainer_email='andre.espaze at logilab.fr',
               url='http://www.python-science.org/project/pyxfoil',
               license='GPLv2',
@@ -210,13 +215,17 @@
 barbar is now in the public domain,
 ho, baby!
                         '''))
-        sys.stdin.write(u'y\n')
+        sys.stdin.write('y\n')
         sys.stdin.seek(0)
         # FIXME Out of memory error.
         main()
-        fp = codecs.open(os.path.join(self.wdir, 'setup.cfg'), encoding='utf-8')
-        contents = fp.read()
-        fp.close()
+
+        path = os.path.join(self.wdir, 'setup.cfg')
+        fp = codecs.open(path, encoding='utf-8')
+        try:
+            contents = fp.read()
+        finally:
+            fp.close()
 
         self.assertEqual(contents, dedent(u"""\
             [metadata]
@@ -225,7 +234,7 @@
             summary = Python bindings for the Xfoil engine
             download_url = UNKNOWN
             home_page = http://www.python-science.org/project/pyxfoil
-            maintainer = Andr\xc3 Espaze
+            maintainer = Andr? Espaze
             maintainer_email = andre.espaze at logilab.fr
             description-file = README.txt
 
diff --git a/distutils2/tests/test_database.py b/distutils2/tests/test_database.py
--- a/distutils2/tests/test_database.py
+++ b/distutils2/tests/test_database.py
@@ -33,8 +33,10 @@
 
 def get_hexdigest(filename):
     fp = open(filename, 'rb')
-    checksum = md5(fp.read())
-    fp.close()
+    try:
+        checksum = md5(fp.read())
+    finally:
+        fp.close()
     return checksum.hexdigest()
 
 
@@ -64,12 +66,13 @@
         # back (to avoid getting a read-only copy of a read-only file).  we
         # could pass a custom copy_function to change the mode of files, but
         # shutil gives no control over the mode of directories :(
+        # see http://bugs.python.org/issue1666318
         for root, dirs, files in os.walk(self.fake_dists_path):
-            os.chmod(root, 00755)
+            os.chmod(root, 0755)
             for f in files:
-                os.chmod(os.path.join(root, f), 00644)
+                os.chmod(os.path.join(root, f), 0644)
             for d in dirs:
-                os.chmod(os.path.join(root, d), 00755)
+                os.chmod(os.path.join(root, d), 0755)
 
 
 class CommonDistributionTests(FakeDistsMixin):
@@ -139,32 +142,36 @@
 
             record_file = os.path.join(distinfo_dir, 'RECORD')
             fp = open(record_file, 'w')
-            record_writer = csv.writer(
-                fp, delimiter=',', quoting=csv.QUOTE_NONE,
-                lineterminator='\n')
+            try:
+                record_writer = csv.writer(
+                    fp, delimiter=',', quoting=csv.QUOTE_NONE,
+                    lineterminator='\n')
 
-            dist_location = distinfo_dir.replace('.dist-info', '')
+                dist_location = distinfo_dir.replace('.dist-info', '')
 
-            for path, dirs, files in os.walk(dist_location):
-                for f in files:
+                for path, dirs, files in os.walk(dist_location):
+                    for f in files:
+                        record_writer.writerow(record_pieces(
+                                               os.path.join(path, f)))
+                for file in ('INSTALLER', 'METADATA', 'REQUESTED'):
                     record_writer.writerow(record_pieces(
-                                           os.path.join(path, f)))
-            for file in ('INSTALLER', 'METADATA', 'REQUESTED'):
-                record_writer.writerow(record_pieces(
-                                       os.path.join(distinfo_dir, file)))
-            record_writer.writerow([relpath(record_file, sys.prefix)])
-            fp.close()
+                                           os.path.join(distinfo_dir, file)))
+                record_writer.writerow([relpath(record_file, sys.prefix)])
+            finally:
+                fp.close()
 
             fp = open(record_file)
-            record_reader = csv.reader(fp, lineterminator='\n')
-            record_data = {}
-            for row in record_reader:
-                if row == []:
-                    continue
-                path, md5_, size = (row[:] +
-                                    [None for i in range(len(row), 3)])
-                record_data[path] = md5_, size
-            fp.close()
+            try:
+                record_reader = csv.reader(fp, lineterminator='\n')
+                record_data = {}
+                for row in record_reader:
+                    if row == []:
+                        continue
+                    path, md5_, size = (row[:] +
+                                        [None for i in range(len(row), 3)])
+                    record_data[path] = md5_, size
+            finally:
+                fp.close()
             self.records[distinfo_dir] = record_data
 
     def test_instantiation(self):
@@ -207,11 +214,13 @@
 
         for distfile in distinfo_files:
             value = dist.get_distinfo_file(distfile)
-            self.assertIsInstance(value, file)
-            # Is it the correct file?
-            self.assertEqual(value.name,
-                             os.path.join(distinfo_dir, distfile))
-            value.close()
+            try:
+                self.assertIsInstance(value, file)
+                # Is it the correct file?
+                self.assertEqual(value.name,
+                                 os.path.join(distinfo_dir, distfile))
+            finally:
+                value.close()
 
         # Test an absolute path that is part of another distributions dist-info
         other_distinfo_file = os.path.join(
@@ -632,14 +641,16 @@
         resources_path = os.path.join(dist_info, 'RESOURCES')
 
         fp = open(metadata_path, 'w')
-        fp.write(dedent("""\
-            Metadata-Version: 1.2
-            Name: test
-            Version: 0.1
-            Summary: test
-            Author: me
-            """))
-        fp.close()
+        try:
+            fp.write(dedent("""\
+                Metadata-Version: 1.2
+                Name: test
+                Version: 0.1
+                Summary: test
+                Author: me
+                """))
+        finally:
+            fp.close()
         test_path = 'test.cfg'
 
         fd, test_resource_path = tempfile.mkstemp()
@@ -647,12 +658,16 @@
         self.addCleanup(os.remove, test_resource_path)
 
         fp = open(test_resource_path, 'w')
-        fp.write('Config')
-        fp.close()
+        try:
+            fp.write('Config')
+        finally:
+            fp.close()
 
         fp = open(resources_path, 'w')
-        fp.write('%s,%s' % (test_path, test_resource_path))
-        fp.close()
+        try:
+            fp.write('%s,%s' % (test_path, test_resource_path))
+        finally:
+            fp.close()
 
         # Add fake site-packages to sys.path to retrieve fake dist
         self.addCleanup(sys.path.remove, temp_site_packages)
@@ -668,8 +683,10 @@
         self.assertRaises(KeyError, get_file_path, dist_name, 'i-dont-exist')
 
         fp = get_file(dist_name, test_path)
-        self.assertEqual(fp.read(), 'Config')
-        fp.close()
+        try:
+            self.assertEqual(fp.read(), 'Config')
+        finally:
+            fp.close()
         self.assertRaises(KeyError, get_file, dist_name, 'i-dont-exist')
 
 
diff --git a/distutils2/tests/test_depgraph.py b/distutils2/tests/test_depgraph.py
--- a/distutils2/tests/test_depgraph.py
+++ b/distutils2/tests/test_depgraph.py
@@ -4,8 +4,8 @@
 import sys
 from StringIO import StringIO
 
-import distutils2.database
 from distutils2 import depgraph
+from distutils2.database import get_distribution, enable_cache, disable_cache
 
 from distutils2.tests import unittest, support
 from distutils2.tests.support import requires_zlib
@@ -31,13 +31,13 @@
         path = os.path.abspath(path)
         sys.path.insert(0, path)
         self.addCleanup(sys.path.remove, path)
-        self.addCleanup(distutils2.database.enable_cache)
-        distutils2.database.disable_cache()
+        self.addCleanup(enable_cache)
+        disable_cache()
 
     def test_generate_graph(self):
         dists = []
         for name in self.DISTROS_DIST:
-            dist = distutils2.database.get_distribution(name)
+            dist = get_distribution(name)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
@@ -62,7 +62,7 @@
     def test_generate_graph_egg(self):
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG:
-            dist = distutils2.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
@@ -105,7 +105,7 @@
     def test_dependent_dists(self):
         dists = []
         for name in self.DISTROS_DIST:
-            dist = distutils2.database.get_distribution(name)
+            dist = get_distribution(name)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
@@ -124,7 +124,7 @@
     def test_dependent_dists_egg(self):
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG:
-            dist = distutils2.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
@@ -159,7 +159,7 @@
 
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG:
-            dist = distutils2.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
@@ -190,7 +190,7 @@
 
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG:
-            dist = distutils2.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
@@ -251,7 +251,7 @@
 
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG + self.BAD_EGGS:
-            dist = distutils2.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
@@ -274,7 +274,7 @@
     def test_repr(self):
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG + self.BAD_EGGS:
-            dist = distutils2.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
diff --git a/distutils2/tests/test_dist.py b/distutils2/tests/test_dist.py
--- a/distutils2/tests/test_dist.py
+++ b/distutils2/tests/test_dist.py
@@ -1,7 +1,7 @@
 """Tests for distutils2.dist."""
-import codecs
 import os
 import sys
+import codecs
 import logging
 import textwrap
 from distutils2._backport import sysconfig
@@ -16,7 +16,7 @@
 from distutils2.tests import captured_stdout
 from distutils2.tests import support, unittest
 from distutils2.tests.support import create_distribution
-from distutils2.tests.support import unload, TESTFN
+from distutils2.tests.support import unload
 
 
 class test_dist(Command):
@@ -50,17 +50,23 @@
         sys.argv[:] = self.argv[1]
         super(DistributionTestCase, self).tearDown()
 
+    @unittest.skip('needs to be updated')
     def test_debug_mode(self):
-        self.addCleanup(os.unlink, TESTFN)
-        f = open(TESTFN, "w")
-        f.write("[global]\n")
-        f.write("command_packages = foo.bar, splat")
-        f.close()
+        tmpdir = self.mkdtemp()
+        setupcfg = os.path.join(tmpdir, 'setup.cfg')
+        f = open(setupcfg, "w")
+        try:
+            f.write("[global]\n")
+            f.write("command_packages = foo.bar, splat")
+        finally:
+            f.close()
 
-        files = [TESTFN]
+        files = [setupcfg]
         sys.argv.append("build")
         __, stdout = captured_stdout(create_distribution, files)
         self.assertEqual(stdout, '')
+        # XXX debug mode does not exist anymore, test logging levels in this
+        # test instead
         distutils2.dist.DEBUG = True
         try:
             __, stdout = captured_stdout(create_distribution, files)
@@ -145,7 +151,6 @@
         self.assertIn('owner', dist.get_option_dict('sdist'))
 
     def test_finalize_options(self):
-
         attrs = {'keywords': 'one,two',
                  'platform': 'one,two'}
 
@@ -165,8 +170,10 @@
             user_filename = os.path.join(temp_home, "pydistutils.cfg")
 
         f = open(user_filename, 'w')
-        f.write('[distutils2]\n')
-        f.close()
+        try:
+            f.write('[distutils2]\n')
+        finally:
+            f.close()
 
         def _expander(path):
             return temp_home
@@ -180,10 +187,8 @@
             d = distutils2.dist.Distribution(attrs={'script_args':
                                                    ['--no-user-cfg']})
             files = d.find_config_files()
-        except:
+        finally:
             os.path.expanduser = old_expander
-            raise
-        os.path.expanduser = old_expander
 
         # make sure --no-user-cfg disables the user cfg file
         self.assertEqual((len(all_files) - 1), len(files))
diff --git a/distutils2/tests/test_install.py b/distutils2/tests/test_install.py
--- a/distutils2/tests/test_install.py
+++ b/distutils2/tests/test_install.py
@@ -7,7 +7,7 @@
 from distutils2.pypi.xmlrpc import Client
 from distutils2.metadata import Metadata
 from distutils2.tests.support import (LoggingCatcher, TempdirManager, unittest,
-                                     fake_dec)
+                                      fake_dec)
 from distutils2._backport.sysconfig import is_python_build
 
 try:
diff --git a/distutils2/tests/test_manifest.py b/distutils2/tests/test_manifest.py
--- a/distutils2/tests/test_manifest.py
+++ b/distutils2/tests/test_manifest.py
@@ -38,8 +38,10 @@
         tmpdir = self.mkdtemp()
         MANIFEST = os.path.join(tmpdir, 'MANIFEST.in')
         f = open(MANIFEST, 'w')
-        f.write(_MANIFEST)
-        f.close()
+        try:
+            f.write(_MANIFEST)
+        finally:
+            f.close()
 
         manifest = Manifest()
         manifest.read_template(MANIFEST)
@@ -53,8 +55,10 @@
 
         # manifest also accepts file-like objects
         f = open(MANIFEST)
-        manifest.read_template(f)
-        f.close()
+        try:
+            manifest.read_template(f)
+        finally:
+            f.close()
 
         # the manifest should have been read and 3 warnings issued
         # (we didn't provide the files)
diff --git a/distutils2/tests/test_markers.py b/distutils2/tests/test_markers.py
--- a/distutils2/tests/test_markers.py
+++ b/distutils2/tests/test_markers.py
@@ -19,7 +19,7 @@
         platform_machine = platform.machine()
         try:
             platform_python_implementation = platform.python_implementation()
-        except AttributeError: #assume CPython
+        except AttributeError:  # FIXME import from compat
             platform_python_implementation = 'CPython'
 
         self.assertTrue(interpret("sys.platform == '%s'" % sys_platform))
diff --git a/distutils2/tests/test_metadata.py b/distutils2/tests/test_metadata.py
--- a/distutils2/tests/test_metadata.py
+++ b/distutils2/tests/test_metadata.py
@@ -1,7 +1,7 @@
 """Tests for distutils2.metadata."""
-import codecs
 import os
 import sys
+import codecs
 import logging
 from StringIO import StringIO
 
@@ -19,8 +19,10 @@
     def test_instantiation(self):
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
         f = codecs.open(PKG_INFO, 'r', encoding='utf-8')
-        contents = f.read()
-        f.close()
+        try:
+            contents = f.read()
+        finally:
+            f.close()
 
         fp = StringIO(contents)
 
@@ -61,8 +63,10 @@
         # see if we can be platform-aware
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
         f = codecs.open(PKG_INFO, 'r', encoding='utf-8')
-        content = f.read() % sys.platform
-        f.close()
+        try:
+            content = f.read() % sys.platform
+        finally:
+            f.close()
         metadata = Metadata(platform_dependent=True)
 
         metadata.read_file(StringIO(content))
@@ -74,24 +78,27 @@
 
         # test with context
         context = {'sys.platform': 'okook'}
-        metadata = Metadata(platform_dependent=True,
-                                        execution_context=context)
+        metadata = Metadata(platform_dependent=True, execution_context=context)
         metadata.read_file(StringIO(content))
         self.assertEqual(metadata['Requires-Dist'], ['foo'])
 
     def test_description(self):
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
         f = codecs.open(PKG_INFO, 'r', encoding='utf-8')
-        content = f.read() % sys.platform
-        f.close()
+        try:
+            content = f.read() % sys.platform
+        finally:
+            f.close()
         metadata = Metadata()
         metadata.read_file(StringIO(content))
 
         # see if we can read the description now
         DESC = os.path.join(os.path.dirname(__file__), 'LONG_DESC.txt')
         f = open(DESC)
-        wanted = f.read()
-        f.close()
+        try:
+            wanted = f.read()
+        finally:
+            f.close()
         self.assertEqual(wanted, metadata['Description'])
 
         # save the file somewhere and make sure we can read it back
@@ -104,8 +111,10 @@
     def test_mapping_api(self):
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
         f = codecs.open(PKG_INFO, 'r', encoding='utf-8')
-        content = f.read() % sys.platform
-        f.close()
+        try:
+            content = f.read() % sys.platform
+        finally:
+            f.close()
         metadata = Metadata(fileobj=StringIO(content))
         self.assertIn('Version', metadata.keys())
         self.assertIn('0.5', metadata.values())
@@ -138,16 +147,20 @@
         PKG_INFO = os.path.join(os.path.dirname(__file__),
                                 'SETUPTOOLS-PKG-INFO')
         f = codecs.open(PKG_INFO, 'r', encoding='utf-8')
-        content = f.read()
-        f.close()
+        try:
+            content = f.read()
+        finally:
+            f.close()
         metadata.read_file(StringIO(content))
         self.assertEqual(metadata['Metadata-Version'], '1.0')
 
         PKG_INFO = os.path.join(os.path.dirname(__file__),
                                 'SETUPTOOLS-PKG-INFO2')
         f = codecs.open(PKG_INFO, 'r', encoding='utf-8')
-        content = f.read()
-        f.close()
+        try:
+            content = f.read()
+        finally:
+            f.close()
 
         metadata.read_file(StringIO(content))
         self.assertEqual(metadata['Metadata-Version'], '1.1')
diff --git a/distutils2/tests/test_mixin2to3.py b/distutils2/tests/test_mixin2to3.py
--- a/distutils2/tests/test_mixin2to3.py
+++ b/distutils2/tests/test_mixin2to3.py
@@ -15,16 +15,20 @@
         code = "print 'test'"
 
         fp = self.mktempfile()
-        fp.write(code)
-        fp.close()
+        try:
+            fp.write(code)
+        finally:
+            fp.close()
 
         mixin2to3 = Mixin2to3()
         mixin2to3._run_2to3([fp.name])
         expected = "print('test')"
 
         fp = open(fp.name)
-        converted = fp.read()
-        fp.close()
+        try:
+            converted = fp.read()
+        finally:
+            fp.close()
 
         self.assertEqual(expected, converted)
 
@@ -41,8 +45,10 @@
             """''')
 
         fp = self.mktempfile()
-        fp.write(doctest)
-        fp.close()
+        try:
+            fp.write(doctest)
+        finally:
+            fp.close()
 
         mixin2to3 = Mixin2to3()
         mixin2to3._run_2to3([fp.name])
@@ -56,8 +62,10 @@
             """\n''')
 
         fp = open(fp.name)
-        converted = fp.read()
-        fp.close()
+        try:
+            converted = fp.read()
+        finally:
+            fp.close()
 
         self.assertEqual(expected, converted)
 
@@ -67,8 +75,10 @@
         code = 'type(x) is not T'
 
         fp = self.mktempfile()
-        fp.write(code)
-        fp.close()
+        try:
+            fp.write(code)
+        finally:
+            fp.close()
 
         mixin2to3 = Mixin2to3()
         mixin2to3._run_2to3(files=[fp.name], doctests=[fp.name],
@@ -77,8 +87,10 @@
         expected = 'not isinstance(x, T)'
 
         fp = open(fp.name)
-        converted = fp.read()
-        fp.close()
+        try:
+            converted = fp.read()
+        finally:
+            fp.close()
 
         self.assertEqual(expected, converted)
 
diff --git a/distutils2/tests/test_msvc9compiler.py b/distutils2/tests/test_msvc9compiler.py
--- a/distutils2/tests/test_msvc9compiler.py
+++ b/distutils2/tests/test_msvc9compiler.py
@@ -119,17 +119,21 @@
         tempdir = self.mkdtemp()
         manifest = os.path.join(tempdir, 'manifest')
         f = open(manifest, 'w')
-        f.write(_MANIFEST)
-        f.close()
+        try:
+            f.write(_MANIFEST)
+        finally:
+            f.close()
 
         compiler = MSVCCompiler()
         compiler._remove_visual_c_ref(manifest)
 
         # see what we got
         f = open(manifest)
-        # removing trailing spaces
-        content = '\n'.join(line.rstrip() for line in f.readlines())
-        f.close()
+        try:
+            # removing trailing spaces
+            content = '\n'.join(line.rstrip() for line in f.readlines())
+        finally:
+            f.close()
 
         # makes sure the manifest was properly cleaned
         self.assertEqual(content, _CLEANED_MANIFEST)
diff --git a/distutils2/tests/test_pypi_server.py b/distutils2/tests/test_pypi_server.py
--- a/distutils2/tests/test_pypi_server.py
+++ b/distutils2/tests/test_pypi_server.py
@@ -1,11 +1,10 @@
 """Tests for distutils2.command.bdist."""
-import sys
-
 import urllib2
 
 try:
     import threading
-    from distutils2.tests.pypi_server import PyPIServer, PYPI_DEFAULT_STATIC_PATH
+    from distutils2.tests.pypi_server import (
+        PyPIServer, PYPI_DEFAULT_STATIC_PATH)
 except ImportError:
     threading = None
     PyPIServer = None
@@ -13,6 +12,7 @@
 
 from distutils2.tests import unittest
 
+
 class PyPIServerTest(unittest.TestCase):
 
     def test_records_requests(self):
@@ -34,12 +34,11 @@
             handler, request_data = server.requests[-1]
             self.assertIn(data, request_data)
             self.assertIn("x-test-header", handler.headers)
-            self.assertEqual(handler.headers["x-test-header"], "Mister Iceberg")
+            self.assertEqual(handler.headers["x-test-header"],
+                             "Mister Iceberg")
 
-        except:
+        finally:
             server.stop()
-            raise
-        server.stop()
 
     def test_serve_static_content(self):
         # PYPI Mocked server can serve static content from disk.
@@ -51,11 +50,12 @@
             url = server.full_address + url_path
             request = urllib2.Request(url)
             response = urllib2.urlopen(request)
-            fp = open(PYPI_DEFAULT_STATIC_PATH + "/test_pypi_server"
-                    + url_path)
-            content = fp.read()
-            fp.close()
-            return response.read().decode() == content
+            file = open(PYPI_DEFAULT_STATIC_PATH + "/test_pypi_server"
+                      + url_path)
+            try:
+                return response.read().decode() == file.read()
+            finally:
+                file.close()
 
         server = PyPIServer(static_uri_paths=["simple", "external"],
             static_filesystem_paths=["test_pypi_server"])
@@ -66,23 +66,24 @@
             request = urllib2.Request(url)
             try:
                 urllib2.urlopen(request)
-            except urllib2.HTTPError:
-                self.assertEqual(sys.exc_info()[1].code, 404)
+            except urllib2.HTTPError, e:
+                self.assertEqual(e.code, 404)
 
             # now try serving a content that do exists
             self.assertTrue(uses_local_files_for(server, "/simple/index.html"))
 
             # and another one in another root path
-            self.assertTrue(uses_local_files_for(server, "/external/index.html"))
+            self.assertTrue(uses_local_files_for(server,
+                                                 "/external/index.html"))
 
-        except:
+        finally:
             server.stop()
-            raise
-        server.stop()
+
 
 PyPIServerTest = unittest.skipIf(threading is None, "Needs threading")(
         PyPIServerTest)
 
+
 def test_suite():
     return unittest.makeSuite(PyPIServerTest)
 
diff --git a/distutils2/tests/test_pypi_simple.py b/distutils2/tests/test_pypi_simple.py
--- a/distutils2/tests/test_pypi_simple.py
+++ b/distutils2/tests/test_pypi_simple.py
@@ -9,7 +9,7 @@
 
 from distutils2.tests import unittest
 from distutils2.tests.support import (TempdirManager, LoggingCatcher,
-                                     fake_dec)
+                                      fake_dec)
 
 try:
     import thread as _thread
@@ -43,8 +43,8 @@
         url = 'http://127.0.0.1:0/nonesuch/test_simple'
         try:
             v = crawler._open_url(url)
-        except Exception:
-            self.assertIn(url, str(sys.exc_info()[1]))
+        except Exception, v:
+            self.assertIn(url, str(v))
         else:
             v.close()
             self.assertIsInstance(v, urllib2.HTTPError)
@@ -57,8 +57,8 @@
                'inquant.contentmirror.plone/trunk')
         try:
             v = crawler._open_url(url)
-        except Exception:
-            self.assertIn(url, str(sys.exc_info()[1]))
+        except Exception, v:
+            self.assertIn(url, str(v))
         else:
             v.close()
             self.assertIsInstance(v, urllib2.HTTPError)
@@ -70,22 +70,23 @@
         urllib2.urlopen = _urlopen
         url = 'http://example.org'
         try:
-            v = crawler._open_url(url)
-        except:
+            try:
+                v = crawler._open_url(url)
+            except Exception, v:
+                self.assertIn('line', str(v))
+            else:
+                v.close()
+                # TODO use self.assertRaises
+                raise AssertionError('Should have raise here!')
+        finally:
             urllib2.urlopen = old_urlopen
-            self.assertIn('line', str(sys.exc_info()[1]))
-        else:
-            v.close()
-            # TODO use self.assertRaises
-            raise AssertionError('Should have raise here!')
-        urllib2.urlopen = old_urlopen
 
         # issue 20
         url = 'http://http://svn.pythonpaste.org/Paste/wphp/trunk'
         try:
             crawler._open_url(url)
-        except Exception:
-            self.assertIn('nonnumeric port', str(sys.exc_info()[1]))
+        except Exception, v:
+            self.assertIn('nonnumeric port', str(v))
 
         # issue #160
         url = server.full_address
@@ -246,12 +247,9 @@
 
             # this should not raise a timeout
             self.assertEqual(4, len(crawler.get_releases("foo")))
-        except:
+        finally:
             mirror.stop()
             server.stop()
-            raise
-        mirror.stop()
-        server.stop()
 
     def test_simple_link_matcher(self):
         # Test that the simple link matcher finds the right links"""
diff --git a/distutils2/tests/test_pypi_xmlrpc.py b/distutils2/tests/test_pypi_xmlrpc.py
--- a/distutils2/tests/test_pypi_xmlrpc.py
+++ b/distutils2/tests/test_pypi_xmlrpc.py
@@ -90,10 +90,11 @@
         self.assertEqual(['Foo'], release.metadata['requires_external'])
         self.assertEqual(['FooFoo'], release.metadata['obsoletes_dist'])
 
-#Compatibility Python pre-2.6
+
 TestXMLRPCClient = unittest.skipIf(threading is None, "Needs threading")(
         TestXMLRPCClient)
 
+
 def test_suite():
     suite = unittest.TestSuite()
     suite.addTest(unittest.makeSuite(TestXMLRPCClient))
diff --git a/distutils2/tests/test_run.py b/distutils2/tests/test_run.py
--- a/distutils2/tests/test_run.py
+++ b/distutils2/tests/test_run.py
@@ -7,12 +7,10 @@
 
 from distutils2 import install
 from distutils2.tests import unittest, support
-from distutils2.tests.support import TESTFN
 from distutils2.run import main
 
 # setup script that uses __file__
 setup_using___file__ = """\
-
 __file__
 
 from distutils2.run import setup
@@ -20,7 +18,6 @@
 """
 
 setup_prints_cwd = """\
-
 import os
 print os.getcwd()
 
@@ -29,34 +26,20 @@
 """
 
 
-class CoreTestCase(support.TempdirManager, support.LoggingCatcher,
-                   unittest.TestCase):
+class RunTestCase(support.TempdirManager,
+                  support.LoggingCatcher,
+                  unittest.TestCase):
 
     def setUp(self):
-        super(CoreTestCase, self).setUp()
+        super(RunTestCase, self).setUp()
         self.old_stdout = sys.stdout
-        self.cleanup_testfn()
         self.old_argv = sys.argv, sys.argv[:]
 
     def tearDown(self):
         sys.stdout = self.old_stdout
-        self.cleanup_testfn()
         sys.argv = self.old_argv[0]
         sys.argv[:] = self.old_argv[1]
-        super(CoreTestCase, self).tearDown()
-
-    def cleanup_testfn(self):
-        path = TESTFN
-        if os.path.isfile(path):
-            os.remove(path)
-        elif os.path.isdir(path):
-            shutil.rmtree(path)
-
-    def write_setup(self, text, path=TESTFN):
-        fp = open(path, "w")
-        fp.write(text)
-        fp.close()
-        return path
+        super(RunTestCase, self).tearDown()
 
     # TODO restore the tests removed six months ago and port them to pysetup
 
@@ -80,7 +63,7 @@
 
 
 def test_suite():
-    return unittest.makeSuite(CoreTestCase)
+    return unittest.makeSuite(RunTestCase)
 
 if __name__ == "__main__":
     unittest.main(defaultTest="test_suite")
diff --git a/distutils2/tests/test_util.py b/distutils2/tests/test_util.py
--- a/distutils2/tests/test_util.py
+++ b/distutils2/tests/test_util.py
@@ -15,10 +15,10 @@
 from distutils2 import util
 from distutils2.dist import Distribution
 from distutils2.util import (
-    convert_path, change_root, split_quoted, strtobool, rfc822_escape,
+    convert_path, change_root, split_quoted, strtobool, rfc822_escape, run_2to3,
     get_compiler_versions, _MAC_OS_X_LD_VERSION, byte_compile, find_packages,
     spawn, get_pypirc_path, generate_pypirc, read_pypirc, resolve_name, iglob,
-    RICH_GLOB, egginfo_to_distinfo, is_setuptools, is_distutils, is_distutils2,
+    RICH_GLOB, egginfo_to_distinfo, is_setuptools, is_distutils, is_packaging,
     get_install_method, cfg_to_args, encode_multipart)
 
 
@@ -381,7 +381,7 @@
 
         res = find_packages([root], ['pkg1.pkg2'])
         self.assertEqual(set(res), set(['pkg1', 'pkg5', 'pkg1.pkg3',
-                                         'pkg1.pkg3.pkg6']))
+                                        'pkg1.pkg3.pkg6']))
 
     def test_resolve_name(self):
         self.assertIs(str, resolve_name('__builtin__.str'))
@@ -423,7 +423,6 @@
         file_handle.write(content)
         file_handle.flush()
         file_handle.seek(0)
-        from distutils2.util import run_2to3
         run_2to3([file_name])
         new_content = "".join(file_handle.read())
         file_handle.close()
@@ -439,7 +438,6 @@
         file_handle.write(content)
         file_handle.flush()
         file_handle.seek(0)
-        from distutils2.util import run_2to3
         run_2to3([file_name], doctests_only=True)
         new_content = "".join(file_handle.readlines())
         file_handle.close()
@@ -457,24 +455,24 @@
         if os.name == 'posix':
             exe = os.path.join(tmpdir, 'foo.sh')
             self.write_file(exe, '#!/bin/sh\nexit 1')
-            os.chmod(exe, 00777)
+            os.chmod(exe, 0777)
         else:
             exe = os.path.join(tmpdir, 'foo.bat')
             self.write_file(exe, 'exit 1')
 
-        os.chmod(exe, 00777)
+        os.chmod(exe, 0777)
         self.assertRaises(PackagingExecError, spawn, [exe])
 
         # now something that works
         if os.name == 'posix':
             exe = os.path.join(tmpdir, 'foo.sh')
             self.write_file(exe, '#!/bin/sh\nexit 0')
-            os.chmod(exe, 00777)
+            os.chmod(exe, 0777)
         else:
             exe = os.path.join(tmpdir, 'foo.bat')
             self.write_file(exe, 'exit 0')
 
-        os.chmod(exe, 00777)
+        os.chmod(exe, 0777)
         spawn([exe])  # should work without any error
 
     def test_server_registration(self):
@@ -507,8 +505,10 @@
         generate_pypirc('tarek', 'xxx')
         self.assertTrue(os.path.exists(rc))
         f = open(rc)
-        content = f.read()
-        f.close()
+        try:
+            content = f.read()
+        finally:
+            f.close()
         self.assertEqual(content, WANTED)
 
     def test_cfg_to_args(self):
@@ -795,16 +795,20 @@
         for f in files:
             path = os.path.join(tempdir, f)
             _f = open(path, 'w')
-            _f.write(f)
-            _f.close()
+            try:
+                _f.write(f)
+            finally:
+                _f.close()
             file_paths.append(path)
 
         record_file = open(record_file_path, 'w')
-        for fpath in file_paths:
-            record_file.write(fpath + '\n')
-        for dpath in dir_paths:
-            record_file.write(dpath + '\n')
-        record_file.close()
+        try:
+            for fpath in file_paths:
+                record_file.write(fpath + '\n')
+            for dpath in dir_paths:
+                record_file.write(dpath + '\n')
+        finally:
+            record_file.close()
         return (tempdir, record_file_path)
 
 
@@ -819,7 +823,7 @@
     def test_empty_package_is_not_based_on_anything(self):
         self.assertFalse(is_setuptools(self._empty_dir))
         self.assertFalse(is_distutils(self._empty_dir))
-        self.assertFalse(is_distutils2(self._empty_dir))
+        self.assertFalse(is_packaging(self._empty_dir))
 
     def test_setup_py_importing_setuptools_is_setuptools_based(self):
         self.assertTrue(is_setuptools(self._setuptools_setup_py_pkg()))
@@ -846,13 +850,13 @@
         self.assertFalse(is_distutils(self._random_setup_py_pkg()))
 
     def test_setup_cfg_with_no_metadata_section_is_not_distutils2_based(self):
-        self.assertFalse(is_distutils2(self._setup_cfg_with_no_metadata_pkg()))
+        self.assertFalse(is_packaging(self._setup_cfg_with_no_metadata_pkg()))
 
-    def test_setup_cfg_with_valid_metadata_section_is_distutils2_based(self):
-        self.assertTrue(is_distutils2(self._valid_setup_cfg_pkg()))
+    def test_setup_cfg_with_valid_metadata_section_is_packaging_based(self):
+        self.assertTrue(is_packaging(self._valid_setup_cfg_pkg()))
 
     def test_setup_cfg_and_invalid_setup_cfg_is_not_distutils2_based(self):
-        self.assertFalse(is_distutils2(self._invalid_setup_cfg_pkg()))
+        self.assertFalse(is_packaging(self._invalid_setup_cfg_pkg()))
 
     def test_get_install_method_with_setuptools_pkg(self):
         path = self._setuptools_setup_py_pkg()
@@ -908,13 +912,13 @@
         expected = ['setup.py file found.', 'PKG-INFO file found.']
         self.assertEqual(expected, self.get_logs(logging.DEBUG))
 
-    def test_is_distutils2_logs_setup_cfg_found(self):
-        is_distutils2(self._valid_setup_cfg_pkg())
+    def test_is_packaging_logs_setup_cfg_found(self):
+        is_packaging(self._valid_setup_cfg_pkg())
         expected = ['setup.cfg file found.']
         self.assertEqual(expected, self.get_logs(logging.DEBUG))
 
-    def test_is_distutils2_logs_setup_cfg_not_found(self):
-        is_distutils2(self._empty_dir)
+    def test_is_packaging_logs_setup_cfg_not_found(self):
+        is_packaging(self._empty_dir)
         expected = ['No setup.cfg file found.']
         self.assertEqual(expected, self.get_logs(logging.DEBUG))
 

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Remove_mentions_of_abifl?=
 =?utf8?q?ags_=28never_available_in_2=2Ex=29?=
Message-ID: <E1R5deJ-0007lL-D5@dinsdale.python.org>

http://hg.python.org/distutils2/rev/71626e848b24
changeset:   1153:71626e848b24
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Sep 18 23:30:28 2011 +0200
summary:
  Remove mentions of abiflags (never available in 2.x)

files:
  distutils2/_backport/sysconfig.cfg |   4 ++--
  distutils2/_backport/sysconfig.py  |  10 +---------
  distutils2/command/build_ext.py    |   2 +-
  3 files changed, 4 insertions(+), 12 deletions(-)


diff --git a/distutils2/_backport/sysconfig.cfg b/distutils2/_backport/sysconfig.cfg
--- a/distutils2/_backport/sysconfig.cfg
+++ b/distutils2/_backport/sysconfig.cfg
@@ -40,8 +40,8 @@
 platstdlib = {platbase}/lib/python{py_version_short}
 purelib = {base}/lib/python{py_version_short}/site-packages
 platlib = {platbase}/lib/python{py_version_short}/site-packages
-include = {base}/include/python{py_version_short}{abiflags}
-platinclude = {platbase}/include/python{py_version_short}{abiflags}
+include = {base}/include/python{py_version_short}
+platinclude = {platbase}/include/python{py_version_short}
 data = {base}
 
 [posix_home]
diff --git a/distutils2/_backport/sysconfig.py b/distutils2/_backport/sysconfig.py
--- a/distutils2/_backport/sysconfig.py
+++ b/distutils2/_backport/sysconfig.py
@@ -316,10 +316,7 @@
     """Return the path of the Makefile."""
     if _PYTHON_BUILD:
         return os.path.join(_PROJECT_BASE, "Makefile")
-    if hasattr(sys, 'abiflags'):
-        config_dir_name = 'config-%s%s' % (_PY_VERSION_SHORT, sys.abiflags)
-    else:
-        config_dir_name = 'config'
+    config_dir_name = 'config'
     return os.path.join(get_path('stdlib'), config_dir_name, 'Makefile')
 
 
@@ -468,11 +465,6 @@
         _CONFIG_VARS['base'] = _PREFIX
         _CONFIG_VARS['platbase'] = _EXEC_PREFIX
         _CONFIG_VARS['projectbase'] = _PROJECT_BASE
-        try:
-            _CONFIG_VARS['abiflags'] = sys.abiflags
-        except AttributeError:
-            # sys.abiflags may not be defined on all platforms.
-            _CONFIG_VARS['abiflags'] = ''
 
         if os.name in ('nt', 'os2'):
             _init_non_posix(_CONFIG_VARS)
diff --git a/distutils2/command/build_ext.py b/distutils2/command/build_ext.py
--- a/distutils2/command/build_ext.py
+++ b/distutils2/command/build_ext.py
@@ -650,7 +650,7 @@
 
         else:
             if sysconfig.get_config_var('Py_ENABLE_SHARED'):
-                template = 'python%d.%d' + sys.abiflags
+                template = 'python%d.%d'
                 pythonlib = template % sys.version_info[:2]
                 return ext.libraries + [pythonlib]
             else:

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Provide_replacement_for_?=
 =?utf8?q?platform=2Epython=5Fimplementation_for_2=2E4_and_2=2E5?=
Message-ID: <E1R5deJ-0007lO-Do@dinsdale.python.org>

http://hg.python.org/distutils2/rev/bd7bfeab5db1
changeset:   1154:bd7bfeab5db1
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Sep 18 23:31:36 2011 +0200
summary:
  Provide replacement for platform.python_implementation for 2.4 and 2.5

files:
  distutils2/compat.py             |  13 +++++++++++++
  distutils2/markers.py            |  13 +++++--------
  distutils2/tests/test_markers.py |   6 ++----
  3 files changed, 20 insertions(+), 12 deletions(-)


diff --git a/distutils2/compat.py b/distutils2/compat.py
--- a/distutils2/compat.py
+++ b/distutils2/compat.py
@@ -4,6 +4,7 @@
 Python 3.2, for internal use only.  Whole modules are in _backport.
 """
 
+import os
 import re
 import sys
 import codecs
@@ -178,3 +179,15 @@
         def wrapped(func):
             return func
         return wrapped
+
+try:
+    from platform import python_implementation
+except ImportError:
+    def python_implementation():
+        if 'PyPy' in sys.version:
+            return 'PyPy'
+        if os.name == 'java':
+            return 'Jython'
+        if sys.version.startswith('IronPython'):
+            return 'IronPython'
+        return 'CPython'
diff --git a/distutils2/markers.py b/distutils2/markers.py
--- a/distutils2/markers.py
+++ b/distutils2/markers.py
@@ -1,16 +1,12 @@
 """Parser for the environment markers micro-language defined in PEP 345."""
 
+import os
 import sys
 import platform
-import os
-
 from tokenize import generate_tokens, NAME, OP, STRING, ENDMARKER
 from StringIO import StringIO as BytesIO
-try:
-    python_implementation = platform.python_implementation()
-except AttributeError:
-    # FIXME import from compat
-    python_implementation = 'CPython'
+
+from distutils2.compat import python_implementation
 
 __all__ = ['interpret']
 
@@ -37,7 +33,8 @@
          'os.name': os.name,
          'platform.version': platform.version(),
          'platform.machine': platform.machine(),
-         'platform.python_implementation': python_implementation}
+         'platform.python_implementation': python_implementation(),
+        }
 
 
 class _Operation(object):
diff --git a/distutils2/tests/test_markers.py b/distutils2/tests/test_markers.py
--- a/distutils2/tests/test_markers.py
+++ b/distutils2/tests/test_markers.py
@@ -2,6 +2,7 @@
 import os
 import sys
 import platform
+from distutils2.compat import python_implementation
 from distutils2.markers import interpret
 
 from distutils2.tests import unittest
@@ -17,10 +18,7 @@
         os_name = os.name
         platform_version = platform.version()
         platform_machine = platform.machine()
-        try:
-            platform_python_implementation = platform.python_implementation()
-        except AttributeError:  # FIXME import from compat
-            platform_python_implementation = 'CPython'
+        platform_python_implementation = python_implementation()
 
         self.assertTrue(interpret("sys.platform == '%s'" % sys_platform))
         self.assertTrue(interpret(

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Remove_some_duplication_?=
	=?utf8?q?in_tests_subpackage=2E?=
Message-ID: <E1R5deJ-0007lS-Ex@dinsdale.python.org>

http://hg.python.org/distutils2/rev/3455b53fc826
changeset:   1155:3455b53fc826
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Sep 18 23:56:21 2011 +0200
summary:
  Remove some duplication in tests subpackage.

This commit adds some missing names found by pyflakes/pylint; the test
suite passed before, but we might as well have correct code.  Some other
functions have been removed; running the test suite through
d2.tests.__main__ now requires Python 2.7.

There?s still quite a bit of unnecessary duplication between __init__
and support, but I won?t fix that now.

files:
  distutils2/tests/__init__.py |    1 +
  distutils2/tests/__main__.py |    3 +-
  distutils2/tests/support.py  |  149 ++++------------------
  3 files changed, 32 insertions(+), 121 deletions(-)


diff --git a/distutils2/tests/__init__.py b/distutils2/tests/__init__.py
--- a/distutils2/tests/__init__.py
+++ b/distutils2/tests/__init__.py
@@ -22,6 +22,7 @@
 here = os.path.dirname(__file__) or os.curdir
 verbose = 1
 
+
 def test_suite():
     suite = unittest.TestSuite()
     for fn in os.listdir(here):
diff --git a/distutils2/tests/__main__.py b/distutils2/tests/__main__.py
--- a/distutils2/tests/__main__.py
+++ b/distutils2/tests/__main__.py
@@ -4,8 +4,9 @@
 
 import os
 import sys
+from test.test_support import reap_children, reap_threads, run_unittest
+
 from distutils2.tests import unittest
-from distutils2.tests.support import reap_children, reap_threads, run_unittest
 
 
 @reap_threads
diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -33,7 +33,9 @@
 """
 
 import os
+import re
 import sys
+import errno
 import codecs
 import shutil
 import logging
@@ -42,11 +44,6 @@
 import weakref
 import tempfile
 try:
-    import _thread, threading
-except ImportError:
-    _thread = None
-    threading = None
-try:
     import zlib
 except ImportError:
     zlib = None
@@ -54,9 +51,17 @@
 from distutils2.dist import Distribution
 from distutils2.tests import unittest
 
-__all__ = ['LoggingCatcher', 'TempdirManager', 'EnvironRestorer',
-           'DummyCommand', 'unittest', 'create_distribution',
-           'skip_unless_symlink', 'requires_zlib']
+# define __all__ to make pydoc more useful
+__all__ = [
+    # TestCase mixins
+    'LoggingCatcher', 'TempdirManager', 'EnvironRestorer',
+    # mocks
+    'DummyCommand', 'TestDistribution',
+    # misc. functions and decorators
+    'fake_dec', 'create_distribution',
+    # imported from this module for backport purposes
+    'unittest', 'requires_zlib', 'skip_unless_symlink',
+]
 
 
 logger = logging.getLogger('distutils2')
@@ -297,7 +302,9 @@
     from test.test_support import skip_unless_symlink
 except ImportError:
     skip_unless_symlink = unittest.skip(
-        'requires test.support.skip_unless_symlink')
+        'requires test.test_support.skip_unless_symlink')
+
+requires_zlib = unittest.skipUnless(zlib, 'requires zlib')
 
 
 def unlink(filename):
@@ -308,113 +315,18 @@
         if error.errno not in (errno.ENOENT, errno.ENOTDIR):
             raise
 
-def _filter_suite(suite, pred):
-    """Recursively filter test cases in a suite based on a predicate."""
-    newtests = []
-    for test in suite._tests:
-        if isinstance(test, unittest.TestSuite):
-            _filter_suite(test, pred)
-            newtests.append(test)
-        else:
-            if pred(test):
-                newtests.append(test)
-    suite._tests = newtests
 
-class Error(Exception):
-    """Base class for regression test exceptions."""
+def strip_python_stderr(stderr):
+    """Strip the stderr of a Python process from potential debug output
+    emitted by the interpreter.
 
-class TestFailed(Error):
-    """Test failed."""
+    This will typically be run on the result of the communicate() method
+    of a subprocess.Popen object.
+    """
+    stderr = re.sub(r"\[\d+ refs\]\r?\n?$", "", stderr).strip()
+    return stderr
 
 
-verbose = True
-failfast = False
-
-def _run_suite(suite):
-    """Run tests from a unittest.TestSuite-derived class."""
-    if verbose:
-        runner = unittest.TextTestRunner(sys.stdout, verbosity=2,
-                                         failfast=failfast)
-    else:
-        runner = BasicTestRunner()
-
-    result = runner.run(suite)
-    if not result.wasSuccessful():
-        if len(result.errors) == 1 and not result.failures:
-            err = result.errors[0][1]
-        elif len(result.failures) == 1 and not result.errors:
-            err = result.failures[0][1]
-        else:
-            err = "multiple errors occurred"
-            if not verbose: err += "; run in verbose mode for details"
-        raise TestFailed(err)
-
-match_tests = None
-
-def run_unittest(*classes):
-    """Run tests from unittest.TestCase-derived classes."""
-    valid_types = (unittest.TestSuite, unittest.TestCase)
-    suite = unittest.TestSuite()
-    for cls in classes:
-        if isinstance(cls, basestring):
-            if cls in sys.modules:
-                suite.addTest(unittest.findTestCases(sys.modules[cls]))
-            else:
-                raise ValueError("str arguments must be keys in sys.modules")
-        elif isinstance(cls, valid_types):
-            suite.addTest(cls)
-        else:
-            suite.addTest(unittest.makeSuite(cls))
-    def case_pred(test):
-        if match_tests is None:
-            return True
-        for name in test.id().split("."):
-            if fnmatch.fnmatchcase(name, match_tests):
-                return True
-        return False
-    _filter_suite(suite, case_pred)
-    _run_suite(suite)
-
-
-def reap_threads(func):
-    """Use this function when threads are being used.  This will
-    ensure that the threads are cleaned up even when the test fails.
-    If threading is unavailable this function does nothing.
-    """
-    if not _thread:
-        return func
-
-    @wraps(func)
-    def decorator(*args):
-        key = threading_setup()
-        try:
-            return func(*args)
-        finally:
-            threading_cleanup(*key)
-    return decorator
-
-def reap_children():
-    """Use this function at the end of test_main() whenever sub-processes
-    are started.  This will help ensure that no extra children (zombies)
-    stick around to hog resources and create problems when looking
-    for refleaks.
-    """
-
-    # Reap all our dead child processes so we don't leave zombies around.
-    # These hog resources and might be causing some of the buildbots to die.
-    if hasattr(os, 'waitpid'):
-        any_process = -1
-        while True:
-            try:
-                # This will raise an exception on Windows.  That's ok.
-                pid, status = os.waitpid(any_process, os.WNOHANG)
-                if pid == 0:
-                    break
-            except:
-                break
-
-requires_zlib = unittest.skipUnless(zlib, 'requires zlib')
-
 # Executing the interpreter in a subprocess
 def _assert_python(expected_success, *args, **env_vars):
     cmd_line = [sys.executable]
@@ -435,28 +347,25 @@
         p.stdout.close()
         p.stderr.close()
     rc = p.returncode
-    err =  strip_python_stderr(err)
+    err = strip_python_stderr(err)
     if (rc and expected_success) or (not rc and not expected_success):
         raise AssertionError(
             "Process return code is %d, "
             "stderr follows:\n%s" % (rc, err.decode('ascii', 'ignore')))
     return rc, out, err
 
+
 def assert_python_ok(*args, **env_vars):
     """
     Assert that running the interpreter with `args` and optional environment
-    variables `env_vars` is ok and return a (return code, stdout, stderr) tuple.
+    variables `env_vars` is ok and return a (return code, stdout, stderr)
+    tuple.
     """
     return _assert_python(True, *args, **env_vars)
 
+
 def unload(name):
     try:
         del sys.modules[name]
     except KeyError:
         pass
-
-try:
-    from test.test_support import skip_unless_symlink
-except ImportError:
-    skip_unless_symlink = unittest.skip(
-        'requires test.test_support.skip_unless_symlink')

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Record_that_d2_does_not_?=
	=?utf8?q?bootstrap_on_2=2E4?=
Message-ID: <E1R5deJ-0007lW-FZ@dinsdale.python.org>

http://hg.python.org/distutils2/rev/3c66d9dfe986
changeset:   1156:3c66d9dfe986
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 00:21:14 2011 +0200
summary:
  Record that d2 does not bootstrap on 2.4

files:
  DEVNOTES.txt |  8 +++++---
  1 files changed, 5 insertions(+), 3 deletions(-)


diff --git a/DEVNOTES.txt b/DEVNOTES.txt
--- a/DEVNOTES.txt
+++ b/DEVNOTES.txt
@@ -1,9 +1,11 @@
 Notes for developers
 ====================
 
-- Distutils2 runs on Python from 2.4 to 3.2 (3.x not implemented yet),
-  so make sure you don't use a syntax that doesn't work under
-  one of these Python versions.
+- Distutils2 runs on Python from 2.4 to 2.7 so make sure you don't use code
+  that doesn't work under one of these Python versions.
+
+- For 2.4, you need to run "python2.4 setup.py build" before you can run tests
+  or pysetup.
 
 - Always run tests.sh before you push a change. This implies
   that you have all Python versions installed from 2.4 to 2.7. Be sure to have 

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_backport_changesets_?=
 =?utf8?q?part_3=3A_backported_modules=2E?=
Message-ID: <E1R5deJ-0007lH-Bx@dinsdale.python.org>

http://hg.python.org/distutils2/rev/d1d251292ee7
changeset:   1152:d1d251292ee7
user:        ??ric Araujo <merwok at netwok.org>
date:        Sun Sep 18 23:10:58 2011 +0200
summary:
  Fix backport changesets part 3: backported modules.

shutil, sysconfig, tarfile and their tests have been updated to the
latest 3.2 version (except for test_tarfile which is not backported yet)
and edited to be compatible with 2.4.

Duplicates added in util during the Great Update have been deleted, as
well as functions I removed recently in packaging.  Unneeded modules in
_backport have been deleted or moved to d2.compat.

files:
  distutils2/_backport/__init__.py             |   10 +-
  distutils2/_backport/functools.py            |   56 -
  distutils2/_backport/path.py                 |   15 -
  distutils2/_backport/shutil.py               |   27 +-
  distutils2/_backport/sysconfig.py            |   24 +-
  distutils2/_backport/tarfile.py              |  894 +++++----
  distutils2/_backport/tests/test_shutil.py    |  233 +-
  distutils2/_backport/tests/test_sysconfig.py |  212 +-
  distutils2/command/build_scripts.py          |    5 +-
  distutils2/command/cmd.py                    |    3 +-
  distutils2/command/sdist.py                  |    4 +-
  distutils2/compat.py                         |  131 +-
  distutils2/create.py                         |   26 +-
  distutils2/install.py                        |    3 +-
  distutils2/pypi/dist.py                      |   14 +-
  distutils2/pypi/simple.py                    |    5 +-
  distutils2/tests/pypi_server.py              |    5 +-
  distutils2/tests/support.py                  |    2 +-
  distutils2/tests/test_command_sdist.py       |    3 +-
  distutils2/tests/test_util.py                |    9 +-
  distutils2/util.py                           |  727 +-------
  21 files changed, 969 insertions(+), 1439 deletions(-)


diff --git a/distutils2/_backport/__init__.py b/distutils2/_backport/__init__.py
--- a/distutils2/_backport/__init__.py
+++ b/distutils2/_backport/__init__.py
@@ -1,8 +1,4 @@
-"""Things that will land in the Python 3.3 std lib but which we must drag along
- us for now to support 2.x."""
+"""Modules copied from the Python 3.2 standard library.
 
-def any(seq):
-    for elem in seq:
-        if elem:
-            return True
-    return False
+Individual classes and objects like the any function are in compat.
+"""
diff --git a/distutils2/_backport/functools.py b/distutils2/_backport/functools.py
deleted file mode 100644
--- a/distutils2/_backport/functools.py
+++ /dev/null
@@ -1,56 +0,0 @@
-"""functools.py - Tools for working with functions and callable objects
-Copied from:
-https://github.com/dln/pycassa/commit/90736f8146c1cac8287f66e8c8b64cb80e011513#diff-1
-
-"""
-
-try:
-    from _functools import partial
-except:
-    class partial(object):
-        "A simple replacement of functools.partial"
-        def __init__(self, func, *args, **kw):
-            self.func = func
-            self.args = args
-            self.keywords = kw
-        def __call__(self, *otherargs, **otherkw):
-            kw = self.keywords.copy()
-            kw.update(otherkw)
-            return self.func(*(self.args + otherargs), **kw)
-
-# update_wrapper() and wraps() are tools to help write
-# wrapper functions that can handle naive introspection
-
-WRAPPER_ASSIGNMENTS = ('__module__', '__name__', '__doc__')
-WRAPPER_UPDATES = ('__dict__',)
-def update_wrapper(wrapper, wrapped, assigned=WRAPPER_ASSIGNMENTS,
-                   updated=WRAPPER_UPDATES):
-    """Update a wrapper function to look like the wrapped function
-
-       wrapper is the function to be updated
-       wrapped is the original function
-       assigned is a tuple naming the attributes assigned directly
-       from the wrapped function to the wrapper function (defaults to
-       functools.WRAPPER_ASSIGNMENTS)
-       updated is a tuple naming the attributes of the wrapper that
-       are updated with the corresponding attribute from the wrapped
-       function (defaults to functools.WRAPPER_UPDATES)
-    """
-    for attr in assigned:
-        setattr(wrapper, attr, getattr(wrapped, attr))
-    for attr in updated:
-        getattr(wrapper, attr).update(getattr(wrapped, attr, {}))
-    # Return the wrapper so this can be used as a decorator via partial()
-    return wrapper
-
-def wraps(wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES):
-    """Decorator factory to apply update_wrapper() to a wrapper function
-
-       Returns a decorator that invokes update_wrapper() with the decorated
-       function as the wrapper argument and the arguments to wraps() as the
-       remaining arguments. Default arguments are as for update_wrapper().
-       This is a convenience function to simplify applying partial() to
-       update_wrapper().
-    """
-    return partial(update_wrapper, wrapped=wrapped,
-                   assigned=assigned, updated=updated)
diff --git a/distutils2/_backport/path.py b/distutils2/_backport/path.py
deleted file mode 100644
--- a/distutils2/_backport/path.py
+++ /dev/null
@@ -1,15 +0,0 @@
-from posixpath import curdir, sep, pardir, join, abspath, commonprefix
-
-def relpath(path, start=curdir):
-    """Return a relative version of a path"""
-    if not path:
-        raise ValueError("no path specified")
-    start_list = abspath(start).split(sep)
-    path_list = abspath(path).split(sep)
-    # Work out how much of the filepath is shared by start and path.
-    i = len(commonprefix([start_list, path_list]))
-    rel_list = [pardir] * (len(start_list)-i) + path_list[i:]
-    if not rel_list:
-        return curdir
-    return join(*rel_list)
-
diff --git a/distutils2/_backport/shutil.py b/distutils2/_backport/shutil.py
--- a/distutils2/_backport/shutil.py
+++ b/distutils2/_backport/shutil.py
@@ -32,7 +32,7 @@
            "ExecError", "make_archive", "get_archive_formats",
            "register_archive_format", "unregister_archive_format",
            "get_unpack_formats", "register_unpack_format",
-           "unregister_unpack_format", "unpack_archive"]
+           "unregister_unpack_format", "unpack_archive", "ignore_patterns"]
 
 class Error(EnvironmentError):
     pass
@@ -202,8 +202,11 @@
     else:
         ignored_names = set()
 
-    if not os.path.exists(dst):
+    try:
         os.makedirs(dst)
+    except OSError, e:
+        if e.errno != errno.EEXIST:
+            raise
 
     errors = []
     for name in names:
@@ -317,6 +320,12 @@
     """
     real_dst = dst
     if os.path.isdir(dst):
+        if _samefile(src, dst):
+            # We might be on a case insensitive filesystem,
+            # perform the rename anyway.
+            os.rename(src, dst)
+            return
+
         real_dst = os.path.join(dst, _basename(src))
         if os.path.exists(real_dst):
             raise Error("Destination path '%s' already exists" % real_dst)
@@ -408,7 +417,7 @@
     from distutils2._backport import tarfile
 
     if logger is not None:
-        logger.info('creating tar archive')
+        logger.info('Creating tar archive')
 
     uid = _get_uid(owner)
     gid = _get_gid(group)
@@ -696,6 +705,7 @@
 def _unpack_tarfile(filename, extract_dir):
     """Unpack tar/tar.gz/tar.bz2 `filename` to `extract_dir`
     """
+    # late import because of circular dependency
     from distutils2._backport import tarfile
     try:
         tarobj = tarfile.open(filename)
@@ -742,16 +752,14 @@
     if extract_dir is None:
         extract_dir = os.getcwd()
 
-    func = None
-
     if format is not None:
         try:
             format_info = _UNPACK_FORMATS[format]
         except KeyError:
             raise ValueError("Unknown unpack format '%s'" % format)
 
-        func = format_info[0]
-        func(filename, extract_dir, **dict(format_info[1]))
+        func = format_info[1]
+        func(filename, extract_dir, **dict(format_info[2]))
     else:
         # we need to look at the registered unpackers supported extensions
         format = _find_unpack_format(filename)
@@ -761,8 +769,3 @@
         func = _UNPACK_FORMATS[format][1]
         kwargs = dict(_UNPACK_FORMATS[format][2])
         func(filename, extract_dir, **kwargs)
-
-    if func is None:
-        raise ValueError('Unknown archive format: %s' % filename)
-
-    return extract_dir
diff --git a/distutils2/_backport/sysconfig.py b/distutils2/_backport/sysconfig.py
--- a/distutils2/_backport/sysconfig.py
+++ b/distutils2/_backport/sysconfig.py
@@ -338,8 +338,10 @@
     config_h = get_config_h_filename()
     try:
         f = open(config_h)
-        parse_config_h(f, vars)
-        f.close()
+        try:
+            parse_config_h(f, vars)
+        finally:
+            f.close()
     except IOError, e:
         msg = "invalid Python installation: unable to open %s" % config_h
         if hasattr(e, "strerror"):
@@ -730,13 +732,13 @@
                 # On OSX the machine type returned by uname is always the
                 # 32-bit variant, even if the executable architecture is
                 # the 64-bit variant
-                if sys.maxsize >= 2**32:
+                if sys.maxint >= 2**32:
                     machine = 'x86_64'
 
             elif machine in ('PowerPC', 'Power_Macintosh'):
                 # Pick a sane name for the PPC architecture.
                 # See 'i386' case
-                if sys.maxsize >= 2**32:
+                if sys.maxint >= 2**32:
                     machine = 'ppc64'
                 else:
                     machine = 'ppc'
@@ -751,18 +753,18 @@
 def _print_dict(title, data):
     for index, (key, value) in enumerate(sorted(data.items())):
         if index == 0:
-            print('%s: ' % (title))
-        print('\t%s = "%s"' % (key, value))
+            print '%s: ' % (title)
+        print '\t%s = "%s"' % (key, value)
 
 
 def _main():
     """Display all information sysconfig detains."""
-    print('Platform: "%s"' % get_platform())
-    print('Python version: "%s"' % get_python_version())
-    print('Current installation scheme: "%s"' % _get_default_scheme())
-    print(u'')
+    print 'Platform: "%s"' % get_platform()
+    print 'Python version: "%s"' % get_python_version()
+    print 'Current installation scheme: "%s"' % _get_default_scheme()
+    print
     _print_dict('Paths', get_paths())
-    print(u'')
+    print
     _print_dict('Variables', get_config_vars())
 
 
diff --git a/distutils2/_backport/tarfile.py b/distutils2/_backport/tarfile.py
--- a/distutils2/_backport/tarfile.py
+++ b/distutils2/_backport/tarfile.py
@@ -1,9 +1,9 @@
 #!/usr/bin/env python
-# -*- coding: iso-8859-1 -*-
+# encoding: utf-8
 #-------------------------------------------------------------------
 # tarfile.py
 #-------------------------------------------------------------------
-# Copyright (C) 2002 Lars Gust?bel <lars at gustaebel.de>
+# Copyright (C) 2002 Lars Gust??bel <lars at gustaebel.de>
 # All rights reserved.
 #
 # Permission  is  hereby granted,  free  of charge,  to  any person
@@ -30,14 +30,13 @@
 """Read from and write to tar format archives.
 """
 
-__version__ = "$Revision: 76780 $"
-# $Source$
+__version__ = "$Revision$"
 
 version     = "0.9.0"
-__author__  = "Lars Gust?bel (lars at gustaebel.de)"
-__date__    = "$Date: 2009-12-13 12:32:27 +0100 (Dim 13 d??c 2009) $"
-__cvsid__   = "$Id: tarfile.py 76780 2009-12-13 11:32:27Z lars.gustaebel $"
-__credits__ = "Gustavo Niemeyer, Niels Gust?bel, Richard Townsend."
+__author__  = u"Lars Gust\u00e4bel (lars at gustaebel.de)"
+__date__    = "$Date: 2011-02-25 17:42:01 +0200 (Fri, 25 Feb 2011) $"
+__cvsid__   = "$Id: tarfile.py 88586 2011-02-25 15:42:01Z marc-andre.lemburg $"
+__credits__ = u"Gustavo Niemeyer, Niels Gust\u00e4bel, Richard Townsend."
 
 #---------
 # Imports
@@ -51,19 +50,26 @@
 import struct
 import copy
 import re
-import operator
-
-if not hasattr(os, 'SEEK_SET'):
-    os.SEEK_SET = 0
 
 try:
     import grp, pwd
 except ImportError:
     grp = pwd = None
 
+# os.symlink on Windows prior to 6.0 raises NotImplementedError
+symlink_exception = (AttributeError, NotImplementedError)
+try:
+    # WindowsError (1314) will be raised if the caller does not hold the
+    # SeCreateSymbolicLinkPrivilege privilege
+    symlink_exception += (WindowsError,)
+except NameError:
+    pass
+
 # from tarfile import *
 __all__ = ["TarFile", "TarInfo", "is_tarfile", "TarError"]
 
+from __builtin__ import open as _open # Since 'open' is TarFile.open
+
 #---------------------------------------------------------
 # tar constants
 #---------------------------------------------------------
@@ -122,6 +128,9 @@
 PAX_FIELDS = ("path", "linkpath", "size", "mtime",
               "uid", "gid", "uname", "gname")
 
+# Fields from a pax header that are affected by hdrcharset.
+PAX_NAME_FIELDS = set(("path", "linkpath", "uname", "gname"))
+
 # Fields in a pax header that are numbers, all other fields
 # are treated as strings.
 PAX_NUMBER_FIELDS = {
@@ -160,27 +169,28 @@
 #---------------------------------------------------------
 # initialization
 #---------------------------------------------------------
-ENCODING = sys.getfilesystemencoding()
-if ENCODING is None:
-    ENCODING = sys.getdefaultencoding()
+if os.name in ("nt", "ce"):
+    ENCODING = "utf-8"
+else:
+    ENCODING = sys.getfilesystemencoding()
 
 #---------------------------------------------------------
 # Some useful functions
 #---------------------------------------------------------
 
-def stn(s, length):
-    """Convert a python string to a null-terminated string buffer.
+def stn(s, length, encoding, errors):
+    """Convert a string to a null-terminated bytes object.
     """
+    s = s.encode(encoding, errors)
     return s[:length] + (length - len(s)) * NUL
 
-def nts(s):
-    """Convert a null-terminated string field to a python string.
+def nts(s, encoding, errors):
+    """Convert a null-terminated bytes object to a string.
     """
-    # Use the string up to the first null char.
     p = s.find("\0")
-    if p == -1:
-        return s
-    return s[:p]
+    if p != -1:
+        s = s[:p]
+    return s.decode(encoding, errors)
 
 def nti(s):
     """Convert a number field to a python number.
@@ -189,7 +199,7 @@
     # itn() below.
     if s[0] != chr(0200):
         try:
-            n = int(nts(s) or "0", 8)
+            n = int(nts(s, "ascii", "strict") or "0", 8)
         except ValueError:
             raise InvalidHeaderError("invalid header")
     else:
@@ -226,26 +236,6 @@
         s = chr(0200) + s
     return s
 
-def uts(s, encoding, errors):
-    """Convert a unicode object to a string.
-    """
-    if errors == "utf-8":
-        # An extra error handler similar to the -o invalid=UTF-8 option
-        # in POSIX.1-2001. Replace untranslatable characters with their
-        # UTF-8 representation.
-        try:
-            return s.encode(encoding, "strict")
-        except UnicodeEncodeError:
-            x = []
-            for c in s:
-                try:
-                    x.append(c.encode(encoding, "strict"))
-                except UnicodeEncodeError:
-                    x.append(c.encode("utf8"))
-            return "".join(x)
-    else:
-        return s.encode(encoding, errors)
-
 def calc_chksums(buf):
     """Calculate the checksum for a member's header by summing up all
        characters except for the chksum field which is treated as if
@@ -376,7 +366,7 @@
         }[mode]
         if hasattr(os, "O_BINARY"):
             mode |= os.O_BINARY
-        self.fd = os.open(name, mode)
+        self.fd = os.open(name, mode, 0666)
 
     def close(self):
         os.close(self.fd)
@@ -421,28 +411,34 @@
         self.pos      = 0L
         self.closed   = False
 
-        if comptype == "gz":
-            try:
-                import zlib
-            except ImportError:
-                raise CompressionError("zlib module is not available")
-            self.zlib = zlib
-            self.crc = zlib.crc32("") & 0xffffffffL
-            if mode == "r":
-                self._init_read_gz()
-            else:
-                self._init_write_gz()
+        try:
+            if comptype == "gz":
+                try:
+                    import zlib
+                except ImportError:
+                    raise CompressionError("zlib module is not available")
+                self.zlib = zlib
+                self.crc = zlib.crc32("")
+                if mode == "r":
+                    self._init_read_gz()
+                else:
+                    self._init_write_gz()
 
-        if comptype == "bz2":
-            try:
-                import bz2
-            except ImportError:
-                raise CompressionError("bz2 module is not available")
-            if mode == "r":
-                self.dbuf = ""
-                self.cmp = bz2.BZ2Decompressor()
-            else:
-                self.cmp = bz2.BZ2Compressor()
+            if comptype == "bz2":
+                try:
+                    import bz2
+                except ImportError:
+                    raise CompressionError("bz2 module is not available")
+                if mode == "r":
+                    self.dbuf = ""
+                    self.cmp = bz2.BZ2Decompressor()
+                else:
+                    self.cmp = bz2.BZ2Compressor()
+        except:
+            if not self._extfileobj:
+                self.fileobj.close()
+            self.closed = True
+            raise
 
     def __del__(self):
         if hasattr(self, "closed") and not self.closed:
@@ -459,7 +455,8 @@
         self.__write("\037\213\010\010%s\002\377" % timestamp)
         if self.name.endswith(".gz"):
             self.name = self.name[:-3]
-        self.__write(self.name + NUL)
+        # RFC1952 says we must use ISO-8859-1 for the FNAME field.
+        self.__write(self.name.encode("iso-8859-1", "replace") + NUL)
 
     def write(self, s):
         """Write string s to the stream.
@@ -582,7 +579,6 @@
             return self.__read(size)
 
         c = len(self.dbuf)
-        t = [self.dbuf]
         while c < size:
             buf = self.__read(self.bufsize)
             if not buf:
@@ -591,27 +587,26 @@
                 buf = self.cmp.decompress(buf)
             except IOError:
                 raise ReadError("invalid compressed data")
-            t.append(buf)
+            self.dbuf += buf
             c += len(buf)
-        t = "".join(t)
-        self.dbuf = t[size:]
-        return t[:size]
+        buf = self.dbuf[:size]
+        self.dbuf = self.dbuf[size:]
+        return buf
 
     def __read(self, size):
         """Return size bytes from stream. If internal buffer is empty,
            read another block from the stream.
         """
         c = len(self.buf)
-        t = [self.buf]
         while c < size:
             buf = self.fileobj.read(self.bufsize)
             if not buf:
                 break
-            t.append(buf)
+            self.buf += buf
             c += len(buf)
-        t = "".join(t)
-        self.buf = t[size:]
-        return t[:size]
+        buf = self.buf[:size]
+        self.buf = self.buf[size:]
+        return buf
 # class _Stream
 
 class _StreamProxy(object):
@@ -665,16 +660,14 @@
             self.bz2obj = bz2.BZ2Compressor()
 
     def read(self, size):
-        b = [self.buf]
         x = len(self.buf)
         while x < size:
             raw = self.fileobj.read(self.blocksize)
             if not raw:
                 break
             data = self.bz2obj.decompress(raw)
-            b.append(data)
+            self.buf += data
             x += len(data)
-        self.buf = "".join(b)
 
         buf = self.buf[:size]
         self.buf = self.buf[size:]
@@ -709,13 +702,35 @@
        object.
     """
 
-    def __init__(self, fileobj, offset, size, sparse=None):
+    def __init__(self, fileobj, offset, size, blockinfo=None):
         self.fileobj = fileobj
         self.offset = offset
         self.size = size
-        self.sparse = sparse
         self.position = 0
 
+        if blockinfo is None:
+            blockinfo = [(0, size)]
+
+        # Construct a map with data and zero blocks.
+        self.map_index = 0
+        self.map = []
+        lastpos = 0
+        realpos = self.offset
+        for offset, size in blockinfo:
+            if offset > lastpos:
+                self.map.append((False, lastpos, offset, None))
+            self.map.append((True, offset, offset + size, realpos))
+            realpos += size
+            lastpos = offset + size
+        if lastpos < self.size:
+            self.map.append((False, lastpos, self.size, None))
+
+    def seekable(self):
+        if not hasattr(self.fileobj, "seekable"):
+            # XXX gzip.GzipFile and bz2.BZ2File
+            return True
+        return self.fileobj.seekable()
+
     def tell(self):
         """Return the current file position.
         """
@@ -734,48 +749,25 @@
         else:
             size = min(size, self.size - self.position)
 
-        if self.sparse is None:
-            return self.readnormal(size)
-        else:
-            return self.readsparse(size)
-
-    def readnormal(self, size):
-        """Read operation for regular files.
-        """
-        self.fileobj.seek(self.offset + self.position)
-        self.position += size
-        return self.fileobj.read(size)
-
-    def readsparse(self, size):
-        """Read operation for sparse files.
-        """
-        data = []
+        buf = ""
         while size > 0:
-            buf = self.readsparsesection(size)
-            if not buf:
-                break
-            size -= len(buf)
-            data.append(buf)
-        return "".join(data)
-
-    def readsparsesection(self, size):
-        """Read a single section of a sparse file.
-        """
-        section = self.sparse.find(self.position)
-
-        if section is None:
-            return ""
-
-        size = min(size, section.offset + section.size - self.position)
-
-        if isinstance(section, _data):
-            realpos = section.realpos + self.position - section.offset
-            self.fileobj.seek(self.offset + realpos)
-            self.position += size
-            return self.fileobj.read(size)
-        else:
-            self.position += size
-            return NUL * size
+            while True:
+                data, start, stop, offset = self.map[self.map_index]
+                if start <= self.position < stop:
+                    break
+                else:
+                    self.map_index += 1
+                    if self.map_index == len(self.map):
+                        self.map_index = 0
+            length = min(size, stop - self.position)
+            if data:
+                self.fileobj.seek(offset + (self.position - start))
+                buf += self.fileobj.read(length)
+            else:
+                buf += NUL * length
+            size -= length
+            self.position += length
+        return buf
 #class _FileInFile
 
 
@@ -789,7 +781,7 @@
         self.fileobj = _FileInFile(tarfile.fileobj,
                                    tarinfo.offset_data,
                                    tarinfo.size,
-                                   getattr(tarinfo, "sparse", None))
+                                   tarinfo.sparse)
         self.name = tarinfo.name
         self.mode = "r"
         self.closed = False
@@ -798,6 +790,15 @@
         self.position = 0
         self.buffer = ""
 
+    def readable(self):
+        return True
+
+    def writable(self):
+        return False
+
+    def seekable(self):
+        return self.fileobj.seekable()
+
     def read(self, size=None):
         """Read at most size bytes from the file. If size is not
            present or None, read all data until EOF is reached.
@@ -822,6 +823,9 @@
         self.position += len(buf)
         return buf
 
+    # XXX TextIOWrapper uses the read1() method.
+    read1 = read
+
     def readline(self, size=-1):
         """Read one entire line from the file. If size is present
            and non-negative, return a string with at most that
@@ -830,15 +834,13 @@
         if self.closed:
             raise ValueError("I/O operation on closed file")
 
-        if "\n" in self.buffer:
-            pos = self.buffer.find("\n") + 1
-        else:
-            buffers = [self.buffer]
+        pos = self.buffer.find("\n") + 1
+        if pos == 0:
+            # no newline found.
             while True:
                 buf = self.fileobj.read(self.blocksize)
-                buffers.append(buf)
+                self.buffer += buf
                 if not buf or "\n" in buf:
-                    self.buffer = "".join(buffers)
                     pos = self.buffer.find("\n") + 1
                     if pos == 0:
                         # no newline found.
@@ -871,20 +873,20 @@
 
         return self.position
 
-    def seek(self, pos, whence=os.SEEK_SET):
+    def seek(self, pos, whence=0):
         """Seek to a position in the file.
         """
         if self.closed:
             raise ValueError("I/O operation on closed file")
 
-        if whence == os.SEEK_SET:
+        if whence == 0:  # os.SEEK_SET
             self.position = min(max(pos, 0), self.size)
-        elif whence == os.SEEK_CUR:
+        elif whence == 1:  # os.SEEK_CUR
             if pos < 0:
                 self.position = max(self.position + pos, 0)
             else:
                 self.position = min(self.position + pos, self.size)
-        elif whence == os.SEEK_END:
+        elif whence == 2:  # os.SEEK_END
             self.position = max(min(self.size + pos, self.size), 0)
         else:
             raise ValueError("Invalid argument")
@@ -918,6 +920,12 @@
        usually created internally.
     """
 
+    __slots__ = ("name", "mode", "uid", "gid", "size", "mtime",
+                 "chksum", "type", "linkname", "uname", "gname",
+                 "devmajor", "devminor",
+                 "offset", "offset_data", "pax_headers", "sparse",
+                 "tarfile", "_sparse_structs", "_link_target")
+
     def __init__(self, name=""):
         """Construct a TarInfo object. name is the optional name
            of the member.
@@ -931,14 +939,15 @@
         self.chksum = 0         # header checksum
         self.type = REGTYPE     # member type
         self.linkname = ""      # link name
-        self.uname = "root"     # user name
-        self.gname = "root"     # group name
+        self.uname = ""         # user name
+        self.gname = ""         # group name
         self.devmajor = 0       # device major number
         self.devminor = 0       # device minor number
 
         self.offset = 0         # the tar header starts here
         self.offset_data = 0    # the file's data starts here
 
+        self.sparse = None      # sparse member information
         self.pax_headers = {}   # pax header information
 
     # In pax headers the "name" and "linkname" field are called
@@ -958,7 +967,7 @@
     def __repr__(self):
         return "<%s %r at %#x>" % (self.__class__.__name__,self.name,id(self))
 
-    def get_info(self, encoding, errors):
+    def get_info(self):
         """Return the TarInfo's attributes as a dictionary.
         """
         info = {
@@ -980,27 +989,23 @@
         if info["type"] == DIRTYPE and not info["name"].endswith("/"):
             info["name"] += "/"
 
-        for key in ("name", "linkname", "uname", "gname"):
-            if type(info[key]) is unicode:
-                info[key] = info[key].encode(encoding, errors)
-
         return info
 
     def tobuf(self, format=DEFAULT_FORMAT, encoding=ENCODING, errors="strict"):
         """Return a tar header as a string of 512 byte blocks.
         """
-        info = self.get_info(encoding, errors)
+        info = self.get_info()
 
         if format == USTAR_FORMAT:
-            return self.create_ustar_header(info)
+            return self.create_ustar_header(info, encoding, errors)
         elif format == GNU_FORMAT:
-            return self.create_gnu_header(info)
+            return self.create_gnu_header(info, encoding, errors)
         elif format == PAX_FORMAT:
-            return self.create_pax_header(info, encoding, errors)
+            return self.create_pax_header(info, encoding)
         else:
             raise ValueError("invalid format")
 
-    def create_ustar_header(self, info):
+    def create_ustar_header(self, info, encoding, errors):
         """Return the object as a ustar header block.
         """
         info["magic"] = POSIX_MAGIC
@@ -1011,23 +1016,23 @@
         if len(info["name"]) > LENGTH_NAME:
             info["prefix"], info["name"] = self._posix_split_name(info["name"])
 
-        return self._create_header(info, USTAR_FORMAT)
+        return self._create_header(info, USTAR_FORMAT, encoding, errors)
 
-    def create_gnu_header(self, info):
+    def create_gnu_header(self, info, encoding, errors):
         """Return the object as a GNU header block sequence.
         """
         info["magic"] = GNU_MAGIC
 
         buf = ""
         if len(info["linkname"]) > LENGTH_LINK:
-            buf += self._create_gnu_long_header(info["linkname"], GNUTYPE_LONGLINK)
+            buf += self._create_gnu_long_header(info["linkname"], GNUTYPE_LONGLINK, encoding, errors)
 
         if len(info["name"]) > LENGTH_NAME:
-            buf += self._create_gnu_long_header(info["name"], GNUTYPE_LONGNAME)
+            buf += self._create_gnu_long_header(info["name"], GNUTYPE_LONGNAME, encoding, errors)
 
-        return buf + self._create_header(info, GNU_FORMAT)
+        return buf + self._create_header(info, GNU_FORMAT, encoding, errors)
 
-    def create_pax_header(self, info, encoding, errors):
+    def create_pax_header(self, info, encoding):
         """Return the object as a ustar header block. If it cannot be
            represented this way, prepend a pax extended header sequence
            with supplement information.
@@ -1045,17 +1050,15 @@
                 # The pax header has priority.
                 continue
 
-            val = info[name].decode(encoding, errors)
-
             # Try to encode the string as ASCII.
             try:
-                val.encode("ascii")
+                info[name].encode("ascii", "strict")
             except UnicodeEncodeError:
-                pax_headers[hname] = val
+                pax_headers[hname] = info[name]
                 continue
 
             if len(info[name]) > length:
-                pax_headers[hname] = val
+                pax_headers[hname] = info[name]
 
         # Test number fields for values that exceed the field limit or values
         # that like to be stored as float.
@@ -1072,17 +1075,17 @@
 
         # Create a pax extended header if necessary.
         if pax_headers:
-            buf = self._create_pax_generic_header(pax_headers)
+            buf = self._create_pax_generic_header(pax_headers, XHDTYPE, encoding)
         else:
             buf = ""
 
-        return buf + self._create_header(info, USTAR_FORMAT)
+        return buf + self._create_header(info, USTAR_FORMAT, "ascii", "replace")
 
     @classmethod
     def create_pax_global_header(cls, pax_headers):
         """Return the object as a pax global header block sequence.
         """
-        return cls._create_pax_generic_header(pax_headers, type=XGLTYPE)
+        return cls._create_pax_generic_header(pax_headers, XGLTYPE, "utf8")
 
     def _posix_split_name(self, name):
         """Split a name longer than 100 chars into a prefix
@@ -1100,12 +1103,12 @@
         return prefix, name
 
     @staticmethod
-    def _create_header(info, format):
+    def _create_header(info, format, encoding, errors):
         """Return a header block. info is a dictionary with file
            information, format must be one of the *_FORMAT constants.
         """
         parts = [
-            stn(info.get("name", ""), 100),
+            stn(info.get("name", ""), 100, encoding, errors),
             itn(info.get("mode", 0) & 07777, 8, format),
             itn(info.get("uid", 0), 8, format),
             itn(info.get("gid", 0), 8, format),
@@ -1113,13 +1116,13 @@
             itn(info.get("mtime", 0), 12, format),
             "        ", # checksum field
             info.get("type", REGTYPE),
-            stn(info.get("linkname", ""), 100),
-            stn(info.get("magic", POSIX_MAGIC), 8),
-            stn(info.get("uname", "root"), 32),
-            stn(info.get("gname", "root"), 32),
+            stn(info.get("linkname", ""), 100, encoding, errors),
+            info.get("magic", POSIX_MAGIC),
+            stn(info.get("uname", ""), 32, encoding, errors),
+            stn(info.get("gname", ""), 32, encoding, errors),
             itn(info.get("devmajor", 0), 8, format),
             itn(info.get("devminor", 0), 8, format),
-            stn(info.get("prefix", ""), 155)
+            stn(info.get("prefix", ""), 155, encoding, errors)
         ]
 
         buf = struct.pack("%ds" % BLOCKSIZE, "".join(parts))
@@ -1138,11 +1141,11 @@
         return payload
 
     @classmethod
-    def _create_gnu_long_header(cls, name, type):
+    def _create_gnu_long_header(cls, name, type, encoding, errors):
         """Return a GNUTYPE_LONGNAME or GNUTYPE_LONGLINK sequence
            for name.
         """
-        name += NUL
+        name = name.encode(encoding, errors) + NUL
 
         info = {}
         info["name"] = "././@LongLink"
@@ -1151,19 +1154,39 @@
         info["magic"] = GNU_MAGIC
 
         # create extended header + name blocks.
-        return cls._create_header(info, USTAR_FORMAT) + \
+        return cls._create_header(info, USTAR_FORMAT, encoding, errors) + \
                 cls._create_payload(name)
 
     @classmethod
-    def _create_pax_generic_header(cls, pax_headers, type=XHDTYPE):
-        """Return a POSIX.1-2001 extended or global header sequence
+    def _create_pax_generic_header(cls, pax_headers, type, encoding):
+        """Return a POSIX.1-2008 extended or global header sequence
            that contains a list of keyword, value pairs. The values
            must be unicode objects.
         """
-        records = []
-        for keyword, value in pax_headers.iteritems():
+        # Check if one of the fields contains surrogate characters and thereby
+        # forces hdrcharset=BINARY, see _proc_pax() for more information.
+        binary = False
+        for keyword, value in pax_headers.items():
+            try:
+                value.encode("utf8", "strict")
+            except UnicodeEncodeError:
+                binary = True
+                break
+
+        records = ""
+        if binary:
+            # Put the hdrcharset field at the beginning of the header.
+            records += "21 hdrcharset=BINARY\n"
+
+        for keyword, value in pax_headers.items():
             keyword = keyword.encode("utf8")
-            value = value.encode("utf8")
+            if binary:
+                # Try to restore the original byte representation of `value'.
+                # Needless to say, that the encoding must match the string.
+                value = value.encode(encoding, "surrogateescape")
+            else:
+                value = value.encode("utf8")
+
             l = len(keyword) + len(value) + 3   # ' ' + '=' + '\n'
             n = p = 0
             while True:
@@ -1171,8 +1194,7 @@
                 if n == p:
                     break
                 p = n
-            records.append("%d %s=%s\n" % (p, keyword, value))
-        records = "".join(records)
+            records += bytes(str(p), "ascii") + " " + keyword + "=" + value + "\n"
 
         # We use a hardcoded "././@PaxHeader" name like star does
         # instead of the one that POSIX recommends.
@@ -1183,12 +1205,12 @@
         info["magic"] = POSIX_MAGIC
 
         # Create pax header + record blocks.
-        return cls._create_header(info, USTAR_FORMAT) + \
+        return cls._create_header(info, USTAR_FORMAT, "ascii", "replace") + \
                 cls._create_payload(records)
 
     @classmethod
-    def frombuf(cls, buf):
-        """Construct a TarInfo object from a 512 byte string buffer.
+    def frombuf(cls, buf, encoding, errors):
+        """Construct a TarInfo object from a 512 byte bytes object.
         """
         if len(buf) == 0:
             raise EmptyHeaderError("empty header")
@@ -1202,8 +1224,7 @@
             raise InvalidHeaderError("bad checksum")
 
         obj = cls()
-        obj.buf = buf
-        obj.name = nts(buf[0:100])
+        obj.name = nts(buf[0:100], encoding, errors)
         obj.mode = nti(buf[100:108])
         obj.uid = nti(buf[108:116])
         obj.gid = nti(buf[116:124])
@@ -1211,18 +1232,36 @@
         obj.mtime = nti(buf[136:148])
         obj.chksum = chksum
         obj.type = buf[156:157]
-        obj.linkname = nts(buf[157:257])
-        obj.uname = nts(buf[265:297])
-        obj.gname = nts(buf[297:329])
+        obj.linkname = nts(buf[157:257], encoding, errors)
+        obj.uname = nts(buf[265:297], encoding, errors)
+        obj.gname = nts(buf[297:329], encoding, errors)
         obj.devmajor = nti(buf[329:337])
         obj.devminor = nti(buf[337:345])
-        prefix = nts(buf[345:500])
+        prefix = nts(buf[345:500], encoding, errors)
 
         # Old V7 tar format represents a directory as a regular
         # file with a trailing slash.
         if obj.type == AREGTYPE and obj.name.endswith("/"):
             obj.type = DIRTYPE
 
+        # The old GNU sparse format occupies some of the unused
+        # space in the buffer for up to 4 sparse structures.
+        # Save the them for later processing in _proc_sparse().
+        if obj.type == GNUTYPE_SPARSE:
+            pos = 386
+            structs = []
+            for i in range(4):
+                try:
+                    offset = nti(buf[pos:pos + 12])
+                    numbytes = nti(buf[pos + 12:pos + 24])
+                except ValueError:
+                    break
+                structs.append((offset, numbytes))
+                pos += 24
+            isextended = bool(buf[482])
+            origsize = nti(buf[483:495])
+            obj._sparse_structs = (structs, isextended, origsize)
+
         # Remove redundant slashes from directories.
         if obj.isdir():
             obj.name = obj.name.rstrip("/")
@@ -1238,7 +1277,7 @@
            tarfile.
         """
         buf = tarfile.fileobj.read(BLOCKSIZE)
-        obj = cls.frombuf(buf)
+        obj = cls.frombuf(buf, tarfile.encoding, tarfile.errors)
         obj.offset = tarfile.fileobj.tell() - BLOCKSIZE
         return obj._proc_member(tarfile)
 
@@ -1299,41 +1338,21 @@
         # the longname information.
         next.offset = self.offset
         if self.type == GNUTYPE_LONGNAME:
-            next.name = nts(buf)
+            next.name = nts(buf, tarfile.encoding, tarfile.errors)
         elif self.type == GNUTYPE_LONGLINK:
-            next.linkname = nts(buf)
+            next.linkname = nts(buf, tarfile.encoding, tarfile.errors)
 
         return next
 
     def _proc_sparse(self, tarfile):
         """Process a GNU sparse header plus extra headers.
         """
-        buf = self.buf
-        sp = _ringbuffer()
-        pos = 386
-        lastpos = 0L
-        realpos = 0L
-        # There are 4 possible sparse structs in the
-        # first header.
-        for i in xrange(4):
-            try:
-                offset = nti(buf[pos:pos + 12])
-                numbytes = nti(buf[pos + 12:pos + 24])
-            except ValueError:
-                break
-            if offset > lastpos:
-                sp.append(_hole(lastpos, offset - lastpos))
-            sp.append(_data(offset, numbytes, realpos))
-            realpos += numbytes
-            lastpos = offset + numbytes
-            pos += 24
+        # We already collected some sparse structures in frombuf().
+        structs, isextended, origsize = self._sparse_structs
+        del self._sparse_structs
 
-        isextended = ord(buf[482])
-        origsize = nti(buf[483:495])
-
-        # If the isextended flag is given,
-        # there are extra headers to process.
-        while isextended == 1:
+        # Collect sparse structures from extended header blocks.
+        while isextended:
             buf = tarfile.fileobj.read(BLOCKSIZE)
             pos = 0
             for i in xrange(21):
@@ -1342,28 +1361,20 @@
                     numbytes = nti(buf[pos + 12:pos + 24])
                 except ValueError:
                     break
-                if offset > lastpos:
-                    sp.append(_hole(lastpos, offset - lastpos))
-                sp.append(_data(offset, numbytes, realpos))
-                realpos += numbytes
-                lastpos = offset + numbytes
+                if offset and numbytes:
+                    structs.append((offset, numbytes))
                 pos += 24
-            isextended = ord(buf[504])
-
-        if lastpos < origsize:
-            sp.append(_hole(lastpos, origsize - lastpos))
-
-        self.sparse = sp
+            isextended = bool(buf[504])
+        self.sparse = structs
 
         self.offset_data = tarfile.fileobj.tell()
         tarfile.offset = self.offset_data + self._block(self.size)
         self.size = origsize
-
         return self
 
     def _proc_pax(self, tarfile):
         """Process an extended or global header as described in
-           POSIX.1-2001.
+           POSIX.1-2008.
         """
         # Read the header information.
         buf = tarfile.fileobj.read(self._block(self.size))
@@ -1376,11 +1387,29 @@
         else:
             pax_headers = tarfile.pax_headers.copy()
 
+        # Check if the pax header contains a hdrcharset field. This tells us
+        # the encoding of the path, linkpath, uname and gname fields. Normally,
+        # these fields are UTF-8 encoded but since POSIX.1-2008 tar
+        # implementations are allowed to store them as raw binary strings if
+        # the translation to UTF-8 fails.
+        match = re.search(r"\d+ hdrcharset=([^\n]+)\n", buf)
+        if match is not None:
+            pax_headers["hdrcharset"] = match.group(1).decode("utf8")
+
+        # For the time being, we don't care about anything other than "BINARY".
+        # The only other value that is currently allowed by the standard is
+        # "ISO-IR 10646 2000 UTF-8" in other words UTF-8.
+        hdrcharset = pax_headers.get("hdrcharset")
+        if hdrcharset == "BINARY":
+            encoding = tarfile.encoding
+        else:
+            encoding = "utf8"
+
         # Parse pax header information. A record looks like that:
         # "%d %s=%s\n" % (length, keyword, value). length is the size
         # of the complete record including the length field itself and
         # the newline. keyword and value are both UTF-8 encoded strings.
-        regex = re.compile(r"(\d+) ([^=]+)=", re.U)
+        regex = re.compile(r"(\d+) ([^=]+)=")
         pos = 0
         while True:
             match = regex.match(buf, pos)
@@ -1391,8 +1420,21 @@
             length = int(length)
             value = buf[match.end(2) + 1:match.start(1) + length - 1]
 
-            keyword = keyword.decode("utf8")
-            value = value.decode("utf8")
+            # Normally, we could just use "utf8" as the encoding and "strict"
+            # as the error handler, but we better not take the risk. For
+            # example, GNU tar <= 1.23 is known to store filenames it cannot
+            # translate to UTF-8 as raw strings (unfortunately without a
+            # hdrcharset=BINARY header).
+            # We first try the strict standard encoding, and if that fails we
+            # fall back on the user's encoding and error handler.
+            keyword = self._decode_pax_field(keyword, "utf8", "utf8",
+                    tarfile.errors)
+            if keyword in PAX_NAME_FIELDS:
+                value = self._decode_pax_field(value, encoding, tarfile.encoding,
+                        tarfile.errors)
+            else:
+                value = self._decode_pax_field(value, "utf8", "utf8",
+                        tarfile.errors)
 
             pax_headers[keyword] = value
             pos += length
@@ -1403,6 +1445,19 @@
         except HeaderError:
             raise SubsequentHeaderError("missing or bad subsequent header")
 
+        # Process GNU sparse information.
+        if "GNU.sparse.map" in pax_headers:
+            # GNU extended sparse format version 0.1.
+            self._proc_gnusparse_01(next, pax_headers)
+
+        elif "GNU.sparse.size" in pax_headers:
+            # GNU extended sparse format version 0.0.
+            self._proc_gnusparse_00(next, pax_headers, buf)
+
+        elif pax_headers.get("GNU.sparse.major") == "1" and pax_headers.get("GNU.sparse.minor") == "0":
+            # GNU extended sparse format version 1.0.
+            self._proc_gnusparse_10(next, pax_headers, tarfile)
+
         if self.type in (XHDTYPE, SOLARIS_XHDTYPE):
             # Patch the TarInfo object with the extended header info.
             next._apply_pax_info(pax_headers, tarfile.encoding, tarfile.errors)
@@ -1419,29 +1474,70 @@
 
         return next
 
+    def _proc_gnusparse_00(self, next, pax_headers, buf):
+        """Process a GNU tar extended sparse header, version 0.0.
+        """
+        offsets = []
+        for match in re.finditer(r"\d+ GNU.sparse.offset=(\d+)\n", buf):
+            offsets.append(int(match.group(1)))
+        numbytes = []
+        for match in re.finditer(r"\d+ GNU.sparse.numbytes=(\d+)\n", buf):
+            numbytes.append(int(match.group(1)))
+        next.sparse = list(zip(offsets, numbytes))
+
+    def _proc_gnusparse_01(self, next, pax_headers):
+        """Process a GNU tar extended sparse header, version 0.1.
+        """
+        sparse = [int(x) for x in pax_headers["GNU.sparse.map"].split(",")]
+        next.sparse = list(zip(sparse[::2], sparse[1::2]))
+
+    def _proc_gnusparse_10(self, next, pax_headers, tarfile):
+        """Process a GNU tar extended sparse header, version 1.0.
+        """
+        fields = None
+        sparse = []
+        buf = tarfile.fileobj.read(BLOCKSIZE)
+        fields, buf = buf.split("\n", 1)
+        fields = int(fields)
+        while len(sparse) < fields * 2:
+            if "\n" not in buf:
+                buf += tarfile.fileobj.read(BLOCKSIZE)
+            number, buf = buf.split("\n", 1)
+            sparse.append(int(number))
+        next.offset_data = tarfile.fileobj.tell()
+        next.sparse = list(zip(sparse[::2], sparse[1::2]))
+
     def _apply_pax_info(self, pax_headers, encoding, errors):
         """Replace fields with supplemental information from a previous
            pax extended or global header.
         """
-        for keyword, value in pax_headers.iteritems():
-            if keyword not in PAX_FIELDS:
-                continue
-
-            if keyword == "path":
-                value = value.rstrip("/")
-
-            if keyword in PAX_NUMBER_FIELDS:
-                try:
-                    value = PAX_NUMBER_FIELDS[keyword](value)
-                except ValueError:
-                    value = 0
-            else:
-                value = uts(value, encoding, errors)
-
-            setattr(self, keyword, value)
+        for keyword, value in pax_headers.items():
+            if keyword == "GNU.sparse.name":
+                setattr(self, "path", value)
+            elif keyword == "GNU.sparse.size":
+                setattr(self, "size", int(value))
+            elif keyword == "GNU.sparse.realsize":
+                setattr(self, "size", int(value))
+            elif keyword in PAX_FIELDS:
+                if keyword in PAX_NUMBER_FIELDS:
+                    try:
+                        value = PAX_NUMBER_FIELDS[keyword](value)
+                    except ValueError:
+                        value = 0
+                if keyword == "path":
+                    value = value.rstrip("/")
+                setattr(self, keyword, value)
 
         self.pax_headers = pax_headers.copy()
 
+    def _decode_pax_field(self, value, encoding, fallback_encoding, fallback_errors):
+        """Decode a single field from a pax record.
+        """
+        try:
+            return value.decode(encoding, "strict")
+        except UnicodeDecodeError:
+            return value.decode(fallback_encoding, fallback_errors)
+
     def _block(self, count):
         """Round up a byte count by BLOCKSIZE and return it,
            e.g. _block(834) => 1024.
@@ -1468,7 +1564,7 @@
     def isfifo(self):
         return self.type == FIFOTYPE
     def issparse(self):
-        return self.type == GNUTYPE_SPARSE
+        return self.sparse is not None
     def isdev(self):
         return self.type in (CHRTYPE, BLKTYPE, FIFOTYPE)
 # class TarInfo
@@ -1501,7 +1597,7 @@
 
     def __init__(self, name=None, mode="r", fileobj=None, format=None,
             tarinfo=None, dereference=None, ignore_zeros=None, encoding=None,
-            errors=None, pax_headers=None, debug=None, errorlevel=None):
+            errors="strict", pax_headers=None, debug=None, errorlevel=None):
         """Open an (uncompressed) tar archive `name'. `mode' is either 'r' to
            read from an existing archive, 'a' to append data to an existing
            file or 'w' to create a new file overwriting an existing one. `mode'
@@ -1545,13 +1641,7 @@
             self.ignore_zeros = ignore_zeros
         if encoding is not None:
             self.encoding = encoding
-
-        if errors is not None:
-            self.errors = errors
-        elif mode == "r":
-            self.errors = "utf-8"
-        else:
-            self.errors = "strict"
+        self.errors = errors
 
         if pax_headers is not None and self.format == PAX_FORMAT:
             self.pax_headers = pax_headers
@@ -1604,18 +1694,6 @@
             self.closed = True
             raise
 
-    def _getposix(self):
-        return self.format == USTAR_FORMAT
-    def _setposix(self, value):
-        import warnings
-        warnings.warn("use the format attribute instead", DeprecationWarning,
-                      2)
-        if value:
-            self.format = USTAR_FORMAT
-        else:
-            self.format = GNU_FORMAT
-    posix = property(_getposix, _setposix)
-
     #--------------------------------------------------------------------------
     # Below are the classmethods which act as alternate constructors to the
     # TarFile class. The open() method is the only one that is needed for
@@ -1689,9 +1767,12 @@
             if filemode not in "rw":
                 raise ValueError("mode must be 'r' or 'w'")
 
-            t = cls(name, filemode,
-                    _Stream(name, filemode, comptype, fileobj, bufsize),
-                    **kwargs)
+            stream = _Stream(name, filemode, comptype, fileobj, bufsize)
+            try:
+                t = cls(name, filemode, stream, **kwargs)
+            except:
+                stream.close()
+                raise
             t._extfileobj = False
             return t
 
@@ -1722,16 +1803,21 @@
         except (ImportError, AttributeError):
             raise CompressionError("gzip module is not available")
 
-        if fileobj is None:
-            fileobj = bltn_open(name, mode + "b")
-
+        extfileobj = fileobj is not None
         try:
-            t = cls.taropen(name, mode,
-                gzip.GzipFile(name, mode, compresslevel, fileobj),
-                **kwargs)
+            fileobj = gzip.GzipFile(name, mode + "b", compresslevel, fileobj)
+            t = cls.taropen(name, mode, fileobj, **kwargs)
         except IOError:
+            if not extfileobj and fileobj is not None:
+                fileobj.close()
+            if fileobj is None:
+                raise
             raise ReadError("not a gzip file")
-        t._extfileobj = False
+        except:
+            if not extfileobj and fileobj is not None:
+                fileobj.close()
+            raise
+        t._extfileobj = extfileobj
         return t
 
     @classmethod
@@ -1755,6 +1841,7 @@
         try:
             t = cls.taropen(name, mode, fileobj, **kwargs)
         except (IOError, EOFError):
+            fileobj.close()
             raise ReadError("not a bzip2 file")
         t._extfileobj = False
         return t
@@ -1890,10 +1977,10 @@
         tarinfo.mode = stmd
         tarinfo.uid = statres.st_uid
         tarinfo.gid = statres.st_gid
-        if stat.S_ISREG(stmd):
+        if type == REGTYPE:
             tarinfo.size = statres.st_size
         else:
-            tarinfo.size = 0L
+            tarinfo.size = 0
         tarinfo.mtime = statres.st_mtime
         tarinfo.type = type
         tarinfo.linkname = linkname
@@ -1938,7 +2025,7 @@
                 sep = "/"
             else:
                 sep = ""
-            print tarinfo.name + (sep),
+            print tarinfo.name + sep,
 
             if verbose:
                 if tarinfo.issym():
@@ -1996,17 +2083,15 @@
         # Append the tar header and data to the archive.
         if tarinfo.isreg():
             f = bltn_open(name, "rb")
-            try:
-                self.addfile(tarinfo, f)
-            finally:
-                f.close()
+            self.addfile(tarinfo, f)
+            f.close()
 
         elif tarinfo.isdir():
             self.addfile(tarinfo)
             if recursive:
                 for f in os.listdir(name):
                     self.add(os.path.join(name, f), os.path.join(arcname, f),
-                            recursive, exclude, filter)
+                            recursive, exclude, filter=filter)
 
         else:
             self.addfile(tarinfo)
@@ -2055,10 +2140,11 @@
                 directories.append(tarinfo)
                 tarinfo = copy.copy(tarinfo)
                 tarinfo.mode = 0700
-            self.extract(tarinfo, path)
+            # Do not set_attrs directories, as we will do that further down
+            self.extract(tarinfo, path, set_attrs=not tarinfo.isdir())
 
         # Reverse sort directories.
-        directories.sort(key=operator.attrgetter('name'))
+        directories.sort(key=lambda a: a.name)
         directories.reverse()
 
         # Set correct owner, mtime and filemode on directories.
@@ -2074,11 +2160,12 @@
                 else:
                     self._dbg(1, "tarfile: %s" % e)
 
-    def extract(self, member, path=""):
+    def extract(self, member, path="", set_attrs=True):
         """Extract a member from the archive to the current working directory,
            using its full name. Its file information is extracted as accurately
            as possible. `member' may be a filename or a TarInfo object. You can
-           specify a different directory using `path'.
+           specify a different directory using `path'. File attributes (owner,
+           mtime, mode) are set unless `set_attrs' is False.
         """
         self._check("r")
 
@@ -2092,7 +2179,8 @@
             tarinfo._link_target = os.path.join(path, tarinfo.linkname)
 
         try:
-            self._extract_member(tarinfo, os.path.join(path, tarinfo.name))
+            self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
+                                 set_attrs=set_attrs)
         except EnvironmentError, e:
             if self.errorlevel > 0:
                 raise
@@ -2139,14 +2227,13 @@
                 raise StreamError("cannot extract (sym)link as file object")
             else:
                 # A (sym)link's file object is its target's file object.
-                return self.extractfile(self._getmember(tarinfo.linkname,
-                                                        tarinfo))
+                return self.extractfile(self._find_link_target(tarinfo))
         else:
             # If there's no data associated with the member (directory, chrdev,
             # blkdev, etc.), return None instead of a file object.
             return None
 
-    def _extract_member(self, tarinfo, targetpath):
+    def _extract_member(self, tarinfo, targetpath, set_attrs=True):
         """Extract the TarInfo object tarinfo to a physical
            file called targetpath.
         """
@@ -2183,10 +2270,11 @@
         else:
             self.makefile(tarinfo, targetpath)
 
-        self.chown(tarinfo, targetpath)
-        if not tarinfo.issym():
-            self.chmod(tarinfo, targetpath)
-            self.utime(tarinfo, targetpath)
+        if set_attrs:
+            self.chown(tarinfo, targetpath)
+            if not tarinfo.issym():
+                self.chmod(tarinfo, targetpath)
+                self.utime(tarinfo, targetpath)
 
     #--------------------------------------------------------------------------
     # Below are the different file methods. They are called via
@@ -2207,13 +2295,18 @@
     def makefile(self, tarinfo, targetpath):
         """Make a file called targetpath.
         """
-        source = self.extractfile(tarinfo)
+        source = self.fileobj
+        source.seek(tarinfo.offset_data)
         target = bltn_open(targetpath, "wb")
-        try:
-            copyfileobj(source, target)
-        finally:
-            source.close()
-            target.close()
+        if tarinfo.sparse is not None:
+            for offset, size in tarinfo.sparse:
+                target.seek(offset)
+                copyfileobj(source, target, size)
+        else:
+            copyfileobj(source, target, tarinfo.size)
+        target.seek(tarinfo.size)
+        target.truncate()
+        target.close()
 
     def makeunknown(self, tarinfo, targetpath):
         """Make a file from a TarInfo object with an unknown type
@@ -2252,26 +2345,28 @@
           instead of a link.
         """
         try:
+            # For systems that support symbolic and hard links.
             if tarinfo.issym():
                 os.symlink(tarinfo.linkname, targetpath)
             else:
                 # See extract().
-                os.link(tarinfo._link_target, targetpath)
-        except AttributeError:
+                if os.path.exists(tarinfo._link_target):
+                    os.link(tarinfo._link_target, targetpath)
+                else:
+                    self._extract_member(self._find_link_target(tarinfo),
+                                         targetpath)
+        except symlink_exception:
             if tarinfo.issym():
-                linkpath = os.path.dirname(tarinfo.name) + "/" + \
-                                        tarinfo.linkname
+                linkpath = os.path.join(os.path.dirname(tarinfo.name),
+                                        tarinfo.linkname)
             else:
                 linkpath = tarinfo.linkname
-
+        else:
             try:
-                self._extract_member(self.getmember(linkpath), targetpath)
-            except (EnvironmentError, KeyError), e:
-                linkpath = linkpath.replace("/", os.sep)
-                try:
-                    shutil.copy2(linkpath, targetpath)
-                except EnvironmentError, e:
-                    raise IOError("link could not be created")
+                self._extract_member(self._find_link_target(tarinfo),
+                                     targetpath)
+            except KeyError:
+                raise ExtractError("unable to resolve link inside archive")
 
     def chown(self, tarinfo, targetpath):
         """Set owner of targetpath according to tarinfo.
@@ -2281,17 +2376,11 @@
             try:
                 g = grp.getgrnam(tarinfo.gname)[2]
             except KeyError:
-                try:
-                    g = grp.getgrgid(tarinfo.gid)[2]
-                except KeyError:
-                    g = os.getgid()
+                g = tarinfo.gid
             try:
                 u = pwd.getpwnam(tarinfo.uname)[2]
             except KeyError:
-                try:
-                    u = pwd.getpwuid(tarinfo.uid)[2]
-                except KeyError:
-                    u = os.getuid()
+                u = tarinfo.uid
             try:
                 if tarinfo.issym() and hasattr(os, "lchown"):
                     os.lchown(targetpath, u, g)
@@ -2370,21 +2459,28 @@
     #--------------------------------------------------------------------------
     # Little helper methods:
 
-    def _getmember(self, name, tarinfo=None):
+    def _getmember(self, name, tarinfo=None, normalize=False):
         """Find an archive member by name from bottom to top.
            If tarinfo is given, it is used as the starting point.
         """
         # Ensure that all members have been loaded.
         members = self.getmembers()
 
-        if tarinfo is None:
-            end = len(members)
-        else:
-            end = members.index(tarinfo)
+        # Limit the member search list up to tarinfo.
+        if tarinfo is not None:
+            members = members[:members.index(tarinfo)]
 
-        for i in xrange(end - 1, -1, -1):
-            if name == members[i].name:
-                return members[i]
+        if normalize:
+            name = os.path.normpath(name)
+
+        for member in reversed(members):
+            if normalize:
+                member_name = os.path.normpath(member.name)
+            else:
+                member_name = member.name
+
+            if name == member_name:
+                return member
 
     def _load(self):
         """Read through the entire archive file and look for readable
@@ -2405,6 +2501,25 @@
         if mode is not None and self.mode not in mode:
             raise IOError("bad operation for mode %r" % self.mode)
 
+    def _find_link_target(self, tarinfo):
+        """Find the target member of a symlink or hardlink member in the
+           archive.
+        """
+        if tarinfo.issym():
+            # Always search the entire archive.
+            linkname = os.path.dirname(tarinfo.name) + "/" + tarinfo.linkname
+            limit = None
+        else:
+            # Search the archive before the link, because a hard link is
+            # just a reference to an already archived file.
+            linkname = tarinfo.linkname
+            limit = tarinfo
+
+        member = self._getmember(linkname, tarinfo=limit, normalize=True)
+        if member is None:
+            raise KeyError("linkname %r not found" % linkname)
+        return member
+
     def __iter__(self):
         """Provide an iterator object.
         """
@@ -2418,6 +2533,20 @@
         """
         if level <= self.debug:
             print >> sys.stderr, msg
+
+    def __enter__(self):
+        self._check()
+        return self
+
+    def __exit__(self, type, value, traceback):
+        if type is None:
+            self.close()
+        else:
+            # An exception occurred. We must not call close() because
+            # it would try to write end-of-archive blocks and padding.
+            if not self._extfileobj:
+                self.fileobj.close()
+            self.closed = True
 # class TarFile
 
 class TarIter(object):
@@ -2456,103 +2585,6 @@
         self.index += 1
         return tarinfo
 
-# Helper classes for sparse file support
-class _section(object):
-    """Base class for _data and _hole.
-    """
-    def __init__(self, offset, size):
-        self.offset = offset
-        self.size = size
-    def __contains__(self, offset):
-        return self.offset <= offset < self.offset + self.size
-
-class _data(_section):
-    """Represent a data section in a sparse file.
-    """
-    def __init__(self, offset, size, realpos):
-        _section.__init__(self, offset, size)
-        self.realpos = realpos
-
-class _hole(_section):
-    """Represent a hole section in a sparse file.
-    """
-    pass
-
-class _ringbuffer(list):
-    """Ringbuffer class which increases performance
-       over a regular list.
-    """
-    def __init__(self):
-        self.idx = 0
-    def find(self, offset):
-        idx = self.idx
-        while True:
-            item = self[idx]
-            if offset in item:
-                break
-            idx += 1
-            if idx == len(self):
-                idx = 0
-            if idx == self.idx:
-                # End of File
-                return None
-        self.idx = idx
-        return item
-
-#---------------------------------------------
-# zipfile compatible TarFile class
-#---------------------------------------------
-TAR_PLAIN = 0           # zipfile.ZIP_STORED
-TAR_GZIPPED = 8         # zipfile.ZIP_DEFLATED
-class TarFileCompat(object):
-    """TarFile class compatible with standard module zipfile's
-       ZipFile class.
-    """
-    def __init__(self, file, mode="r", compression=TAR_PLAIN):
-        from warnings import warnpy3k
-        warnpy3k("the TarFileCompat class has been removed in Python 3.0",
-                stacklevel=2)
-        if compression == TAR_PLAIN:
-            self.tarfile = TarFile.taropen(file, mode)
-        elif compression == TAR_GZIPPED:
-            self.tarfile = TarFile.gzopen(file, mode)
-        else:
-            raise ValueError("unknown compression constant")
-        if mode[0:1] == "r":
-            members = self.tarfile.getmembers()
-            for m in members:
-                m.filename = m.name
-                m.file_size = m.size
-                m.date_time = time.gmtime(m.mtime)[:6]
-    def namelist(self):
-        return map(lambda m: m.name, self.infolist())
-    def infolist(self):
-        return filter(lambda m: m.type in REGULAR_TYPES,
-                      self.tarfile.getmembers())
-    def printdir(self):
-        self.tarfile.list()
-    def testzip(self):
-        return
-    def getinfo(self, name):
-        return self.tarfile.getmember(name)
-    def read(self, name):
-        return self.tarfile.extractfile(self.tarfile.getmember(name)).read()
-    def write(self, filename, arcname=None, compress_type=None):
-        self.tarfile.add(filename, arcname)
-    def writestr(self, zinfo, bytes):
-        try:
-            from cStringIO import StringIO
-        except ImportError:
-            from StringIO import StringIO
-        import calendar
-        tinfo = TarInfo(zinfo.filename)
-        tinfo.size = len(bytes)
-        tinfo.mtime = calendar.timegm(zinfo.date_time)
-        self.tarfile.addfile(tinfo, StringIO(bytes))
-    def close(self):
-        self.tarfile.close()
-#class TarFileCompat
-
 #--------------------
 # exported functions
 #--------------------
@@ -2561,10 +2593,8 @@
        are able to handle, else return False.
     """
     try:
-        try:
-            t = open(name)
-        finally:
-            t.close()
+        t = open(name)
+        t.close()
         return True
     except TarError:
         return False
diff --git a/distutils2/_backport/tests/test_shutil.py b/distutils2/_backport/tests/test_shutil.py
--- a/distutils2/_backport/tests/test_shutil.py
+++ b/distutils2/_backport/tests/test_shutil.py
@@ -1,12 +1,13 @@
 import os
 import sys
-import tempfile
 import stat
 import tarfile
+import tempfile
 from os.path import splitdrive
 from StringIO import StringIO
 
 from distutils.spawn import find_executable, spawn
+from distutils2.compat import wraps
 from distutils2._backport import shutil
 from distutils2._backport.shutil import (
     _make_tarball, _make_zipfile, make_archive, unpack_archive,
@@ -17,6 +18,7 @@
 from distutils2.tests import unittest, support
 from test.test_support import TESTFN
 
+
 try:
     import bz2
     BZ2_SUPPORTED = True
@@ -43,6 +45,21 @@
 except ImportError:
     ZIP_SUPPORT = find_executable('zip')
 
+def _fake_rename(*args, **kwargs):
+    # Pretend the destination path is on a different filesystem.
+    raise OSError()
+
+def mock_rename(func):
+    @wraps(func)
+    def wrap(*args, **kwargs):
+        try:
+            builtin_rename = os.rename
+            os.rename = _fake_rename
+            return func(*args, **kwargs)
+        finally:
+            os.rename = builtin_rename
+    return wrap
+
 class TestShutil(unittest.TestCase):
 
     def setUp(self):
@@ -266,27 +283,45 @@
             shutil.rmtree(src_dir)
             shutil.rmtree(os.path.dirname(dst_dir))
 
-    @support.skip_unless_symlink
+    @unittest.skipUnless(hasattr(os, 'link'), 'requires os.link')
     def test_dont_copy_file_onto_link_to_itself(self):
+        # Temporarily disable test on Windows.
+        if os.name == 'nt':
+            return
         # bug 851123.
         os.mkdir(TESTFN)
         src = os.path.join(TESTFN, 'cheese')
         dst = os.path.join(TESTFN, 'shop')
         try:
             f = open(src, 'w')
-            f.write('cheddar')
-            f.close()
+            try:
+                f.write('cheddar')
+            finally:
+                f.close()
 
-            if hasattr(os, "link"):
-                os.link(src, dst)
-                self.assertRaises(shutil.Error, shutil.copyfile, src, dst)
-                f = open(src, 'r')
-                try:
-                    self.assertEqual(f.read(), 'cheddar')
-                finally:
-                    f.close()
-                os.remove(dst)
+            os.link(src, dst)
+            self.assertRaises(shutil.Error, shutil.copyfile, src, dst)
+            f = open(src, 'r')
+            try:
+                self.assertEqual(f.read(), 'cheddar')
+            finally:
+                f.close()
+            os.remove(dst)
+        finally:
+            shutil.rmtree(TESTFN, ignore_errors=True)
 
+    @support.skip_unless_symlink
+    def test_dont_copy_file_onto_symlink_to_itself(self):
+        # bug 851123.
+        os.mkdir(TESTFN)
+        src = os.path.join(TESTFN, 'cheese')
+        dst = os.path.join(TESTFN, 'shop')
+        try:
+            f = open(src, 'w')
+            try:
+                f.write('cheddar')
+            finally:
+                f.close()
             # Using `src` here would mean we end up with a symlink pointing
             # to TESTFN/TESTFN/cheese, while it should point at
             # TESTFN/cheese.
@@ -299,10 +334,7 @@
                 f.close()
             os.remove(dst)
         finally:
-            try:
-                shutil.rmtree(TESTFN)
-            except OSError:
-                pass
+            shutil.rmtree(TESTFN, ignore_errors=True)
 
     @support.skip_unless_symlink
     def test_rmtree_on_symlink(self):
@@ -329,26 +361,26 @@
             finally:
                 os.remove(TESTFN)
 
-    @unittest.skipUnless(hasattr(os, 'mkfifo'), 'requires os.mkfifo')
-    def test_copytree_named_pipe(self):
-        os.mkdir(TESTFN)
-        try:
-            subdir = os.path.join(TESTFN, "subdir")
-            os.mkdir(subdir)
-            pipe = os.path.join(subdir, "mypipe")
-            os.mkfifo(pipe)
+        @support.skip_unless_symlink
+        def test_copytree_named_pipe(self):
+            os.mkdir(TESTFN)
             try:
-                shutil.copytree(TESTFN, TESTFN2)
-            except shutil.Error, e:
-                errors = e.args[0]
-                self.assertEqual(len(errors), 1)
-                src, dst, error_msg = errors[0]
-                self.assertEqual("`%s` is a named pipe" % pipe, error_msg)
-            else:
-                self.fail("shutil.Error should have been raised")
-        finally:
-            shutil.rmtree(TESTFN, ignore_errors=True)
-            shutil.rmtree(TESTFN2, ignore_errors=True)
+                subdir = os.path.join(TESTFN, "subdir")
+                os.mkdir(subdir)
+                pipe = os.path.join(subdir, "mypipe")
+                os.mkfifo(pipe)
+                try:
+                    shutil.copytree(TESTFN, TESTFN2)
+                except shutil.Error, e:
+                    errors = e.args[0]
+                    self.assertEqual(len(errors), 1)
+                    src, dst, error_msg = errors[0]
+                    self.assertEqual("`%s` is a named pipe" % pipe, error_msg)
+                else:
+                    self.fail("shutil.Error should have been raised")
+            finally:
+                shutil.rmtree(TESTFN, ignore_errors=True)
+                shutil.rmtree(TESTFN2, ignore_errors=True)
 
     def test_copytree_special_func(self):
 
@@ -363,7 +395,7 @@
             copied.append((src, dst))
 
         shutil.copytree(src_dir, dst_dir, copy_function=_copy)
-        self.assertEquals(len(copied), 2)
+        self.assertEqual(len(copied), 2)
 
     @support.skip_unless_symlink
     def test_copytree_dangling_symlinks(self):
@@ -386,6 +418,41 @@
         shutil.copytree(src_dir, dst_dir, symlinks=True)
         self.assertIn('test.txt', os.listdir(dst_dir))
 
+    def _copy_file(self, method):
+        fname = 'test.txt'
+        tmpdir = self.mkdtemp()
+        self.write_file([tmpdir, fname])
+        file1 = os.path.join(tmpdir, fname)
+        tmpdir2 = self.mkdtemp()
+        method(file1, tmpdir2)
+        file2 = os.path.join(tmpdir2, fname)
+        return (file1, file2)
+
+    @unittest.skipUnless(hasattr(os, 'chmod'), 'requires os.chmod')
+    def test_copy(self):
+        # Ensure that the copied file exists and has the same mode bits.
+        file1, file2 = self._copy_file(shutil.copy)
+        self.assertTrue(os.path.exists(file2))
+        self.assertEqual(os.stat(file1).st_mode, os.stat(file2).st_mode)
+
+    @unittest.skipUnless(hasattr(os, 'chmod'), 'requires os.chmod')
+    @unittest.skipUnless(hasattr(os, 'utime'), 'requires os.utime')
+    def test_copy2(self):
+        # Ensure that the copied file exists and has the same mode and
+        # modification time bits.
+        file1, file2 = self._copy_file(shutil.copy2)
+        self.assertTrue(os.path.exists(file2))
+        file1_stat = os.stat(file1)
+        file2_stat = os.stat(file2)
+        self.assertEqual(file1_stat.st_mode, file2_stat.st_mode)
+        for attr in 'st_atime', 'st_mtime':
+            # The modification times may be truncated in the new file.
+            self.assertLessEqual(getattr(file1_stat, attr),
+                                 getattr(file2_stat, attr) + 1)
+        if hasattr(os, 'chflags') and hasattr(file1_stat, 'st_flags'):
+            self.assertEqual(getattr(file1_stat, 'st_flags'),
+                             getattr(file2_stat, 'st_flags'))
+
     @unittest.skipUnless(zlib, "requires zlib")
     def test_make_tarball(self):
         # creating something to tar
@@ -396,6 +463,8 @@
         self.write_file([tmpdir, 'sub', 'file3'], 'xxx')
 
         tmpdir2 = self.mkdtemp()
+        # force shutil to create the directory
+        os.rmdir(tmpdir2)
         unittest.skipUnless(splitdrive(tmpdir)[0] == splitdrive(tmpdir2)[0],
                             "source and target should be on same drive")
 
@@ -481,7 +550,7 @@
 
         self.assertTrue(os.path.exists(tarball2))
         # let's compare both tarballs
-        self.assertEquals(self._tarinfo(tarball), self._tarinfo(tarball2))
+        self.assertEqual(self._tarinfo(tarball), self._tarinfo(tarball2))
 
         # trying an uncompressed one
         base_name = os.path.join(tmpdir2, 'archive')
@@ -514,6 +583,8 @@
         self.write_file([tmpdir, 'file2'], 'xxx')
 
         tmpdir2 = self.mkdtemp()
+        # force shutil to create the directory
+        os.rmdir(tmpdir2)
         base_name = os.path.join(tmpdir2, 'archive')
         _make_zipfile(base_name, tmpdir)
 
@@ -576,8 +647,8 @@
         archive = tarfile.open(archive_name)
         try:
             for member in archive.getmembers():
-                self.assertEquals(member.uid, 0)
-                self.assertEquals(member.gid, 0)
+                self.assertEqual(member.uid, 0)
+                self.assertEqual(member.gid, 0)
         finally:
             archive.close()
 
@@ -592,7 +663,7 @@
                 make_archive('xxx', 'xxx', root_dir=self.mkdtemp())
             except Exception:
                 pass
-            self.assertEquals(os.getcwd(), current_dir)
+            self.assertEqual(os.getcwd(), current_dir)
         finally:
             unregister_archive_format('xxx')
 
@@ -639,16 +710,24 @@
             # let's try to unpack it now
             unpack_archive(filename, tmpdir2)
             diff = self._compare_dirs(tmpdir, tmpdir2)
-            self.assertEquals(diff, [])
+            self.assertEqual(diff, [])
+
+            # and again, this time with the format specified
+            tmpdir3 = self.mkdtemp()
+            unpack_archive(filename, tmpdir3, format=format)
+            diff = self._compare_dirs(tmpdir, tmpdir3)
+            self.assertEqual(diff, [])
+        self.assertRaises(shutil.ReadError, unpack_archive, TESTFN)
+        self.assertRaises(ValueError, unpack_archive, TESTFN, format='xxx')
 
     def test_unpack_registery(self):
 
         formats = get_unpack_formats()
 
         def _boo(filename, extract_dir, extra):
-            self.assertEquals(extra, 1)
-            self.assertEquals(filename, 'stuff.boo')
-            self.assertEquals(extract_dir, 'xx')
+            self.assertEqual(extra, 1)
+            self.assertEqual(filename, 'stuff.boo')
+            self.assertEqual(extract_dir, 'xx')
 
         register_unpack_format('Boo', ['.boo', '.b2'], _boo, [('extra', 1)])
         unpack_archive('stuff.boo', 'xx')
@@ -665,7 +744,7 @@
 
         # let's leave a clean state
         unregister_unpack_format('Boo2')
-        self.assertEquals(get_unpack_formats(), formats)
+        self.assertEqual(get_unpack_formats(), formats)
 
 
 class TestMove(unittest.TestCase):
@@ -676,15 +755,6 @@
         self.dst_dir = tempfile.mkdtemp()
         self.src_file = os.path.join(self.src_dir, filename)
         self.dst_file = os.path.join(self.dst_dir, filename)
-        # Try to create a dir in the current directory, hoping that it is
-        # not located on the same filesystem as the system tmp dir.
-        try:
-            self.dir_other_fs = tempfile.mkdtemp(
-                dir=os.path.dirname(__file__))
-            self.file_other_fs = os.path.join(self.dir_other_fs,
-                filename)
-        except OSError:
-            self.dir_other_fs = None
         f = open(self.src_file, "wb")
         try:
             f.write("spam")
@@ -692,7 +762,7 @@
             f.close()
 
     def tearDown(self):
-        for d in (self.src_dir, self.dst_dir, self.dir_other_fs):
+        for d in (self.src_dir, self.dst_dir):
             try:
                 if d:
                     shutil.rmtree(d)
@@ -729,21 +799,15 @@
         # Move a file inside an existing dir on the same filesystem.
         self._check_move_file(self.src_file, self.dst_dir, self.dst_file)
 
+    @mock_rename
     def test_move_file_other_fs(self):
         # Move a file to an existing dir on another filesystem.
-        if not self.dir_other_fs:
-            # skip
-            return
-        self._check_move_file(self.src_file, self.file_other_fs,
-            self.file_other_fs)
+        self.test_move_file()
 
+    @mock_rename
     def test_move_file_to_dir_other_fs(self):
         # Move a file to another location on another filesystem.
-        if not self.dir_other_fs:
-            # skip
-            return
-        self._check_move_file(self.src_file, self.dir_other_fs,
-            self.file_other_fs)
+        self.test_move_file_to_dir()
 
     def test_move_dir(self):
         # Move a dir to another location on the same filesystem.
@@ -756,32 +820,20 @@
             except:
                 pass
 
+    @mock_rename
     def test_move_dir_other_fs(self):
         # Move a dir to another location on another filesystem.
-        if not self.dir_other_fs:
-            # skip
-            return
-        dst_dir = tempfile.mktemp(dir=self.dir_other_fs)
-        try:
-            self._check_move_dir(self.src_dir, dst_dir, dst_dir)
-        finally:
-            try:
-                shutil.rmtree(dst_dir)
-            except:
-                pass
+        self.test_move_dir()
 
     def test_move_dir_to_dir(self):
         # Move a dir inside an existing dir on the same filesystem.
         self._check_move_dir(self.src_dir, self.dst_dir,
             os.path.join(self.dst_dir, os.path.basename(self.src_dir)))
 
+    @mock_rename
     def test_move_dir_to_dir_other_fs(self):
         # Move a dir inside an existing dir on another filesystem.
-        if not self.dir_other_fs:
-            # skip
-            return
-        self._check_move_dir(self.src_dir, self.dir_other_fs,
-            os.path.join(self.dir_other_fs, os.path.basename(self.src_dir)))
+        self.test_move_dir_to_dir()
 
     def test_existing_file_inside_dest_dir(self):
         # A file with the same name inside the destination dir already exists.
@@ -932,6 +984,23 @@
         self.assertTrue(srcfile._exited_with[0] is None)
         self.assertTrue(srcfile._raised)
 
+    def test_move_dir_caseinsensitive(self):
+        # Renames a folder to the same name
+        # but a different case.
+
+        self.src_dir = tempfile.mkdtemp()
+        dst_dir = os.path.join(
+                os.path.dirname(self.src_dir),
+                os.path.basename(self.src_dir).upper())
+        self.assertNotEqual(self.src_dir, dst_dir)
+
+        try:
+            shutil.move(self.src_dir, dst_dir)
+            self.assertTrue(os.path.isdir(dst_dir))
+        finally:
+            if os.path.exists(dst_dir):
+                os.rmdir(dst_dir)
+
 
 def test_suite():
     suite = unittest.TestSuite()
diff --git a/distutils2/_backport/tests/test_sysconfig.py b/distutils2/_backport/tests/test_sysconfig.py
--- a/distutils2/_backport/tests/test_sysconfig.py
+++ b/distutils2/_backport/tests/test_sysconfig.py
@@ -1,5 +1,3 @@
-"""Tests for sysconfig."""
-
 import os
 import sys
 import subprocess
@@ -10,29 +8,21 @@
 
 from distutils2._backport import sysconfig
 from distutils2._backport.sysconfig import (
-        _expand_globals, _expand_vars, _get_default_scheme, _subst_vars,
-        get_config_var, get_config_vars, get_path, get_paths, get_platform,
-        get_scheme_names, _main, _SCHEMES)
+    get_paths, get_platform, get_config_vars, get_path, get_path_names,
+    _SCHEMES, _get_default_scheme, _expand_vars, get_scheme_names,
+    get_config_var, _main)
 
 from distutils2.tests import unittest
-from distutils2.tests.support import EnvironRestorer
+from distutils2.tests.support import skip_unless_symlink
+
 from test.test_support import TESTFN, unlink
 
-try:
-    from test.test_support import skip_unless_symlink
-except ImportError:
-    skip_unless_symlink = unittest.skip(
-        'requires test.test_support.skip_unless_symlink')
 
-
-class TestSysConfig(EnvironRestorer, unittest.TestCase):
-
-    restore_environ = ['MACOSX_DEPLOYMENT_TARGET', 'PATH']
+class TestSysConfig(unittest.TestCase):
 
     def setUp(self):
         super(TestSysConfig, self).setUp()
         self.sys_path = sys.path[:]
-        self.makefile = None
         # patching os.uname
         if hasattr(os, 'uname'):
             self.uname = os.uname
@@ -45,17 +35,21 @@
         self.name = os.name
         self.platform = sys.platform
         self.version = sys.version
-        self.maxint = sys.maxint
         self.sep = os.sep
         self.join = os.path.join
         self.isabs = os.path.isabs
         self.splitdrive = os.path.splitdrive
         self._config_vars = copy(sysconfig._CONFIG_VARS)
+        self._added_envvars = []
+        self._changed_envvars = []
+        for var in ('MACOSX_DEPLOYMENT_TARGET', 'PATH'):
+            if var in os.environ:
+                self._changed_envvars.append((var, os.environ[var]))
+            else:
+                self._added_envvars.append(var)
 
     def tearDown(self):
         sys.path[:] = self.sys_path
-        if self.makefile is not None:
-            os.unlink(self.makefile)
         self._cleanup_testfn()
         if self.uname is not None:
             os.uname = self.uname
@@ -64,12 +58,16 @@
         os.name = self.name
         sys.platform = self.platform
         sys.version = self.version
-        sys.maxint = self.maxint
         os.sep = self.sep
         os.path.join = self.join
         os.path.isabs = self.isabs
         os.path.splitdrive = self.splitdrive
         sysconfig._CONFIG_VARS = copy(self._config_vars)
+        for var, value in self._changed_envvars:
+            os.environ[var] = value
+        for var in self._added_envvars:
+            os.environ.pop(var, None)
+
         super(TestSysConfig, self).tearDown()
 
     def _set_uname(self, uname):
@@ -85,19 +83,8 @@
         elif os.path.isdir(path):
             shutil.rmtree(path)
 
-    # TODO use a static list or remove the test
-    #def test_get_path_names(self):
-    #    self.assertEqual(get_path_names(), sysconfig._SCHEME_KEYS)
-
-    def test_nested_var_substitution(self):
-        # Assert that the {curly brace token} expansion pattern will replace
-        # only the inner {something} on nested expressions like {py{something}} on
-        # the first pass.
-
-        # We have no plans to make use of this, but it keeps the option open for
-        # the future, at the cost only of disallowing { itself as a piece of a
-        # substitution key (which would be weird).
-        self.assertEqual(_subst_vars('{py{version}}', {'version': '31'}), '{py31}')
+    def test_get_path_names(self):
+        self.assertEqual(get_path_names(), _SCHEMES.options('posix_prefix'))
 
     def test_get_paths(self):
         scheme = get_paths()
@@ -108,10 +95,10 @@
         self.assertEqual(scheme, wanted)
 
     def test_get_path(self):
-        # xxx make real tests here
+        # XXX make real tests here
         for scheme in _SCHEMES.sections():
             for name, _ in _SCHEMES.items(scheme):
-                get_path(name, scheme)
+                res = get_path(name, scheme)
 
     def test_get_config_vars(self):
         cvars = get_config_vars()
@@ -146,37 +133,43 @@
                        '\n[GCC 4.0.1 (Apple Computer, Inc. build 5341)]')
         sys.platform = 'darwin'
         self._set_uname(('Darwin', 'macziade', '8.11.1',
-                        ('Darwin Kernel Version 8.11.1: '
-                         'Wed Oct 10 18:23:28 PDT 2007; '
-                         'root:xnu-792.25.20~1/RELEASE_I386'), 'PowerPC'))
-        os.environ['MACOSX_DEPLOYMENT_TARGET'] = '10.3'
+                   ('Darwin Kernel Version 8.11.1: '
+                    'Wed Oct 10 18:23:28 PDT 2007; '
+                    'root:xnu-792.25.20~1/RELEASE_I386'), 'PowerPC'))
+        get_config_vars()['MACOSX_DEPLOYMENT_TARGET'] = '10.3'
 
         get_config_vars()['CFLAGS'] = ('-fno-strict-aliasing -DNDEBUG -g '
                                        '-fwrapv -O3 -Wall -Wstrict-prototypes')
 
-        sys.maxint = 2147483647
-        self.assertEqual(get_platform(), 'macosx-10.3-ppc')
-        sys.maxint = 9223372036854775807
-        self.assertEqual(get_platform(), 'macosx-10.3-ppc64')
-
+        maxint = sys.maxint
+        try:
+            sys.maxint = 2147483647
+            self.assertEqual(get_platform(), 'macosx-10.3-ppc')
+            sys.maxint = 9223372036854775807
+            self.assertEqual(get_platform(), 'macosx-10.3-ppc64')
+        finally:
+            sys.maxint = maxint
 
         self._set_uname(('Darwin', 'macziade', '8.11.1',
-                         ('Darwin Kernel Version 8.11.1: '
-                          'Wed Oct 10 18:23:28 PDT 2007; '
-                          'root:xnu-792.25.20~1/RELEASE_I386'), 'i386'))
+                   ('Darwin Kernel Version 8.11.1: '
+                    'Wed Oct 10 18:23:28 PDT 2007; '
+                    'root:xnu-792.25.20~1/RELEASE_I386'), 'i386'))
         get_config_vars()['MACOSX_DEPLOYMENT_TARGET'] = '10.3'
-        os.environ['MACOSX_DEPLOYMENT_TARGET'] = '10.3'
+        get_config_vars()['MACOSX_DEPLOYMENT_TARGET'] = '10.3'
 
         get_config_vars()['CFLAGS'] = ('-fno-strict-aliasing -DNDEBUG -g '
                                        '-fwrapv -O3 -Wall -Wstrict-prototypes')
-
-        sys.maxint = 2147483647
-        self.assertEqual(get_platform(), 'macosx-10.3-i386')
-        sys.maxint = 9223372036854775807
-        self.assertEqual(get_platform(), 'macosx-10.3-x86_64')
+        maxint = sys.maxint
+        try:
+            sys.maxint = 2147483647
+            self.assertEqual(get_platform(), 'macosx-10.3-i386')
+            sys.maxint = 9223372036854775807
+            self.assertEqual(get_platform(), 'macosx-10.3-x86_64')
+        finally:
+            sys.maxint = maxint
 
         # macbook with fat binaries (fat, universal or fat64)
-        os.environ['MACOSX_DEPLOYMENT_TARGET'] = '10.4'
+        get_config_vars()['MACOSX_DEPLOYMENT_TARGET'] = '10.4'
         get_config_vars()['CFLAGS'] = ('-arch ppc -arch i386 -isysroot '
                                        '/Developer/SDKs/MacOSX10.4u.sdk  '
                                        '-fno-strict-aliasing -fno-common '
@@ -214,9 +207,9 @@
             get_config_vars()['CFLAGS'] = ('-arch %s -isysroot '
                                            '/Developer/SDKs/MacOSX10.4u.sdk  '
                                            '-fno-strict-aliasing -fno-common '
-                                           '-dynamic -DNDEBUG -g -O3'%(arch,))
+                                           '-dynamic -DNDEBUG -g -O3' % arch)
 
-            self.assertEqual(get_platform(), 'macosx-10.4-%s'%(arch,))
+            self.assertEqual(get_platform(), 'macosx-10.4-%s' % arch)
 
         # linux debian sarge
         os.name = 'posix'
@@ -234,10 +227,6 @@
         config_h = sysconfig.get_config_h_filename()
         self.assertTrue(os.path.isfile(config_h), config_h)
 
-    def test_get_makefile_filename(self):
-        makefile = sysconfig.get_makefile_filename()
-        self.assertTrue(os.path.isfile(makefile), makefile)
-
     def test_get_scheme_names(self):
         wanted = ('nt', 'nt_user', 'os2', 'os2_home', 'osx_framework_user',
                   'posix_home', 'posix_prefix', 'posix_user')
@@ -268,14 +257,14 @@
 
     @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_user_similar(self):
-        # Issue 8759 : make sure the posix scheme for the users
+        # Issue #8759: make sure the posix scheme for the users
         # is similar to the global posix_prefix one
         base = get_config_var('base')
         user = get_config_var('userbase')
         for name in ('stdlib', 'platstdlib', 'purelib', 'platlib'):
             global_path = get_path(name, 'posix_prefix')
             user_path = get_path(name, 'posix_user')
-            self.assertEqual(user_path, global_path.replace(base, user))
+            self.assertEqual(user_path, global_path.replace(base, user, 1))
 
     def test_main(self):
         # just making sure _main() runs and returns things in the stdout
@@ -291,25 +280,96 @@
 
         self.assertIn(ldflags, ldshared)
 
-    def test_expand_globals(self):
-        config = RawConfigParser()
-        config.add_section('globals')
-        config.set('globals', 'foo', 'ok')
-        config.add_section('posix')
-        config.set('posix', 'config', '/etc')
-        config.set('posix', 'more', '{config}/ok')
+    @unittest.skipUnless(sys.platform == "darwin", "test only relevant on MacOSX")
+    def test_platform_in_subprocess(self):
+        my_platform = sysconfig.get_platform()
 
-        _expand_globals(config)
+        # Test without MACOSX_DEPLOYMENT_TARGET in the environment
 
-        self.assertEqual(config.get('posix', 'foo'), 'ok')
-        self.assertEqual(config.get('posix', 'more'), '/etc/ok')
+        env = os.environ.copy()
+        if 'MACOSX_DEPLOYMENT_TARGET' in env:
+            del env['MACOSX_DEPLOYMENT_TARGET']
 
-        # we might not have globals after all
-        # extending again (==no more globals section)
-        _expand_globals(config)
+        devnull_fp = open('/dev/null', 'w')
+        try:
+            p = subprocess.Popen([
+                    sys.executable, '-c',
+                    'from distutils2._backport import sysconfig; '
+                    'print sysconfig.get_platform()',
+                ],
+                stdout=subprocess.PIPE,
+                stderr=devnull_fp,
+                env=env)
+        finally:
+            fp.close()
+        test_platform = p.communicate()[0].strip()
+        test_platform = test_platform.decode('utf-8')
+        status = p.wait()
+
+        self.assertEqual(status, 0)
+        self.assertEqual(my_platform, test_platform)
+
+        # Test with MACOSX_DEPLOYMENT_TARGET in the environment, and
+        # using a value that is unlikely to be the default one.
+        env = os.environ.copy()
+        env['MACOSX_DEPLOYMENT_TARGET'] = '10.1'
+
+        dev_null = open('/dev/null')
+        try:
+            p = subprocess.Popen([
+                    sys.executable, '-c',
+                    'from distutils2._backport import sysconfig; '
+                    'print sysconfig.get_platform()',
+                ],
+                stdout=subprocess.PIPE,
+                stderr=dev_null,
+                env=env)
+            test_platform = p.communicate()[0].strip()
+            test_platform = test_platform.decode('utf-8')
+            status = p.wait()
+
+            self.assertEqual(status, 0)
+            self.assertEqual(my_platform, test_platform)
+        finally:
+            dev_null.close()
+
+
+class MakefileTests(unittest.TestCase):
+
+    @unittest.skipIf(sys.platform.startswith('win'),
+                     'Test is not Windows compatible')
+    def test_get_makefile_filename(self):
+        makefile = sysconfig.get_makefile_filename()
+        self.assertTrue(os.path.isfile(makefile), makefile)
+
+    def test_parse_makefile(self):
+        self.addCleanup(unlink, TESTFN)
+        makefile = open(TESTFN, "w")
+        try:
+            print >> makefile, "var1=a$(VAR2)"
+            print >> makefile, "VAR2=b$(var3)"
+            print >> makefile, "var3=42"
+            print >> makefile, "var4=$/invalid"
+            print >> makefile, "var5=dollar$$5"
+        finally:
+            makefile.close()
+        vars = sysconfig._parse_makefile(TESTFN)
+        self.assertEqual(vars, {
+            'var1': 'ab42',
+            'VAR2': 'b42',
+            'var3': 42,
+            'var4': '$/invalid',
+            'var5': 'dollar$5',
+        })
+
 
 def test_suite():
-    return unittest.makeSuite(TestSysConfig)
+    suite = unittest.TestSuite()
+    load = unittest.defaultTestLoader.loadTestsFromTestCase
+    suite.addTest(load(TestSysConfig))
+    suite.addTest(load(MakefileTests))
+    return suite
+
 
 if __name__ == '__main__':
     unittest.main(defaultTest='test_suite')
diff --git a/distutils2/command/build_scripts.py b/distutils2/command/build_scripts.py
--- a/distutils2/command/build_scripts.py
+++ b/distutils2/command/build_scripts.py
@@ -3,11 +3,12 @@
 import os
 import re
 
-from distutils2._backport import sysconfig
 from distutils2.command.cmd import Command
-from distutils2.util import convert_path, newer, detect_encoding, fsencode
+from distutils2.util import convert_path, newer
 from distutils2 import logger
 from distutils2.compat import Mixin2to3
+from distutils2.compat import detect_encoding, fsencode
+from distutils2._backport import sysconfig
 
 
 # check if Python is called on the first line with this expression
diff --git a/distutils2/command/cmd.py b/distutils2/command/cmd.py
--- a/distutils2/command/cmd.py
+++ b/distutils2/command/cmd.py
@@ -2,11 +2,10 @@
 
 import os
 import re
-from shutil import copyfile, move
 from distutils2 import util
 from distutils2 import logger
-from distutils2.util import make_archive
 from distutils2.errors import PackagingOptionError
+from distutils2._backport.shutil import copyfile, move, make_archive
 
 
 class Command(object):
diff --git a/distutils2/command/sdist.py b/distutils2/command/sdist.py
--- a/distutils2/command/sdist.py
+++ b/distutils2/command/sdist.py
@@ -4,15 +4,15 @@
 import re
 import sys
 from StringIO import StringIO
-from shutil import rmtree
 
 from distutils2 import logger
-from distutils2.util import resolve_name, get_archive_formats
+from distutils2.util import resolve_name
 from distutils2.errors import (PackagingPlatformError, PackagingOptionError,
                                PackagingModuleError, PackagingFileError)
 from distutils2.command import get_command_names
 from distutils2.command.cmd import Command
 from distutils2.manifest import Manifest
+from distutils2._backport.shutil import get_archive_formats, rmtree
 
 
 def show_formats():
diff --git a/distutils2/compat.py b/distutils2/compat.py
--- a/distutils2/compat.py
+++ b/distutils2/compat.py
@@ -1,17 +1,18 @@
 """Compatibility helpers.
 
-This module provides classes, variables and imports which are used to
-support distutils2 across Python 2.x and 3.x.
+This module provides individual classes or objects backported from
+Python 3.2, for internal use only.  Whole modules are in _backport.
 """
 
+import re
+import sys
+import codecs
 from distutils2 import logger
 
 
 # XXX Having two classes with the same name is not a good thing.
 # XXX 2to3-related code should move from util to this module
 
-# TODO Move common code here: PY3 (bool indicating if we're on 3.x), any, etc.
-
 try:
     from distutils2.util import Mixin2to3 as _Mixin2to3
     _CONVERT = True
@@ -55,3 +56,125 @@
 
         def _run_2to3(self, files, doctests=[], fixers=[]):
             pass
+
+
+# The rest of this file does not exist in packaging
+# functions are sorted alphabetically and are not included in __all__
+
+try:
+    any
+except NameError:
+    def any(seq):
+        for elem in seq:
+            if elem:
+                return True
+        return False
+
+
+_cookie_re = re.compile("coding[:=]\s*([-\w.]+)")
+
+def _get_normal_name(orig_enc):
+    """Imitates get_normal_name in tokenizer.c."""
+    # Only care about the first 12 characters.
+    enc = orig_enc[:12].lower().replace("_", "-")
+    if enc == "utf-8" or enc.startswith("utf-8-"):
+        return "utf-8"
+    if enc in ("latin-1", "iso-8859-1", "iso-latin-1") or \
+       enc.startswith(("latin-1-", "iso-8859-1-", "iso-latin-1-")):
+        return "iso-8859-1"
+    return orig_enc
+
+def detect_encoding(readline):
+    """
+    The detect_encoding() function is used to detect the encoding that should
+    be used to decode a Python source file.  It requires one argment, readline,
+    in the same way as the tokenize() generator.
+
+    It will call readline a maximum of twice, and return the encoding used
+    (as a string) and a list of any lines (left as bytes) it has read in.
+
+    It detects the encoding from the presence of a utf-8 bom or an encoding
+    cookie as specified in pep-0263.  If both a bom and a cookie are present,
+    but disagree, a SyntaxError will be raised.  If the encoding cookie is an
+    invalid charset, raise a SyntaxError.  Note that if a utf-8 bom is found,
+    'utf-8-sig' is returned.
+
+    If no encoding is specified, then the default of 'utf-8' will be returned.
+    """
+    bom_found = False
+    encoding = None
+    default = 'utf-8'
+    def read_or_stop():
+        try:
+            return readline()
+        except StopIteration:
+            return ''
+
+    def find_cookie(line):
+        try:
+            line_string = line.decode('ascii')
+        except UnicodeDecodeError:
+            return None
+
+        matches = _cookie_re.findall(line_string)
+        if not matches:
+            return None
+        encoding = _get_normal_name(matches[0])
+        try:
+            codec = codecs.lookup(encoding)
+        except LookupError:
+            # This behaviour mimics the Python interpreter
+            raise SyntaxError("unknown encoding: " + encoding)
+
+        if bom_found:
+            if codec.name != 'utf-8':
+                # This behaviour mimics the Python interpreter
+                raise SyntaxError('encoding problem: utf-8')
+            encoding += '-sig'
+        return encoding
+
+    first = read_or_stop()
+    if first.startswith(codecs.BOM_UTF8):
+        bom_found = True
+        first = first[3:]
+        default = 'utf-8-sig'
+    if not first:
+        return default, []
+
+    encoding = find_cookie(first)
+    if encoding:
+        return encoding, [first]
+
+    second = read_or_stop()
+    if not second:
+        return default, [first]
+
+    encoding = find_cookie(second)
+    if encoding:
+        return encoding, [first, second]
+
+    return default, [first, second]
+
+
+def fsencode(filename):
+    """
+    Encode filename to the filesystem encoding with 'surrogateescape' error
+    handler, return bytes unchanged. On Windows, use 'strict' error handler if
+    the file system encoding is 'mbcs' (which is the default encoding).
+    """
+    if isinstance(filename, str):
+        return filename
+    elif isinstance(filename, unicode):
+        return filename.encode(sys.getfilesystemencoding())
+    else:
+        raise TypeError("expect bytes or str, not %s" % type(filename).__name__)
+
+
+try:
+    from functools import wraps
+except ImportError:
+    def wraps(func=None):
+        """No-op replacement for functools.wraps"""
+        def wrapped(func):
+            return func
+        return wrapped
diff --git a/distutils2/create.py b/distutils2/create.py
--- a/distutils2/create.py
+++ b/distutils2/create.py
@@ -27,16 +27,17 @@
 import shutil
 from textwrap import dedent
 from ConfigParser import RawConfigParser
-from distutils2.util import cmp_to_key, detect_encoding
+
 # importing this with an underscore as it should be replaced by the
 # dict form or another structures for all purposes
 from distutils2._trove import all_classifiers as _CLASSIFIERS_LIST
+from distutils2.compat import detect_encoding
 from distutils2.version import is_valid_version
 from distutils2._backport import sysconfig
 try:
     any
 except NameError:
-    from distutils2._backport import any
+    from distutils2.compat import any
 try:
     from hashlib import md5
 except ImportError:
@@ -367,8 +368,9 @@
                       ('description', 'summary'),
                       ('long_description', 'description'),
                       ('url', 'home_page'),
-                      ('platforms', 'platform'),
-                      # backport only for 2.5+
+                      ('platforms', 'platform'))
+            if sys.version >= '2.5':
+                labels += (
                       ('provides', 'provides-dist'),
                       ('obsoletes', 'obsoletes-dist'),
                       ('requires', 'requires-dist'))
@@ -388,21 +390,9 @@
                     dist.data_files = [('', dist.data_files)]
                 # add tokens in the destination paths
                 vars = {'distribution.name': data['name']}
-                path_tokens = list(sysconfig.get_paths(vars=vars).items())
-
-                # TODO replace this with a key function
-                def length_comparison(x, y):
-                    len_x = len(x[1])
-                    len_y = len(y[1])
-                    if len_x == len_y:
-                        return 0
-                    elif len_x < len_y:
-                        return -1
-                    else:
-                        return 1
-
+                path_tokens = sysconfig.get_paths(vars=vars).items()
                 # sort tokens to use the longest one first
-                path_tokens.sort(key=cmp_to_key(length_comparison))
+                path_tokens = sorted(path_tokens, key=lambda x: len(x[1]))
                 for dest, srcs in (dist.data_files or []):
                     dest = os.path.join(sys.prefix, dest)
                     dest = dest.replace(os.path.sep, '/')
diff --git a/distutils2/install.py b/distutils2/install.py
--- a/distutils2/install.py
+++ b/distutils2/install.py
@@ -17,7 +17,7 @@
 from distutils2 import logger
 from distutils2.dist import Distribution
 from distutils2.util import (_is_archive_file, ask, get_install_method,
-                             egginfo_to_distinfo, unpack_archive)
+                             egginfo_to_distinfo)
 from distutils2.pypi import wrapper
 from distutils2.version import get_version_predicate
 from distutils2.database import get_distributions, get_distribution
@@ -27,6 +27,7 @@
                                InstallationConflict, CCompilerError)
 from distutils2.pypi.errors import ProjectNotFound, ReleaseNotFound
 from distutils2 import database
+from distutils2._backport.shutil import unpack_archive
 from distutils2._backport.sysconfig import (get_config_var, get_path,
                                             is_python_build)
 
diff --git a/distutils2/pypi/dist.py b/distutils2/pypi/dist.py
--- a/distutils2/pypi/dist.py
+++ b/distutils2/pypi/dist.py
@@ -8,20 +8,20 @@
 """
 
 import re
-try:
-    import hashlib
-except ImportError:
-    from distutils2._backport import hashlib
 import tempfile
 import urllib
 import urlparse
 from distutils2.errors import IrrationalVersionError
 from distutils2.version import (suggest_normalized_version, NormalizedVersion,
-                               get_version_predicate)
+                                get_version_predicate)
 from distutils2.metadata import Metadata
 from distutils2.pypi.errors import (HashDoesNotMatch, UnsupportedHashName,
-                                   CantParseArchiveName)
-from distutils2.util import unpack_archive
+                                    CantParseArchiveName)
+from distutils2._backport.shutil import unpack_archive
+try:
+    import hashlib
+except ImportError:
+    from distutils2._backport import hashlib
 
 
 __all__ = ['ReleaseInfo', 'DistInfo', 'ReleasesList', 'get_infos_from_url']
diff --git a/distutils2/pypi/simple.py b/distutils2/pypi/simple.py
--- a/distutils2/pypi/simple.py
+++ b/distutils2/pypi/simple.py
@@ -15,11 +15,8 @@
 import os
 
 from fnmatch import translate
-try:
-    from functools import wraps
-except ImportError:
-    from distutils2._backport.functools import wraps
 from distutils2 import logger
+from distutils2.compat import wraps
 from distutils2.metadata import Metadata
 from distutils2.version import get_version_predicate
 from distutils2 import __version__ as distutils2_version
diff --git a/distutils2/tests/pypi_server.py b/distutils2/tests/pypi_server.py
--- a/distutils2/tests/pypi_server.py
+++ b/distutils2/tests/pypi_server.py
@@ -39,11 +39,8 @@
 from SimpleXMLRPCServer import SimpleXMLRPCServer
 
 from distutils2.tests import unittest
+from distutils2.compat import wraps
 
-try:
-    from functools import wraps
-except ImportError:
-    from distutils2._backport.functools import wraps
 
 
 PYPI_DEFAULT_STATIC_PATH = os.path.join(
diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -384,7 +384,7 @@
     if not _thread:
         return func
 
-    @functools.wraps(func)
+    @wraps(func)
     def decorator(*args):
         key = threading_setup()
         try:
diff --git a/distutils2/tests/test_command_sdist.py b/distutils2/tests/test_command_sdist.py
--- a/distutils2/tests/test_command_sdist.py
+++ b/distutils2/tests/test_command_sdist.py
@@ -20,8 +20,9 @@
 from distutils2.dist import Distribution
 from distutils2.tests import unittest
 from distutils2.errors import PackagingOptionError
-from distutils2.util import find_executable, get_archive_formats
+from distutils2.util import find_executable
 from distutils2.tests import support
+from distutils2._backport.shutil import get_archive_formats
 
 
 MANIFEST = """\
diff --git a/distutils2/tests/test_util.py b/distutils2/tests/test_util.py
--- a/distutils2/tests/test_util.py
+++ b/distutils2/tests/test_util.py
@@ -15,7 +15,7 @@
 from distutils2 import util
 from distutils2.dist import Distribution
 from distutils2.util import (
-    convert_path, change_root, split_quoted, strtobool, rfc822_escape, run_2to3,
+    convert_path, change_root, split_quoted, strtobool, run_2to3,
     get_compiler_versions, _MAC_OS_X_LD_VERSION, byte_compile, find_packages,
     spawn, get_pypirc_path, generate_pypirc, read_pypirc, resolve_name, iglob,
     RICH_GLOB, egginfo_to_distinfo, is_setuptools, is_distutils, is_packaging,
@@ -255,13 +255,6 @@
         for n in no:
             self.assertFalse(strtobool(n))
 
-    def test_rfc822_escape(self):
-        header = 'I am a\npoor\nlonesome\nheader\n'
-        res = rfc822_escape(header)
-        wanted = ('I am a%(8s)spoor%(8s)slonesome%(8s)s'
-                  'header%(8s)s') % {'8s': '\n' + 8 * ' '}
-        self.assertEqual(res, wanted)
-
     def test_find_exe_version(self):
         # the ld version scheme under MAC OS is:
         #   ^@(#)PROGRAM:ld  PROJECT:ld64-VERSION
diff --git a/distutils2/util.py b/distutils2/util.py
--- a/distutils2/util.py
+++ b/distutils2/util.py
@@ -8,8 +8,6 @@
 import codecs
 import shutil
 import string
-import tarfile
-import zipfile
 import posixpath
 import subprocess
 from fnmatch import fnmatchcase
@@ -30,6 +28,30 @@
                                InstallationException, PackagingInternalError)
 from distutils2._backport import sysconfig
 
+__all__ = [
+    # file dependencies
+    'newer', 'newer_group',
+    # helpers for commands (dry-run system)
+    'execute', 'write_file',
+    # spawning programs
+    'find_executable', 'spawn',
+    # path manipulation
+    'convert_path', 'change_root',
+    # 2to3 conversion
+    'Mixin2to3', 'run_2to3',
+    # packaging compatibility helpers
+    'cfg_to_args', 'generate_setup_py',
+    'egginfo_to_distinfo',
+    'get_install_method',
+    # misc
+    'ask', 'check_environ', 'encode_multipart', 'resolve_name',
+    # querying for information  TODO move to sysconfig
+    'get_compiler_versions', 'get_platform', 'set_platform',
+    # configuration  TODO move to packaging.config
+    'get_pypirc_path', 'read_pypirc', 'generate_pypirc',
+    'strtobool', 'split_multiline',
+]
+
 _PLATFORM = None
 _DEFAULT_INSTALLER = 'distutils2'
 
@@ -159,31 +181,6 @@
     _environ_checked = True
 
 
-def subst_vars(s, local_vars):
-    """Perform shell/Perl-style variable substitution on 'string'.
-
-    Every occurrence of '$' followed by a name is considered a variable, and
-    variable is substituted by the value found in the 'local_vars'
-    dictionary, or in 'os.environ' if it's not in 'local_vars'.
-    'os.environ' is first checked/augmented to guarantee that it contains
-    certain values: see 'check_environ()'.  Raise ValueError for any
-    variables not found in either 'local_vars' or 'os.environ'.
-    """
-    check_environ()
-
-    def _subst(match, local_vars=local_vars):
-        var_name = match.group(1)
-        if var_name in local_vars:
-            return str(local_vars[var_name])
-        else:
-            return os.environ[var_name]
-
-    try:
-        return re.sub(r'\$([a-zA-Z_][a-zA-Z_0-9]*)', _subst, s)
-    except KeyError, e:
-        raise ValueError("invalid variable '$%s'" % e)
-
-
 # Needed by 'split_quoted()'
 _wordchars_re = _squote_re = _dquote_re = None
 
@@ -195,6 +192,8 @@
     _dquote_re = re.compile(r'"(?:[^"\\]|\\.)*"')
 
 
+# TODO replace with shlex.split after testing
+
 def split_quoted(s):
     """Split a string up according to Unix shell-like rules for quotes and
     backslashes.
@@ -446,15 +445,6 @@
                               file, cfile_base)
 
 
-def rfc822_escape(header):
-    """Return a form of *header* suitable for inclusion in an RFC 822-header.
-
-    This function ensures there are 8 spaces after each newline.
-    """
-    lines = header.split('\n')
-    sep = '\n' + 8 * ' '
-    return sep.join(lines)
-
 _RE_VERSION = re.compile('(\d+\.\d+(\.\d+)*)')
 _MAC_OS_X_LD_VERSION = re.compile('^@\(#\)PROGRAM:ld  '
                                   'PROJECT:ld64-((\d+)(\.\d+)*)')
@@ -554,6 +544,10 @@
     """Create *filename* and write *contents* to it.
 
     *contents* is a sequence of strings without line terminators.
+
+    This functions is not intended to replace the usual with open + write
+    idiom in all cases, only with Command.execute, which runs depending on
+    the dry_run argument and also logs its arguments).
     """
     f = open(filename, "w")
     try:
@@ -575,6 +569,7 @@
 
 
 def _under(path, root):
+    # XXX use os.path
     path = path.split(os.sep)
     root = root.split(os.sep)
     if len(root) > len(path):
@@ -677,105 +672,6 @@
     return base, ext
 
 
-def unzip_file(filename, location, flatten=True):
-    """Unzip the file *filename* into the *location* directory."""
-    if not os.path.exists(location):
-        os.makedirs(location)
-    zipfp = open(filename, 'rb')
-
-    zip = zipfile.ZipFile(zipfp)
-    leading = has_leading_dir(zip.namelist()) and flatten
-    for name in zip.namelist():
-        data = zip.read(name)
-        fn = name
-        if leading:
-            fn = split_leading_dir(name)[1]
-        fn = os.path.join(location, fn)
-        dir = os.path.dirname(fn)
-        if not os.path.exists(dir):
-            os.makedirs(dir)
-        if fn.endswith('/') or fn.endswith('\\'):
-            # A directory
-            if not os.path.exists(fn):
-                os.makedirs(fn)
-        else:
-            fp = open(fn, 'wb')
-            fp.write(data)
-            fp.close()
-    zipfp.close()
-
-
-def untar_file(filename, location):
-    """Untar the file *filename* into the *location* directory."""
-    if not os.path.exists(location):
-        os.makedirs(location)
-    if filename.lower().endswith('.gz') or filename.lower().endswith('.tgz'):
-        mode = 'r:gz'
-    elif (filename.lower().endswith('.bz2')
-          or filename.lower().endswith('.tbz')):
-        mode = 'r:bz2'
-    elif filename.lower().endswith('.tar'):
-        mode = 'r'
-    else:
-        mode = 'r:*'
-
-    tar = tarfile.open(filename, mode)
-    leading = has_leading_dir(member.name for member in tar.getmembers())
-    for member in tar.getmembers():
-        fn = member.name
-        if leading:
-            fn = split_leading_dir(fn)[1]
-        path = os.path.join(location, fn)
-        if member.isdir():
-            if not os.path.exists(path):
-                os.makedirs(path)
-        else:
-            try:
-                fp = tar.extractfile(member)
-            except (KeyError, AttributeError):
-                # Some corrupt tar files seem to produce this
-                # (specifically bad symlinks)
-                continue
-            try:
-                if not os.path.exists(os.path.dirname(path)):
-                    os.makedirs(os.path.dirname(path))
-                    destfp = open(path, 'wb')
-                    shutil.copyfileobj(fp, destfp)
-                    destfp.close()
-            except:
-                fp.close()
-                raise
-            fp.close()
-
-def has_leading_dir(paths):
-    """Return true if all the paths have the same leading path name.
-
-    In other words, check that everything is in one subdirectory in an
-    archive.
-    """
-    common_prefix = None
-    for path in paths:
-        prefix, rest = split_leading_dir(path)
-        if not prefix:
-            return False
-        elif common_prefix is None:
-            common_prefix = prefix
-        elif prefix != common_prefix:
-            return False
-    return True
-
-
-def split_leading_dir(path):
-    path = str(path)
-    path = path.lstrip('/').lstrip('\\')
-    if '/' in path and (('\\' in path and path.find('/') < path.find('\\'))
-                        or '\\' not in path):
-        return path.split('/', 1)
-    elif '\\' in path:
-        return path.split('\\', 1)
-    else:
-        return path, ''
-
 if sys.platform == 'darwin':
     _cfg_target = None
     _cfg_target_split = None
@@ -1563,11 +1459,12 @@
     for key, values in fields:
         # handle multiple entries for the same name
         if not isinstance(values, (tuple, list)):
-            values=[values]
+            values = [values]
 
         for value in values:
             l.extend((
                 '--' + boundary,
+                # XXX should encode to match packaging but it causes bugs
                 ('Content-Disposition: form-data; name="%s"' % key), '', value))
 
     for key, filename, value in files:
@@ -1584,561 +1481,3 @@
     body = '\r\n'.join(l)
     content_type = 'multipart/form-data; boundary=' + boundary
     return content_type, body
-
-# shutil stuff
-
-try:
-    import bz2
-    _BZ2_SUPPORTED = True
-except ImportError:
-    _BZ2_SUPPORTED = False
-
-try:
-    from pwd import getpwnam
-except ImportError:
-    getpwnam = None
-
-try:
-    from grp import getgrnam
-except ImportError:
-    getgrnam = None
-
-def _get_gid(name):
-    """Returns a gid, given a group name."""
-    if getgrnam is None or name is None:
-        return None
-    try:
-        result = getgrnam(name)
-    except KeyError:
-        result = None
-    if result is not None:
-        return result[2]
-    return None
-
-def _get_uid(name):
-    """Returns an uid, given a user name."""
-    if getpwnam is None or name is None:
-        return None
-    try:
-        result = getpwnam(name)
-    except KeyError:
-        result = None
-    if result is not None:
-        return result[2]
-    return None
-
-def _make_tarball(base_name, base_dir, compress="gzip", verbose=0, dry_run=0,
-                  owner=None, group=None, logger=None):
-    """Create a (possibly compressed) tar file from all the files under
-    'base_dir'.
-
-    'compress' must be "gzip" (the default), "bzip2", or None.
-
-    'owner' and 'group' can be used to define an owner and a group for the
-    archive that is being built. If not provided, the current owner and group
-    will be used.
-
-    The output tar file will be named 'base_name' +  ".tar", possibly plus
-    the appropriate compression extension (".gz", or ".bz2").
-
-    Returns the output filename.
-    """
-    tar_compression = {'gzip': 'gz', None: ''}
-    compress_ext = {'gzip': '.gz'}
-
-    if _BZ2_SUPPORTED:
-        tar_compression['bzip2'] = 'bz2'
-        compress_ext['bzip2'] = '.bz2'
-
-    # flags for compression program, each element of list will be an argument
-    if compress is not None and compress not in compress_ext.keys():
-        raise ValueError("bad value for 'compress', or compression format not "
-                         "supported : %s" % compress)
-
-    archive_name = base_name + '.tar' + compress_ext.get(compress, '')
-    archive_dir = os.path.dirname(archive_name)
-
-    if not os.path.exists(archive_dir):
-        if logger is not None:
-            logger.info("creating %s" % archive_dir)
-        if not dry_run:
-            os.makedirs(archive_dir)
-
-    # creating the tarball
-    if logger is not None:
-        logger.info('Creating tar archive')
-
-    uid = _get_uid(owner)
-    gid = _get_gid(group)
-
-    def _set_uid_gid(tarinfo):
-        if gid is not None:
-            tarinfo.gid = gid
-            tarinfo.gname = group
-        if uid is not None:
-            tarinfo.uid = uid
-            tarinfo.uname = owner
-        return tarinfo
-
-    if not dry_run:
-        tar = tarfile.open(archive_name, 'w|%s' % tar_compression[compress])
-        try:
-            #tar.add(base_dir, filter=_set_uid_gid)
-            tar.add(base_dir)
-        finally:
-            tar.close()
-
-    return archive_name
-
-def _call_external_zip(base_dir, zip_filename, verbose=False, dry_run=False):
-    # XXX see if we want to keep an external call here
-    if verbose:
-        zipoptions = "-r"
-    else:
-        zipoptions = "-rq"
-    from distutils.errors import DistutilsExecError
-    from distutils.spawn import spawn
-    try:
-        spawn(["zip", zipoptions, zip_filename, base_dir], dry_run=dry_run)
-    except DistutilsExecError:
-        # XXX really should distinguish between "couldn't find
-        # external 'zip' command" and "zip failed".
-        raise ExecError("unable to create zip file '%s': "
-            "could neither import the 'zipfile' module nor "
-            "find a standalone zip utility") % zip_filename
-
-def _make_zipfile(base_name, base_dir, verbose=0, dry_run=0, logger=None):
-    """Create a zip file from all the files under 'base_dir'.
-
-    The output zip file will be named 'base_name' + ".zip".  Uses either the
-    "zipfile" Python module (if available) or the InfoZIP "zip" utility
-    (if installed and found on the default search path).  If neither tool is
-    available, raises ExecError.  Returns the name of the output zip
-    file.
-    """
-    zip_filename = base_name + ".zip"
-    archive_dir = os.path.dirname(base_name)
-
-    if not os.path.exists(archive_dir):
-        if logger is not None:
-            logger.info("creating %s", archive_dir)
-        if not dry_run:
-            os.makedirs(archive_dir)
-
-    # If zipfile module is not available, try spawning an external 'zip'
-    # command.
-    try:
-        import zipfile
-    except ImportError:
-        zipfile = None
-
-    if zipfile is None:
-        _call_external_zip(base_dir, zip_filename, verbose, dry_run)
-    else:
-        if logger is not None:
-            logger.info("creating '%s' and adding '%s' to it",
-                        zip_filename, base_dir)
-
-        if not dry_run:
-            zip = zipfile.ZipFile(zip_filename, "w",
-                                  compression=zipfile.ZIP_DEFLATED)
-
-            for dirpath, dirnames, filenames in os.walk(base_dir):
-                for name in filenames:
-                    path = os.path.normpath(os.path.join(dirpath, name))
-                    if os.path.isfile(path):
-                        zip.write(path, path)
-                        if logger is not None:
-                            logger.info("adding '%s'", path)
-            zip.close()
-
-    return zip_filename
-
-_ARCHIVE_FORMATS = {
-    'gztar': (_make_tarball, [('compress', 'gzip')], "gzip'ed tar-file"),
-    'bztar': (_make_tarball, [('compress', 'bzip2')], "bzip2'ed tar-file"),
-    'tar':   (_make_tarball, [('compress', None)], "uncompressed tar file"),
-    'zip':   (_make_zipfile, [],"ZIP file")
-    }
-
-if _BZ2_SUPPORTED:
-    _ARCHIVE_FORMATS['bztar'] = (_make_tarball, [('compress', 'bzip2')],
-                                "bzip2'ed tar-file")
-
-def get_archive_formats():
-    """Returns a list of supported formats for archiving and unarchiving.
-
-    Each element of the returned sequence is a tuple (name, description)
-    """
-    formats = [(name, registry[2]) for name, registry in
-               _ARCHIVE_FORMATS.items()]
-    formats.sort()
-    return formats
-
-def register_archive_format(name, function, extra_args=None, description=''):
-    """Registers an archive format.
-
-    name is the name of the format. function is the callable that will be
-    used to create archives. If provided, extra_args is a sequence of
-    (name, value) tuples that will be passed as arguments to the callable.
-    description can be provided to describe the format, and will be returned
-    by the get_archive_formats() function.
-    """
-    if extra_args is None:
-        extra_args = []
-    if not isinstance(function, collections.Callable):
-        raise TypeError('The %s object is not callable' % function)
-    if not isinstance(extra_args, (tuple, list)):
-        raise TypeError('extra_args needs to be a sequence')
-    for element in extra_args:
-        if not isinstance(element, (tuple, list)) or len(element) !=2 :
-            raise TypeError('extra_args elements are : (arg_name, value)')
-
-    _ARCHIVE_FORMATS[name] = (function, extra_args, description)
-
-def unregister_archive_format(name):
-    del _ARCHIVE_FORMATS[name]
-
-def make_archive(base_name, format, root_dir=None, base_dir=None, verbose=0,
-                 dry_run=0, owner=None, group=None, logger=None):
-    """Create an archive file (eg. zip or tar).
-
-    'base_name' is the name of the file to create, minus any format-specific
-    extension; 'format' is the archive format: one of "zip", "tar", "bztar"
-    or "gztar".
-
-    'root_dir' is a directory that will be the root directory of the
-    archive; ie. we typically chdir into 'root_dir' before creating the
-    archive.  'base_dir' is the directory where we start archiving from;
-    ie. 'base_dir' will be the common prefix of all files and
-    directories in the archive.  'root_dir' and 'base_dir' both default
-    to the current directory.  Returns the name of the archive file.
-
-    'owner' and 'group' are used when creating a tar archive. By default,
-    uses the current owner and group.
-    """
-    save_cwd = os.getcwd()
-    base_name = fsencode(base_name)
-    if root_dir is not None:
-        if logger is not None:
-            logger.debug("changing into '%s'", root_dir)
-        base_name = os.path.abspath(base_name)
-        if not dry_run:
-            os.chdir(root_dir)
-
-    if base_dir is None:
-        base_dir = os.curdir
-
-    kwargs = {'dry_run': dry_run, 'logger': logger}
-
-    try:
-        format_info = _ARCHIVE_FORMATS[format]
-    except KeyError:
-        raise ValueError("unknown archive format '%s'" % format)
-
-    func = format_info[0]
-    for arg, val in format_info[1]:
-        kwargs[arg] = val
-
-    if format != 'zip':
-        kwargs['owner'] = owner
-        kwargs['group'] = group
-
-    try:
-        filename = func(base_name, base_dir, **kwargs)
-    finally:
-        if root_dir is not None:
-            if logger is not None:
-                logger.debug("changing back to '%s'", save_cwd)
-            os.chdir(save_cwd)
-
-    return filename
-
-
-def get_unpack_formats():
-    """Returns a list of supported formats for unpacking.
-
-    Each element of the returned sequence is a tuple
-    (name, extensions, description)
-    """
-    formats = [(name, info[0], info[3]) for name, info in
-               _UNPACK_FORMATS.items()]
-    formats.sort()
-    return formats
-
-def _check_unpack_options(extensions, function, extra_args):
-    """Checks what gets registered as an unpacker."""
-    # first make sure no other unpacker is registered for this extension
-    existing_extensions = {}
-    for name, info in _UNPACK_FORMATS.items():
-        for ext in info[0]:
-            existing_extensions[ext] = name
-
-    for extension in extensions:
-        if extension in existing_extensions:
-            msg = '%s is already registered for "%s"'
-            raise RegistryError(msg % (extension,
-                                       existing_extensions[extension]))
-
-    if not isinstance(function, collections.Callable):
-        raise TypeError('The registered function must be a callable')
-
-
-def register_unpack_format(name, extensions, function, extra_args=None,
-                           description=''):
-    """Registers an unpack format.
-
-    `name` is the name of the format. `extensions` is a list of extensions
-    corresponding to the format.
-
-    `function` is the callable that will be
-    used to unpack archives. The callable will receive archives to unpack.
-    If it's unable to handle an archive, it needs to raise a ReadError
-    exception.
-
-    If provided, `extra_args` is a sequence of
-    (name, value) tuples that will be passed as arguments to the callable.
-    description can be provided to describe the format, and will be returned
-    by the get_unpack_formats() function.
-    """
-    if extra_args is None:
-        extra_args = []
-    _check_unpack_options(extensions, function, extra_args)
-    _UNPACK_FORMATS[name] = extensions, function, extra_args, description
-
-def unregister_unpack_format(name):
-    """Removes the pack format from the registery."""
-    del _UNPACK_FORMATS[name]
-
-def _ensure_directory(path):
-    """Ensure that the parent directory of `path` exists"""
-    dirname = os.path.dirname(path)
-    if not os.path.isdir(dirname):
-        os.makedirs(dirname)
-
-def _unpack_zipfile(filename, extract_dir):
-    """Unpack zip `filename` to `extract_dir`
-    """
-    try:
-        import zipfile
-    except ImportError:
-        raise ReadError('zlib not supported, cannot unpack this archive.')
-
-    if not zipfile.is_zipfile(filename):
-        raise ReadError("%s is not a zip file" % filename)
-
-    zip = zipfile.ZipFile(filename)
-    try:
-        for info in zip.infolist():
-            name = info.filename
-
-            # don't extract absolute paths or ones with .. in them
-            if name.startswith('/') or '..' in name:
-                continue
-
-            target = os.path.join(extract_dir, *name.split('/'))
-            if not target:
-                continue
-
-            _ensure_directory(target)
-            if not name.endswith('/'):
-                # file
-                data = zip.read(info.filename)
-                f = open(target,'wb')
-                try:
-                    f.write(data)
-                finally:
-                    f.close()
-                    del data
-    finally:
-        zip.close()
-
-def _unpack_tarfile(filename, extract_dir):
-    """Unpack tar/tar.gz/tar.bz2 `filename` to `extract_dir`
-    """
-    try:
-        tarobj = tarfile.open(filename)
-    except tarfile.TarError:
-        raise ReadError(
-            "%s is not a compressed or uncompressed tar file" % filename)
-    try:
-        tarobj.extractall(extract_dir)
-    finally:
-        tarobj.close()
-
-_UNPACK_FORMATS = {
-    'gztar': (['.tar.gz', '.tgz'], _unpack_tarfile, [], "gzip'ed tar-file"),
-    'tar':   (['.tar'], _unpack_tarfile, [], "uncompressed tar file"),
-    'zip':   (['.zip'], _unpack_zipfile, [], "ZIP file")
-    }
-
-if _BZ2_SUPPORTED:
-    _UNPACK_FORMATS['bztar'] = (['.bz2'], _unpack_tarfile, [],
-                                "bzip2'ed tar-file")
-
-def _find_unpack_format(filename):
-    for name, info in _UNPACK_FORMATS.items():
-        for extension in info[0]:
-            if filename.endswith(extension):
-                return name
-    return None
-
-def unpack_archive(filename, extract_dir=None, format=None):
-    """Unpack an archive.
-
-    `filename` is the name of the archive.
-
-    `extract_dir` is the name of the target directory, where the archive
-    is unpacked. If not provided, the current working directory is used.
-
-    `format` is the archive format: one of "zip", "tar", or "gztar". Or any
-    other registered format. If not provided, unpack_archive will use the
-    filename extension and see if an unpacker was registered for that
-    extension.
-
-    In case none is found, a ValueError is raised.
-    """
-    if extract_dir is None:
-        extract_dir = os.getcwd()
-
-    if format is not None:
-        try:
-            format_info = _UNPACK_FORMATS[format]
-        except KeyError:
-            raise ValueError("Unknown unpack format '%s'" % format)
-
-        func = format_info[1]
-        func(filename, extract_dir, **dict(format_info[2]))
-    else:
-        # we need to look at the registered unpackers supported extensions
-        format = _find_unpack_format(filename)
-        if format is None:
-            raise ReadError("Unknown archive format '%s'" % filename)
-
-        func = _UNPACK_FORMATS[format][1]
-        kwargs = dict(_UNPACK_FORMATS[format][2])
-        func(filename, extract_dir, **kwargs)
-
-# tokenize stuff
-
-cookie_re = re.compile("coding[:=]\s*([-\w.]+)")
-
-def _get_normal_name(orig_enc):
-    """Imitates get_normal_name in tokenizer.c."""
-    # Only care about the first 12 characters.
-    enc = orig_enc[:12].lower().replace("_", "-")
-    if enc == "utf-8" or enc.startswith("utf-8-"):
-        return "utf-8"
-    if enc in ("latin-1", "iso-8859-1", "iso-latin-1") or \
-       enc.startswith(("latin-1-", "iso-8859-1-", "iso-latin-1-")):
-        return "iso-8859-1"
-    return orig_enc
-
-def detect_encoding(readline):
-    """
-    The detect_encoding() function is used to detect the encoding that should
-    be used to decode a Python source file.  It requires one argment, readline,
-    in the same way as the tokenize() generator.
-
-    It will call readline a maximum of twice, and return the encoding used
-    (as a string) and a list of any lines (left as bytes) it has read in.
-
-    It detects the encoding from the presence of a utf-8 bom or an encoding
-    cookie as specified in pep-0263.  If both a bom and a cookie are present,
-    but disagree, a SyntaxError will be raised.  If the encoding cookie is an
-    invalid charset, raise a SyntaxError.  Note that if a utf-8 bom is found,
-    'utf-8-sig' is returned.
-
-    If no encoding is specified, then the default of 'utf-8' will be returned.
-    """
-    bom_found = False
-    encoding = None
-    default = 'utf-8'
-    def read_or_stop():
-        try:
-            return readline()
-        except StopIteration:
-            return ''
-
-    def find_cookie(line):
-        try:
-            line_string = line.decode('ascii')
-        except UnicodeDecodeError:
-            return None
-
-        matches = cookie_re.findall(line_string)
-        if not matches:
-            return None
-        encoding = _get_normal_name(matches[0])
-        try:
-            codec = codecs.lookup(encoding)
-        except LookupError:
-            # This behaviour mimics the Python interpreter
-            raise SyntaxError("unknown encoding: " + encoding)
-
-        if bom_found:
-            if codec.name != 'utf-8':
-                # This behaviour mimics the Python interpreter
-                raise SyntaxError('encoding problem: utf-8')
-            encoding += '-sig'
-        return encoding
-
-    first = read_or_stop()
-    if first.startswith(codecs.BOM_UTF8):
-        bom_found = True
-        first = first[3:]
-        default = 'utf-8-sig'
-    if not first:
-        return default, []
-
-    encoding = find_cookie(first)
-    if encoding:
-        return encoding, [first]
-
-    second = read_or_stop()
-    if not second:
-        return default, [first]
-
-    encoding = find_cookie(second)
-    if encoding:
-        return encoding, [first, second]
-
-    return default, [first, second]
-
-# functools stuff
-
-def cmp_to_key(mycmp):
-    """Convert a cmp= function into a key= function"""
-    class K(object):
-        __slots__ = ['obj']
-        def __init__(self, obj):
-            self.obj = obj
-        def __lt__(self, other):
-            return mycmp(self.obj, other.obj) < 0
-        def __gt__(self, other):
-            return mycmp(self.obj, other.obj) > 0
-        def __eq__(self, other):
-            return mycmp(self.obj, other.obj) == 0
-        def __le__(self, other):
-            return mycmp(self.obj, other.obj) <= 0
-        def __ge__(self, other):
-            return mycmp(self.obj, other.obj) >= 0
-        def __ne__(self, other):
-            return mycmp(self.obj, other.obj) != 0
-        __hash__ = None
-    return K
-
-# os stuff
-
-def fsencode(filename):
-    """
-    Encode filename to the filesystem encoding with 'surrogateescape' error
-    handler, return bytes unchanged. On Windows, use 'strict' error handler if
-    the file system encoding is 'mbcs' (which is the default encoding).
-    """
-    if isinstance(filename, str):
-        return filename
-    elif isinstance(filename, unicode):
-        return filename.encode(sys.getfilesystemencoding())
-    else:
-        raise TypeError("expect bytes or str, not %s" % type(filename).__name__)

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Refactor_support_code_fo?=
 =?utf8?q?r_compiling_xxmodule=2Ec=2E?=
Message-ID: <E1R5deJ-0007la-GP@dinsdale.python.org>

http://hg.python.org/distutils2/rev/14261f7dbdd4
changeset:   1157:14261f7dbdd4
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 02:08:16 2011 +0200
summary:
  Refactor support code for compiling xxmodule.c.

This file is needed in other tests, so it?s better to have the support
code in tests.support.  It?s also simpler to just have a skip instead of
custom print/return/test suite fiddling.  Unfortunately, the xxmodule.c
file (resurrected from the repo, and also identical to the version in
Python 2.7) cannot be compiled by Python 2.4 and 2.5 on my computer, so
the test is skipped.

The code to fix up build_ext for Unix shared builds and Windows debug
builds was also moved to support for future reuse.

Finally, I fixed code using sysconfig._CONFIG_VARS directly so that it
calls get_config_var first, so that _CONFIG_VARS is a dict instead of
None.

files:
  distutils2/tests/support.py                |   70 +-
  distutils2/tests/test_command_build_ext.py |   72 +-
  distutils2/tests/xxmodule.c                |  379 ++++++++++
  setup.cfg                                  |    3 +
  4 files changed, 458 insertions(+), 66 deletions(-)


diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -21,12 +21,12 @@
             ...  # other setup code
 
 Also provided is a DummyCommand class, useful to mock commands in the
-tests of another command that needs them, a create_distribution function
-and a skip_unless_symlink decorator.
+tests of another command that needs them, for example to fake
+compilation in build_ext (this requires that the mock build_ext command
+be injected into the distribution object's command_obj dictionary).
 
-Also provided is a DummyCommand class, useful to mock commands in the
-tests of another command that needs them, a create_distribution function
-and a skip_unless_symlink decorator.
+For tests that need to compile an extension module, use the
+copy_xxmodule_c and fixup_build_ext functions.
 
 Each class or function has a docstring to explain its purpose and usage.
 Existing tests should also be used as examples.
@@ -50,6 +50,7 @@
 
 from distutils2.dist import Distribution
 from distutils2.tests import unittest
+from distutils2._backport import sysconfig
 
 # define __all__ to make pydoc more useful
 __all__ = [
@@ -58,7 +59,7 @@
     # mocks
     'DummyCommand', 'TestDistribution',
     # misc. functions and decorators
-    'fake_dec', 'create_distribution',
+    'fake_dec', 'create_distribution', 'copy_xxmodule_c', 'fixup_build_ext',
     # imported from this module for backport purposes
     'unittest', 'requires_zlib', 'skip_unless_symlink',
 ]
@@ -298,12 +299,69 @@
     return _wrap
 
 
+def copy_xxmodule_c(directory):
+    """Helper for tests that need the xxmodule.c source file.
+
+    Example use:
+
+        def test_compile(self):
+            copy_xxmodule_c(self.tmpdir)
+            self.assertIn('xxmodule.c', os.listdir(self.tmpdir))
+
+    If the source file can be found, it will be copied to *directory*.  If not,
+    the test will be skipped.  Errors during copy are not caught.
+    """
+    filename = _get_xxmodule_path()
+    if filename is None:
+        raise unittest.SkipTest('cannot find xxmodule.c (test must run in '
+                                'the python build dir)')
+    shutil.copy(filename, directory)
+
+
+def _get_xxmodule_path():
+    path = os.path.join(os.path.dirname(__file__), 'xxmodule.c')
+    if os.path.exists(path):
+        return path
+
+
+def fixup_build_ext(cmd):
+    """Function needed to make build_ext tests pass.
+
+    When Python was built with --enable-shared on Unix, -L. is not enough to
+    find libpython<blah>.so, because regrtest runs in a tempdir, not in the
+    source directory where the .so lives.
+
+    When Python was built with in debug mode on Windows, build_ext commands
+    need their debug attribute set, and it is not done automatically for
+    some reason.
+
+    This function handles both of these things.  Example use:
+
+        cmd = build_ext(dist)
+        support.fixup_build_ext(cmd)
+        cmd.ensure_finalized()
+    """
+    if os.name == 'nt':
+        cmd.debug = sys.executable.endswith('_d.exe')
+    elif sysconfig.get_config_var('Py_ENABLE_SHARED'):
+        # To further add to the shared builds fun on Unix, we can't just add
+        # library_dirs to the Extension() instance because that doesn't get
+        # plumbed through to the final compiler command.
+        runshared = sysconfig.get_config_var('RUNSHARED')
+        if runshared is None:
+            cmd.library_dirs = ['.']
+        else:
+            name, equals, value = runshared.partition('=')
+            cmd.library_dirs = value.split(os.pathsep)
+
+
 try:
     from test.test_support import skip_unless_symlink
 except ImportError:
     skip_unless_symlink = unittest.skip(
         'requires test.test_support.skip_unless_symlink')
 
+
 requires_zlib = unittest.skipUnless(zlib, 'requires zlib')
 
 
diff --git a/distutils2/tests/test_command_build_ext.py b/distutils2/tests/test_command_build_ext.py
--- a/distutils2/tests/test_command_build_ext.py
+++ b/distutils2/tests/test_command_build_ext.py
@@ -4,26 +4,15 @@
 import shutil
 import textwrap
 from StringIO import StringIO
-from distutils2._backport import sysconfig
-from distutils2._backport.sysconfig import _CONFIG_VARS
 from distutils2.dist import Distribution
 from distutils2.errors import (UnknownFileError, CompileError,
                                PackagingPlatformError)
 from distutils2.command.build_ext import build_ext
 from distutils2.compiler.extension import Extension
-from distutils2.tests.support import assert_python_ok
+from distutils2._backport import sysconfig
 
 from distutils2.tests import support, unittest, verbose
-
-
-def _get_source_filename():
-    # use installed copy if available
-    tests_f = os.path.join(os.path.dirname(__file__), 'xxmodule.c')
-    if os.path.exists(tests_f):
-        return tests_f
-    # otherwise try using copy from build directory
-    srcdir = sysconfig.get_config_var('srcdir')
-    return os.path.join(srcdir, 'Modules', 'xxmodule.c')
+from distutils2.tests.support import assert_python_ok
 
 
 class BuildExtTestCase(support.TempdirManager,
@@ -32,9 +21,6 @@
     def setUp(self):
         super(BuildExtTestCase, self).setUp()
         self.tmp_dir = self.mkdtemp()
-        filename = _get_source_filename()
-        if os.path.exists(filename):
-            shutil.copy(filename, self.tmp_dir)
         if sys.version > "2.6":
             self.old_user_base = site.USER_BASE
             site.USER_BASE = self.mkdtemp()
@@ -45,40 +31,16 @@
 
         super(BuildExtTestCase, self).tearDown()
 
-    def _fixup_command(self, cmd):
-        # When Python was build with --enable-shared, -L. is not good enough
-        # to find the libpython<blah>.so.  This is because regrtest runs it
-        # under a tempdir, not in the top level where the .so lives.  By the
-        # time we've gotten here, Python's already been chdir'd to the
-        # tempdir.
-        #
-        # To further add to the fun, we can't just add library_dirs to the
-        # Extension() instance because that doesn't get plumbed through to the
-        # final compiler command.
-        if (sysconfig.get_config_var('Py_ENABLE_SHARED') and
-            not sys.platform.startswith('win')):
-            runshared = sysconfig.get_config_var('RUNSHARED')
-            if runshared is None:
-                cmd.library_dirs = ['.']
-            else:
-                name, equals, value = runshared.partition('=')
-                cmd.library_dirs = value.split(os.pathsep)
-
+    @unittest.skipIf(sys.version_info[:2] < (2, 6),
+                     "can't compile xxmodule successfully")
     def test_build_ext(self):
+        support.copy_xxmodule_c(self.tmp_dir)
         xx_c = os.path.join(self.tmp_dir, 'xxmodule.c')
-        if not os.path.exists(xx_c):
-            # skipping if we cannot find it
-            return
         xx_ext = Extension('xx', [xx_c])
         dist = Distribution({'name': 'xx', 'ext_modules': [xx_ext]})
         dist.package_dir = self.tmp_dir
         cmd = build_ext(dist)
-        self._fixup_command(cmd)
-
-        if os.name == "nt":
-            # On Windows, we must build a debug version iff running
-            # a debug build of Python
-            cmd.debug = sys.executable.endswith("_d.exe")
+        support.fixup_build_ext(cmd)
         cmd.build_lib = self.tmp_dir
         cmd.build_temp = self.tmp_dir
 
@@ -118,16 +80,16 @@
 
         sys.platform = 'sunos'  # fooling finalize_options
 
-        old_var = _CONFIG_VARS.get('Py_ENABLE_SHARED')
-        _CONFIG_VARS['Py_ENABLE_SHARED'] = 1
+        old_var = sysconfig.get_config_var('Py_ENABLE_SHARED')
+        sysconfig._CONFIG_VARS['Py_ENABLE_SHARED'] = 1
         try:
             cmd.ensure_finalized()
         finally:
             sys.platform = old
             if old_var is None:
-                del _CONFIG_VARS['Py_ENABLE_SHARED']
+                del sysconfig._CONFIG_VARS['Py_ENABLE_SHARED']
             else:
-                _CONFIG_VARS['Py_ENABLE_SHARED'] = old_var
+                sysconfig._CONFIG_VARS['Py_ENABLE_SHARED'] = old_var
 
         # make sure we get some library dirs under solaris
         self.assertGreater(len(cmd.library_dirs), 0)
@@ -265,13 +227,10 @@
         dist = Distribution({'name': 'xx',
                              'ext_modules': [ext]})
         cmd = build_ext(dist)
-        self._fixup_command(cmd)
+        support.fixup_build_ext(cmd)
         cmd.ensure_finalized()
         self.assertEqual(len(cmd.get_outputs()), 1)
 
-        if os.name == "nt":
-            cmd.debug = sys.executable.endswith("_d.exe")
-
         cmd.build_lib = os.path.join(self.tmp_dir, 'build')
         cmd.build_temp = os.path.join(self.tmp_dir, 'tempt')
 
@@ -454,14 +413,7 @@
 
 
 def test_suite():
-    src = _get_source_filename()
-    if not os.path.exists(src):
-        if verbose:
-            print ('test_command_build_ext: Cannot find source code (test'
-                   ' must run in python build dir)')
-        return unittest.TestSuite()
-    else:
-        return unittest.makeSuite(BuildExtTestCase)
+    return unittest.makeSuite(BuildExtTestCase)
 
 if __name__ == '__main__':
     unittest.main(defaultTest='test_suite')
diff --git a/distutils2/tests/xxmodule.c b/distutils2/tests/xxmodule.c
new file mode 100644
--- /dev/null
+++ b/distutils2/tests/xxmodule.c
@@ -0,0 +1,379 @@
+
+/* Use this file as a template to start implementing a module that
+   also declares object types. All occurrences of 'Xxo' should be changed
+   to something reasonable for your objects. After that, all other
+   occurrences of 'xx' should be changed to something reasonable for your
+   module. If your module is named foo your sourcefile should be named
+   foomodule.c.
+
+   You will probably want to delete all references to 'x_attr' and add
+   your own types of attributes instead.  Maybe you want to name your
+   local variables other than 'self'.  If your object type is needed in
+   other files, you'll have to create a file "foobarobject.h"; see
+   intobject.h for an example. */
+
+/* Xxo objects */
+
+#include "Python.h"
+
+static PyObject *ErrorObject;
+
+typedef struct {
+	PyObject_HEAD
+	PyObject	*x_attr;	/* Attributes dictionary */
+} XxoObject;
+
+static PyTypeObject Xxo_Type;
+
+#define XxoObject_Check(v)	(Py_TYPE(v) == &Xxo_Type)
+
+static XxoObject *
+newXxoObject(PyObject *arg)
+{
+	XxoObject *self;
+	self = PyObject_New(XxoObject, &Xxo_Type);
+	if (self == NULL)
+		return NULL;
+	self->x_attr = NULL;
+	return self;
+}
+
+/* Xxo methods */
+
+static void
+Xxo_dealloc(XxoObject *self)
+{
+	Py_XDECREF(self->x_attr);
+	PyObject_Del(self);
+}
+
+static PyObject *
+Xxo_demo(XxoObject *self, PyObject *args)
+{
+	if (!PyArg_ParseTuple(args, ":demo"))
+		return NULL;
+	Py_INCREF(Py_None);
+	return Py_None;
+}
+
+static PyMethodDef Xxo_methods[] = {
+	{"demo",	(PyCFunction)Xxo_demo,	METH_VARARGS,
+		PyDoc_STR("demo() -> None")},
+	{NULL,		NULL}		/* sentinel */
+};
+
+static PyObject *
+Xxo_getattr(XxoObject *self, char *name)
+{
+	if (self->x_attr != NULL) {
+		PyObject *v = PyDict_GetItemString(self->x_attr, name);
+		if (v != NULL) {
+			Py_INCREF(v);
+			return v;
+		}
+	}
+	return Py_FindMethod(Xxo_methods, (PyObject *)self, name);
+}
+
+static int
+Xxo_setattr(XxoObject *self, char *name, PyObject *v)
+{
+	if (self->x_attr == NULL) {
+		self->x_attr = PyDict_New();
+		if (self->x_attr == NULL)
+			return -1;
+	}
+	if (v == NULL) {
+		int rv = PyDict_DelItemString(self->x_attr, name);
+		if (rv < 0)
+			PyErr_SetString(PyExc_AttributeError,
+			        "delete non-existing Xxo attribute");
+		return rv;
+	}
+	else
+		return PyDict_SetItemString(self->x_attr, name, v);
+}
+
+static PyTypeObject Xxo_Type = {
+	/* The ob_type field must be initialized in the module init function
+	 * to be portable to Windows without using C++. */
+	PyVarObject_HEAD_INIT(NULL, 0)
+	"xxmodule.Xxo",		/*tp_name*/
+	sizeof(XxoObject),	/*tp_basicsize*/
+	0,			/*tp_itemsize*/
+	/* methods */
+	(destructor)Xxo_dealloc, /*tp_dealloc*/
+	0,			/*tp_print*/
+	(getattrfunc)Xxo_getattr, /*tp_getattr*/
+	(setattrfunc)Xxo_setattr, /*tp_setattr*/
+	0,			/*tp_compare*/
+	0,			/*tp_repr*/
+	0,			/*tp_as_number*/
+	0,			/*tp_as_sequence*/
+	0,			/*tp_as_mapping*/
+	0,			/*tp_hash*/
+        0,                      /*tp_call*/
+        0,                      /*tp_str*/
+        0,                      /*tp_getattro*/
+        0,                      /*tp_setattro*/
+        0,                      /*tp_as_buffer*/
+        Py_TPFLAGS_DEFAULT,     /*tp_flags*/
+        0,                      /*tp_doc*/
+        0,                      /*tp_traverse*/
+        0,                      /*tp_clear*/
+        0,                      /*tp_richcompare*/
+        0,                      /*tp_weaklistoffset*/
+        0,                      /*tp_iter*/
+        0,                      /*tp_iternext*/
+        0,                      /*tp_methods*/
+        0,                      /*tp_members*/
+        0,                      /*tp_getset*/
+        0,                      /*tp_base*/
+        0,                      /*tp_dict*/
+        0,                      /*tp_descr_get*/
+        0,                      /*tp_descr_set*/
+        0,                      /*tp_dictoffset*/
+        0,                      /*tp_init*/
+        0,                      /*tp_alloc*/
+        0,                      /*tp_new*/
+        0,                      /*tp_free*/
+        0,                      /*tp_is_gc*/
+};
+/* --------------------------------------------------------------------- */
+
+/* Function of two integers returning integer */
+
+PyDoc_STRVAR(xx_foo_doc,
+"foo(i,j)\n\
+\n\
+Return the sum of i and j.");
+
+static PyObject *
+xx_foo(PyObject *self, PyObject *args)
+{
+	long i, j;
+	long res;
+	if (!PyArg_ParseTuple(args, "ll:foo", &i, &j))
+		return NULL;
+	res = i+j; /* XXX Do something here */
+	return PyInt_FromLong(res);
+}
+
+
+/* Function of no arguments returning new Xxo object */
+
+static PyObject *
+xx_new(PyObject *self, PyObject *args)
+{
+	XxoObject *rv;
+
+	if (!PyArg_ParseTuple(args, ":new"))
+		return NULL;
+	rv = newXxoObject(args);
+	if (rv == NULL)
+		return NULL;
+	return (PyObject *)rv;
+}
+
+/* Example with subtle bug from extensions manual ("Thin Ice"). */
+
+static PyObject *
+xx_bug(PyObject *self, PyObject *args)
+{
+	PyObject *list, *item;
+
+	if (!PyArg_ParseTuple(args, "O:bug", &list))
+		return NULL;
+
+	item = PyList_GetItem(list, 0);
+	/* Py_INCREF(item); */
+	PyList_SetItem(list, 1, PyInt_FromLong(0L));
+	PyObject_Print(item, stdout, 0);
+	printf("\n");
+	/* Py_DECREF(item); */
+
+	Py_INCREF(Py_None);
+	return Py_None;
+}
+
+/* Test bad format character */
+
+static PyObject *
+xx_roj(PyObject *self, PyObject *args)
+{
+	PyObject *a;
+	long b;
+	if (!PyArg_ParseTuple(args, "O#:roj", &a, &b))
+		return NULL;
+	Py_INCREF(Py_None);
+	return Py_None;
+}
+
+
+/* ---------- */
+
+static PyTypeObject Str_Type = {
+	/* The ob_type field must be initialized in the module init function
+	 * to be portable to Windows without using C++. */
+	PyVarObject_HEAD_INIT(NULL, 0)
+	"xxmodule.Str",		/*tp_name*/
+	0,			/*tp_basicsize*/
+	0,			/*tp_itemsize*/
+	/* methods */
+	0,			/*tp_dealloc*/
+	0,			/*tp_print*/
+	0,			/*tp_getattr*/
+	0,			/*tp_setattr*/
+	0,			/*tp_compare*/
+	0,			/*tp_repr*/
+	0,			/*tp_as_number*/
+	0,			/*tp_as_sequence*/
+	0,			/*tp_as_mapping*/
+	0,			/*tp_hash*/
+	0,			/*tp_call*/
+	0,			/*tp_str*/
+	0,			/*tp_getattro*/
+	0,			/*tp_setattro*/
+	0,			/*tp_as_buffer*/
+	Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
+	0,			/*tp_doc*/
+	0,			/*tp_traverse*/
+	0,			/*tp_clear*/
+	0,			/*tp_richcompare*/
+	0,			/*tp_weaklistoffset*/
+	0,			/*tp_iter*/
+	0,			/*tp_iternext*/
+	0,			/*tp_methods*/
+	0,			/*tp_members*/
+	0,			/*tp_getset*/
+	0, /* see initxx */	/*tp_base*/
+	0,			/*tp_dict*/
+	0,			/*tp_descr_get*/
+	0,			/*tp_descr_set*/
+	0,			/*tp_dictoffset*/
+	0,			/*tp_init*/
+	0,			/*tp_alloc*/
+	0,			/*tp_new*/
+	0,			/*tp_free*/
+	0,			/*tp_is_gc*/
+};
+
+/* ---------- */
+
+static PyObject *
+null_richcompare(PyObject *self, PyObject *other, int op)
+{
+	Py_INCREF(Py_NotImplemented);
+	return Py_NotImplemented;
+}
+
+static PyTypeObject Null_Type = {
+	/* The ob_type field must be initialized in the module init function
+	 * to be portable to Windows without using C++. */
+	PyVarObject_HEAD_INIT(NULL, 0)
+	"xxmodule.Null",	/*tp_name*/
+	0,			/*tp_basicsize*/
+	0,			/*tp_itemsize*/
+	/* methods */
+	0,			/*tp_dealloc*/
+	0,			/*tp_print*/
+	0,			/*tp_getattr*/
+	0,			/*tp_setattr*/
+	0,			/*tp_compare*/
+	0,			/*tp_repr*/
+	0,			/*tp_as_number*/
+	0,			/*tp_as_sequence*/
+	0,			/*tp_as_mapping*/
+	0,			/*tp_hash*/
+	0,			/*tp_call*/
+	0,			/*tp_str*/
+	0,			/*tp_getattro*/
+	0,			/*tp_setattro*/
+	0,			/*tp_as_buffer*/
+	Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
+	0,			/*tp_doc*/
+	0,			/*tp_traverse*/
+	0,			/*tp_clear*/
+	null_richcompare,	/*tp_richcompare*/
+	0,			/*tp_weaklistoffset*/
+	0,			/*tp_iter*/
+	0,			/*tp_iternext*/
+	0,			/*tp_methods*/
+	0,			/*tp_members*/
+	0,			/*tp_getset*/
+	0, /* see initxx */	/*tp_base*/
+	0,			/*tp_dict*/
+	0,			/*tp_descr_get*/
+	0,			/*tp_descr_set*/
+	0,			/*tp_dictoffset*/
+	0,			/*tp_init*/
+	0,			/*tp_alloc*/
+	0, /* see initxx */	/*tp_new*/
+	0,			/*tp_free*/
+	0,			/*tp_is_gc*/
+};
+
+
+/* ---------- */
+
+
+/* List of functions defined in the module */
+
+static PyMethodDef xx_methods[] = {
+	{"roj",		xx_roj,		METH_VARARGS,
+		PyDoc_STR("roj(a,b) -> None")},
+	{"foo",		xx_foo,		METH_VARARGS,
+	 	xx_foo_doc},
+	{"new",		xx_new,		METH_VARARGS,
+		PyDoc_STR("new() -> new Xx object")},
+	{"bug",		xx_bug,		METH_VARARGS,
+		PyDoc_STR("bug(o) -> None")},
+	{NULL,		NULL}		/* sentinel */
+};
+
+PyDoc_STRVAR(module_doc,
+"This is a template module just for instruction.");
+
+/* Initialization function for the module (*must* be called initxx) */
+
+PyMODINIT_FUNC
+initxx(void)
+{
+	PyObject *m;
+
+	/* Due to cross platform compiler issues the slots must be filled
+	 * here. It's required for portability to Windows without requiring
+	 * C++. */
+	Null_Type.tp_base = &PyBaseObject_Type;
+	Null_Type.tp_new = PyType_GenericNew;
+	Str_Type.tp_base = &PyUnicode_Type;
+
+	/* Finalize the type object including setting type of the new type
+	 * object; doing it here is required for portability, too. */
+	if (PyType_Ready(&Xxo_Type) < 0)
+		return;
+
+	/* Create the module and add the functions */
+	m = Py_InitModule3("xx", xx_methods, module_doc);
+	if (m == NULL)
+		return;
+
+	/* Add some symbolic constants to the module */
+	if (ErrorObject == NULL) {
+		ErrorObject = PyErr_NewException("xx.error", NULL, NULL);
+		if (ErrorObject == NULL)
+			return;
+	}
+	Py_INCREF(ErrorObject);
+	PyModule_AddObject(m, "error", ErrorObject);
+
+	/* Add Str */
+	if (PyType_Ready(&Str_Type) < 0)
+		return;
+	PyModule_AddObject(m, "Str", (PyObject *)&Str_Type);
+
+	/* Add Null */
+	if (PyType_Ready(&Null_Type) < 0)
+		return;
+	PyModule_AddObject(m, "Null", (PyObject *)&Null_Type);
+}
diff --git a/setup.cfg b/setup.cfg
--- a/setup.cfg
+++ b/setup.cfg
@@ -34,10 +34,13 @@
 package_data =
     distutils2._backport = sysconfig.cfg
     distutils2.command = wininst*.exe
+    distutils2.tests = xxmodule.c
 scripts =
     pysetup
 
 # TODO build hashlib for Python < 2.4
+# TODO add all test data files
+# FIXME cfg_to_args should support comments in multi-line fields
 
 [build_ext]
 # needed so that tests work without mucking with sys.path

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Rework_test=5Fold=5Freco?=
 =?utf8?q?rd_a_bit_to_make_the_test_more_exact?=
Message-ID: <E1R5deJ-0007le-HK@dinsdale.python.org>

http://hg.python.org/distutils2/rev/944804585806
changeset:   1158:944804585806
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 02:41:00 2011 +0200
summary:
  Rework test_old_record a bit to make the test more exact
(i.e. to check the files found are what we expect)

files:
  distutils2/tests/test_command_install_dist.py |  20 +++++----
  1 files changed, 11 insertions(+), 9 deletions(-)


diff --git a/distutils2/tests/test_command_install_dist.py b/distutils2/tests/test_command_install_dist.py
--- a/distutils2/tests/test_command_install_dist.py
+++ b/distutils2/tests/test_command_install_dist.py
@@ -175,24 +175,26 @@
     def test_old_record(self):
         # test pre-PEP 376 --record option (outside dist-info dir)
         install_dir = self.mkdtemp()
-        pkgdir, dist = self.create_dist()
+        project_dir, dist = self.create_dist(scripts=['hello'])
+        os.chdir(project_dir)
+        self.write_file('hello', "print 'o hai'")
 
-        dist = Distribution()
         cmd = install_dist(dist)
         dist.command_obj['install_dist'] = cmd
         cmd.root = install_dir
-        cmd.record = os.path.join(pkgdir, 'filelist')
+        cmd.record = os.path.join(project_dir, 'filelist')
         cmd.ensure_finalized()
         cmd.run()
 
-        # let's check the record file was created with four
-        # lines, one for each .dist-info entry: METADATA,
-        # INSTALLER, REQUSTED, RECORD
         f = open(cmd.record)
-        lines = f.readlines()
-        f.close()
-        self.assertEqual(len(lines), 4)
+        try:
+            content = f.read()
+        finally:
+            f.close()
 
+        found = [os.path.basename(line) for line in content.splitlines()]
+        expected = ['hello', 'METADATA', 'INSTALLER', 'REQUESTED', 'RECORD']
+        self.assertEqual(found, expected)
 
         # XXX test that fancy_getopt is okay with options named
         # record and no-record but unrelated

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Add_a_test_for_extension?=
 =?utf8?q?_modules_in_the_old-style_record_file?=
Message-ID: <E1R5deJ-0007li-IL@dinsdale.python.org>

http://hg.python.org/distutils2/rev/07b01918bcbb
changeset:   1159:07b01918bcbb
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 02:45:27 2011 +0200
summary:
  Add a test for extension modules in the old-style record file

files:
  distutils2/tests/test_command_install_dist.py |  41 ++++++++++
  1 files changed, 41 insertions(+), 0 deletions(-)


diff --git a/distutils2/tests/test_command_install_dist.py b/distutils2/tests/test_command_install_dist.py
--- a/distutils2/tests/test_command_install_dist.py
+++ b/distutils2/tests/test_command_install_dist.py
@@ -3,7 +3,9 @@
 import os
 import sys
 
+from distutils2.command.build_ext import build_ext
 from distutils2.command.install_dist import install_dist
+from distutils2.compiler.extension import Extension
 from distutils2.dist import Distribution
 from distutils2.errors import PackagingOptionError
 
@@ -15,6 +17,12 @@
 _CONFIG_VARS = get_config_vars()
 
 
+def _make_ext_name(modname):
+    if os.name == 'nt' and sys.executable.endswith('_d.exe'):
+        modname += '_d'
+    return modname + get_config_var('SO')
+
+
 class InstallTestCase(support.TempdirManager,
                       support.LoggingCatcher,
                       unittest.TestCase):
@@ -199,6 +207,39 @@
         # XXX test that fancy_getopt is okay with options named
         # record and no-record but unrelated
 
+    @unittest.skipIf(sys.version_info[:2] < (2, 6),
+                     "can't compile xxmodule successfully")
+    def test_old_record_extensions(self):
+        # test pre-PEP 376 --record option with ext modules
+        install_dir = self.mkdtemp()
+        project_dir, dist = self.create_dist(ext_modules=[
+            Extension('xx', ['xxmodule.c'])])
+        os.chdir(project_dir)
+        support.copy_xxmodule_c(project_dir)
+
+        buildextcmd = build_ext(dist)
+        support.fixup_build_ext(buildextcmd)
+        buildextcmd.ensure_finalized()
+
+        cmd = install_dist(dist)
+        dist.command_obj['install_dist'] = cmd
+        dist.command_obj['build_ext'] = buildextcmd
+        cmd.root = install_dir
+        cmd.record = os.path.join(project_dir, 'filelist')
+        cmd.ensure_finalized()
+        cmd.run()
+
+        f =open(cmd.record)
+        try:
+            content = f.read()
+        finally:
+            f.close()
+
+        found = [os.path.basename(line) for line in content.splitlines()]
+        expected = [_make_ext_name('xx'),
+                    'METADATA', 'INSTALLER', 'REQUESTED', 'RECORD']
+        self.assertEqual(found, expected)
+
 
 def test_suite():
     return unittest.makeSuite(InstallTestCase)

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Minor_cleanup?=
Message-ID: <E1R5deJ-0007lm-JB@dinsdale.python.org>

http://hg.python.org/distutils2/rev/6de141008f4e
changeset:   1160:6de141008f4e
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 03:05:11 2011 +0200
summary:
  Minor cleanup

- Rename an attribute and create it in initialize_options instead of
  finalize_options to match the other install_* classes
- Remove unnecessary method call in tests
- Fix typo/wording in help text

files:
  distutils2/command/install_distinfo.py            |  18 +++++-----
  distutils2/tests/test_command_install_distinfo.py |   5 --
  2 files changed, 9 insertions(+), 14 deletions(-)


diff --git a/distutils2/command/install_distinfo.py b/distutils2/command/install_distinfo.py
--- a/distutils2/command/install_distinfo.py
+++ b/distutils2/command/install_distinfo.py
@@ -32,7 +32,7 @@
         ('no-record', None,
          "do not generate a RECORD file"),
         ('no-resources', None,
-         "do not generate a RESSOURCES list installed file"),
+         "do not generate a RESOURCES file"),
     ]
 
     boolean_options = ['requested', 'no-record', 'no-resources']
@@ -45,6 +45,7 @@
         self.requested = None
         self.no_record = None
         self.no_resources = None
+        self.outfiles = []
 
     def finalize_options(self):
         self.set_undefined_options('install_dist',
@@ -54,7 +55,7 @@
                                    ('install_dir', 'distinfo_dir'))
 
         if self.installer is None:
-            # FIXME distutils or distutils2?
+            # FIXME distutils or packaging or distutils2?
             # + document default in the option help text above and in install
             self.installer = 'distutils'
         if self.requested is None:
@@ -69,7 +70,6 @@
         basename = metadata.get_fullname(filesafe=True) + ".dist-info"
 
         self.distinfo_dir = os.path.join(self.distinfo_dir, basename)
-        self.outputs = []
 
     def run(self):
         # FIXME dry-run should be used at a finer level, so that people get
@@ -89,20 +89,20 @@
             metadata_path = os.path.join(self.distinfo_dir, 'METADATA')
             logger.info('creating %s', metadata_path)
             self.distribution.metadata.write(metadata_path)
-            self.outputs.append(metadata_path)
+            self.outfiles.append(metadata_path)
 
             installer_path = os.path.join(self.distinfo_dir, 'INSTALLER')
             logger.info('creating %s', installer_path)
             f = open(installer_path, 'w')
             f.write(self.installer)
             f.close()
-            self.outputs.append(installer_path)
+            self.outfiles.append(installer_path)
 
             if self.requested:
                 requested_path = os.path.join(self.distinfo_dir, 'REQUESTED')
                 logger.info('creating %s', requested_path)
                 open(requested_path, 'wb').close()
-                self.outputs.append(requested_path)
+                self.outfiles.append(requested_path)
 
 
             if not self.no_resources:
@@ -119,7 +119,7 @@
                         writer.writerow(tuple)
 
                     f.close()
-                    self.outputs.append(resources_path)
+                    self.outfiles.append(resources_path)
 
             if not self.no_record:
                 record_path = os.path.join(self.distinfo_dir, 'RECORD')
@@ -146,7 +146,7 @@
 
                 # add the RECORD file itself
                 writer.writerow((record_path, '', ''))
-                self.outputs.append(record_path)
+                self.outfiles.append(record_path)
 
     def get_outputs(self):
-        return self.outputs
+        return self.outfiles
diff --git a/distutils2/tests/test_command_install_distinfo.py b/distutils2/tests/test_command_install_distinfo.py
--- a/distutils2/tests/test_command_install_distinfo.py
+++ b/distutils2/tests/test_command_install_distinfo.py
@@ -47,7 +47,6 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.initialize_options()
         cmd.distinfo_dir = install_dir
         cmd.ensure_finalized()
         cmd.run()
@@ -83,7 +82,6 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.initialize_options()
         cmd.distinfo_dir = install_dir
         cmd.installer = 'bacon-python'
         cmd.ensure_finalized()
@@ -107,7 +105,6 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.initialize_options()
         cmd.distinfo_dir = install_dir
         cmd.requested = False
         cmd.ensure_finalized()
@@ -128,7 +125,6 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.initialize_options()
         cmd.distinfo_dir = install_dir
         cmd.no_record = True
         cmd.ensure_finalized()
@@ -166,7 +162,6 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.initialize_options()
         cmd.distinfo_dir = install_dir
         cmd.ensure_finalized()
         cmd.run()

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Add_a_simple_test_for_th?=
	=?utf8?q?e_packaging_RECORD_file=2E?=
Message-ID: <E1R5deJ-0007lq-Jk@dinsdale.python.org>

http://hg.python.org/distutils2/rev/1272d8fbe018
changeset:   1161:1272d8fbe018
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 03:10:40 2011 +0200
summary:
  Add a simple test for the packaging RECORD file.

The existing test_record is not easily extendable to add script files or
extension modules: it collects all files from fake_dists and generates a
RECORD file at runtime.  I felt more comfortable adding a new test
written from scratch, more self-contained (just one project with
well-defined files) and more stupid (the checksums and sizes are
computed once and hard-coded).

files:
  distutils2/tests/test_command_install_dist.py     |   2 +-
  distutils2/tests/test_command_install_distinfo.py |  80 +++++++++-
  2 files changed, 76 insertions(+), 6 deletions(-)


diff --git a/distutils2/tests/test_command_install_dist.py b/distutils2/tests/test_command_install_dist.py
--- a/distutils2/tests/test_command_install_dist.py
+++ b/distutils2/tests/test_command_install_dist.py
@@ -229,7 +229,7 @@
         cmd.ensure_finalized()
         cmd.run()
 
-        f =open(cmd.record)
+        f = open(cmd.record)
         try:
             content = f.read()
         finally:
diff --git a/distutils2/tests/test_command_install_distinfo.py b/distutils2/tests/test_command_install_distinfo.py
--- a/distutils2/tests/test_command_install_distinfo.py
+++ b/distutils2/tests/test_command_install_distinfo.py
@@ -2,16 +2,19 @@
 
 import os
 import csv
+import codecs
+
+from distutils2.command.install_distinfo import install_distinfo
+from distutils2.command.cmd import Command
+from distutils2.compiler.extension import Extension
+from distutils2.metadata import Metadata
+from distutils2.tests import unittest, support
+from distutils2._backport import sysconfig
 try:
     import hashlib
 except:
     from distutils2._backport import hashlib
 
-from distutils2.command.install_distinfo import install_distinfo
-from distutils2.command.cmd import Command
-from distutils2.metadata import Metadata
-from distutils2.tests import unittest, support
-
 
 class DummyInstallCmd(Command):
 
@@ -134,6 +137,73 @@
         self.checkLists(os.listdir(dist_info),
                         ['METADATA', 'REQUESTED', 'INSTALLER'])
 
+    def test_record_basic(self):
+        install_dir = self.mkdtemp()
+        modules_dest = os.path.join(install_dir, 'lib')
+        scripts_dest = os.path.join(install_dir, 'bin')
+        project_dir, dist = self.create_dist(
+            name='Spamlib', version='0.1',
+            py_modules=['spam'], scripts=['spamd'],
+            ext_modules=[Extension('_speedspam', ['_speedspam.c'])])
+
+        # using a real install_dist command is too painful, so we use a mock
+        # class that's only a holder for options to be used by install_distinfo
+        # and we create placeholder files manually instead of using build_*.
+        # the install_* commands will still be consulted by install_distinfo.
+        os.chdir(project_dir)
+        self.write_file('spam', '# Python module')
+        self.write_file('spamd', '# Python script')
+        extmod = '_speedspam' + sysconfig.get_config_var('SO')
+        self.write_file(extmod, '')
+
+        install = DummyInstallCmd(dist)
+        install.outputs = ['spam', 'spamd', extmod]
+        install.install_lib = modules_dest
+        install.install_scripts = scripts_dest
+        dist.command_obj['install_dist'] = install
+
+        cmd = install_distinfo(dist)
+        cmd.ensure_finalized()
+        dist.command_obj['install_distinfo'] = cmd
+        cmd.run()
+
+        # checksum and size are not hard-coded for METADATA as it is
+        # platform-dependent (line endings)
+        metadata = os.path.join(modules_dest, 'Spamlib-0.1.dist-info',
+                                'METADATA')
+        fp = open(metadata, 'rb')
+        try:
+            content = fp.read()
+        finally:
+            fp.close()
+
+        metadata_size = str(len(content))
+        metadata_md5 = hashlib.md5(content).hexdigest()
+
+        record = os.path.join(modules_dest, 'Spamlib-0.1.dist-info', 'RECORD')
+        fp = codecs.open(record, encoding='utf-8')
+        try:
+            content = fp.read()
+        finally:
+            fp.close()
+
+        found = []
+        for line in content.splitlines():
+            filename, checksum, size = line.split(',')
+            filename = os.path.basename(filename)
+            found.append((filename, checksum, size))
+
+        expected = [
+            ('spam', '6ab2f288ef2545868effe68757448b45', '15'),
+            ('spamd', 'd13e6156ce78919a981e424b2fdcd974', '15'),
+            (extmod, 'd41d8cd98f00b204e9800998ecf8427e', '0'),
+            ('METADATA', metadata_md5, metadata_size),
+            ('INSTALLER', '44e3fde05f3f537ed85831969acf396d', '9'),
+            ('REQUESTED', 'd41d8cd98f00b204e9800998ecf8427e', '0'),
+            ('RECORD', '', ''),
+        ]
+        self.assertEqual(found, expected)
+
     def test_record(self):
         pkg_dir, dist = self.create_dist(name='foo',
                                          version='1.0')

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Remove_--all_options_for?=
 =?utf8?q?_pysetup_actions_metadata_and_list=2E?=
Message-ID: <E1R5deJ-0007lu-Kn@dinsdale.python.org>

http://hg.python.org/distutils2/rev/534f05852780
changeset:   1162:534f05852780
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 03:18:36 2011 +0200
summary:
  Remove --all options for pysetup actions metadata and list.

When called without option (?-f field? or ?--all?), ?pysetup metadata?
didn?t do anything useful.  Now it prints out all metadata fields.
?pysetup list? without arguments already printed all installed
distributions found, so the option was superfluous.  Less typing, more
consistency, more happiness!

In addition, change ?releases? for ?projects? in the description of the
list action.  Strictly speaking, one installed distribution satisfies
the requirement for a release (i.e. version) of a project, but as
currently only one release per project can be installed at a time, the
two are somewhat equivalent, and ?project? is more understandable in
help texts (which call their argument ?dist?, by the way..)

Finally, all help texts have been moved near to the function they
document.

files:
  distutils2/run.py |  242 +++++++++++++++------------------
  1 files changed, 107 insertions(+), 135 deletions(-)


diff --git a/distutils2/run.py b/distutils2/run.py
--- a/distutils2/run.py
+++ b/distutils2/run.py
@@ -30,119 +30,6 @@
     pysetup action --help
 """
 
-create_usage = """\
-Usage: pysetup create
-   or: pysetup create --help
-
-Create a new Python project.
-"""
-
-generate_usage = """\
-Usage: pysetup generate-setup
-   or: pysetup generate-setup --help
-
-Generate a setup.py script for backward-compatibility purposes.
-"""
-
-
-graph_usage = """\
-Usage: pysetup graph dist
-   or: pysetup graph --help
-
-Print dependency graph for the distribution.
-
-positional arguments:
-   dist  installed distribution name
-"""
-
-install_usage = """\
-Usage: pysetup install [dist]
-   or: pysetup install [archive]
-   or: pysetup install [src_dir]
-   or: pysetup install --help
-
-Install a Python distribution from the indexes, source directory, or sdist.
-
-positional arguments:
-   archive  path to source distribution (zip, tar.gz)
-   dist     distribution name to install from the indexes
-   scr_dir  path to source directory
-
-"""
-
-metadata_usage = """\
-Usage: pysetup metadata [dist] [-f field ...]
-   or: pysetup metadata [dist] [--all]
-   or: pysetup metadata --help
-
-Print metadata for the distribution.
-
-positional arguments:
-   dist  installed distribution name
-
-optional arguments:
-   -f     metadata field to print
-   --all  print all metadata fields
-"""
-
-remove_usage = """\
-Usage: pysetup remove dist [-y]
-   or: pysetup remove --help
-
-Uninstall a Python distribution.
-
-positional arguments:
-   dist  installed distribution name
-
-optional arguments:
-   -y  auto confirm distribution removal
-"""
-
-run_usage = """\
-Usage: pysetup run [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
-   or: pysetup run --help
-   or: pysetup run --list-commands
-   or: pysetup run cmd --help
-"""
-
-list_usage = """\
-Usage: pysetup list dist [dist ...]
-   or: pysetup list --help
-   or: pysetup list --all
-
-Print name, version and location for the matching installed distributions.
-
-positional arguments:
-   dist  installed distribution name
-
-optional arguments:
-   --all  list all installed distributions
-"""
-
-search_usage = """\
-Usage: pysetup search [project] [--simple [url]] [--xmlrpc [url] [--fieldname value ...] --operator or|and]
-   or: pysetup search --help
-
-Search the indexes for the matching projects.
-
-positional arguments:
-    project     the project pattern to search for
-
-optional arguments:
-    --xmlrpc [url]      wether to use the xmlrpc index or not. If an url is
-                        specified, it will be used rather than the default one.
-
-    --simple [url]      wether to use the simple index or not. If an url is
-                        specified, it will be used rather than the default one.
-
-    --fieldname value   Make a search on this field. Can only be used if
-                        --xmlrpc has been selected or is the default index.
-
-    --operator or|and   Defines what is the operator to use when doing xmlrpc
-                        searchs with multiple fieldnames. Can only be used if
-                        --xmlrpc has been selected or is the default index.
-"""
-
 global_options = [
     # The fourth entry for verbose means that it can be repeated.
     ('verbose', 'v', "run verbosely (default)", True),
@@ -205,19 +92,37 @@
         return wrapper
 
 
- at action_help(create_usage)
+ at action_help("""\
+Usage: pysetup create
+   or: pysetup create --help
+
+Create a new Python project.
+""")
 def _create(distpatcher, args, **kw):
     from distutils2.create import main
     return main()
 
 
- at action_help(generate_usage)
+ at action_help("""\
+Usage: pysetup generate-setup
+   or: pysetup generate-setup --help
+
+Generate a setup.py script for backward-compatibility purposes.
+""")
 def _generate(distpatcher, args, **kw):
     generate_setup_py()
     logger.info('The setup.py was generated')
 
 
- at action_help(graph_usage)
+ at action_help("""\
+Usage: pysetup graph dist
+   or: pysetup graph --help
+
+Print dependency graph for the distribution.
+
+positional arguments:
+   dist  installed distribution name
+""")
 def _graph(dispatcher, args, **kw):
     name = args[1]
     dist = get_distribution(name, use_egg_info=True)
@@ -230,7 +135,19 @@
         print graph.repr_node(dist)
 
 
- at action_help(install_usage)
+ at action_help("""\
+Usage: pysetup install [dist]
+   or: pysetup install [archive]
+   or: pysetup install [src_dir]
+   or: pysetup install --help
+
+Install a Python distribution from the indexes, source directory, or sdist.
+
+positional arguments:
+   archive  path to source distribution (zip, tar.gz)
+   dist     distribution name to install from the indexes
+   scr_dir  path to source directory
+""")
 def _install(dispatcher, args, **kw):
     # first check if we are in a source directory
     if len(args) < 2:
@@ -250,9 +167,21 @@
         return not install(target)
 
 
- at action_help(metadata_usage)
+ at action_help("""\
+Usage: pysetup metadata [dist]
+   or: pysetup metadata [dist] [-f field ...]
+   or: pysetup metadata --help
+
+Print metadata for the distribution.
+
+positional arguments:
+   dist  installed distribution name
+
+optional arguments:
+   -f     metadata field to print; omit to get all fields
+""")
 def _metadata(dispatcher, args, **kw):
-    opts = _parse_args(args[1:], 'f:', ['all'])
+    opts = _parse_args(args[1:], 'f:', [])
     if opts['args']:
         name = opts['args'][0]
         dist = get_distribution(name, use_egg_info=True)
@@ -269,13 +198,10 @@
 
     metadata = dist.metadata
 
-    if 'all' in opts:
+    if 'f' in opts:
+        keys = (k for k in opts['f'] if k in metadata)
+    else:
         keys = metadata.keys()
-    else:
-        if 'f' in opts:
-            keys = (k for k in opts['f'] if k in metadata)
-        else:
-            keys = ()
 
     for key in keys:
         if key in metadata:
@@ -288,7 +214,18 @@
                 print '   ', value.replace('\n', '\n    ')
 
 
- at action_help(remove_usage)
+ at action_help("""\
+Usage: pysetup remove dist [-y]
+   or: pysetup remove --help
+
+Uninstall a Python distribution.
+
+positional arguments:
+   dist  installed distribution name
+
+optional arguments:
+   -y  auto confirm distribution removal
+""")
 def _remove(distpatcher, args, **kw):
     opts = _parse_args(args[1:], 'y', [])
     if 'y' in opts:
@@ -307,7 +244,12 @@
     return retcode
 
 
- at action_help(run_usage)
+ at action_help("""\
+Usage: pysetup run [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
+   or: pysetup run --help
+   or: pysetup run --list-commands
+   or: pysetup run cmd --help
+""")
 def _run(dispatcher, args, **kw):
     parser = dispatcher.parser
     args = args[1:]
@@ -347,16 +289,24 @@
     return dist
 
 
- at action_help(list_usage)
+ at action_help("""\
+Usage: pysetup list [dist ...]
+   or: pysetup list --help
+
+Print name, version and location for the matching installed distributions.
+
+positional arguments:
+   dist  installed distribution name; omit to get all distributions
+""")
 def _list(dispatcher, args, **kw):
-    opts = _parse_args(args[1:], '', ['all'])
+    opts = _parse_args(args[1:], '', [])
     dists = get_distributions(use_egg_info=True)
-    if 'all' in opts or opts['args'] == []:
+    if opts['args']:
+        results = (d for d in dists if d.name.lower() in opts['args'])
+        listall = False
+    else:
         results = dists
         listall = True
-    else:
-        results = (d for d in dists if d.name.lower() in opts['args'])
-        listall = False
 
     number = 0
     for dist in results:
@@ -373,7 +323,29 @@
         logger.info('Found %d projects installed.', number)
 
 
- at action_help(search_usage)
+ at action_help("""\
+Usage: pysetup search [project] [--simple [url]] [--xmlrpc [url] [--fieldname value ...] --operator or|and]
+   or: pysetup search --help
+
+Search the indexes for the matching projects.
+
+positional arguments:
+    project     the project pattern to search for
+
+optional arguments:
+    --xmlrpc [url]      whether to use the xmlrpc index or not. If an url is
+                        specified, it will be used rather than the default one.
+
+    --simple [url]      whether to use the simple index or not. If an url is
+                        specified, it will be used rather than the default one.
+
+    --fieldname value   Make a search on this field. Can only be used if
+                        --xmlrpc has been selected or is the default index.
+
+    --operator or|and   Defines what is the operator to use when doing xmlrpc
+                        searchs with multiple fieldnames. Can only be used if
+                        --xmlrpc has been selected or is the default index.
+""")
 def _search(dispatcher, args, **kw):
     """The search action.
 
@@ -392,7 +364,7 @@
     ('install', 'Install a project', _install),
     ('remove', 'Remove a project', _remove),
     ('search', 'Search for a project in the indexes', _search),
-    ('list', 'List installed releases', _list),
+    ('list', 'List installed projects', _list),
     ('graph', 'Display a graph', _graph),
     ('create', 'Create a project', _create),
     ('generate-setup', 'Generate a backward-comptatible setup.py', _generate),

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Cleanup=3A_move_code_out?=
	=?utf8?q?_of_a_try_block?=
Message-ID: <E1R5deJ-0007ly-Ld@dinsdale.python.org>

http://hg.python.org/distutils2/rev/1ab1204d1ad1
changeset:   1163:1ab1204d1ad1
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 03:22:30 2011 +0200
summary:
  Cleanup: move code out of a try block

files:
  distutils2/command/__init__.py |  8 ++++----
  1 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/distutils2/command/__init__.py b/distutils2/command/__init__.py
--- a/distutils2/command/__init__.py
+++ b/distutils2/command/__init__.py
@@ -51,9 +51,9 @@
     """Return the registered command"""
     try:
         cls = _COMMANDS[name]
-        if isinstance(cls, basestring):
-            cls = resolve_name(cls)
-            _COMMANDS[name] = cls
-        return cls
     except KeyError:
         raise PackagingModuleError("Invalid command %s" % name)
+    if isinstance(cls, str):
+        cls = resolve_name(cls)
+        _COMMANDS[name] = cls
+    return cls

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Remove_display_options_?=
 =?utf8?q?=28--name=2C_etc=2E=29_from_the_Distribution_class=2E?=
Message-ID: <E1R5deJ-0007m3-MW@dinsdale.python.org>

http://hg.python.org/distutils2/rev/9728bc6fd11f
changeset:   1164:9728bc6fd11f
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 03:44:02 2011 +0200
summary:
  Remove display options (--name, etc.) from the Distribution class.

These options were used to implement ?setup.py --name?,
?setup.py --version?, etc. which are now handled by the pysetup metadata
action or direct parsing of the setup.cfg file.

As a side effect, the Distribution class no longer accepts a 'url' key
in its *attrs* argument: it has to be 'home_page' or 'home-page' to be
recognized as a valid metadata field and passed down to the
dist.metadata object.

files:
  distutils2/dist.py                          |  63 +--------
  distutils2/tests/test_command_bdist_dumb.py |   2 +-
  distutils2/tests/test_command_register.py   |   2 +-
  distutils2/tests/test_command_sdist.py      |   2 +-
  distutils2/tests/test_dist.py               |  17 +-
  5 files changed, 21 insertions(+), 65 deletions(-)


diff --git a/distutils2/dist.py b/distutils2/dist.py
--- a/distutils2/dist.py
+++ b/distutils2/dist.py
@@ -42,7 +42,7 @@
 
     # 'global_options' describes the command-line options that may be
     # supplied to the setup script prior to any actual commands.
-    # Eg. "pysetup -n" or "pysetup --dry-run" both take advantage of
+    # Eg. "pysetup run -n" or "pysetup run --dry-run" both take advantage of
     # these global options.  This list should be kept to a bare minimum,
     # since every global option is also valid as a command option -- and we
     # don't want to pollute the commands with too many options that they
@@ -66,47 +66,6 @@
     display_options = [
         ('help-commands', None,
          "list all available commands"),
-        # XXX this is obsoleted by the pysetup metadata action
-        ('name', None,
-         "print package name"),
-        ('version', 'V',
-         "print package version"),
-        ('fullname', None,
-         "print <package name>-<version>"),
-        ('author', None,
-         "print the author's name"),
-        ('author-email', None,
-         "print the author's email address"),
-        ('maintainer', None,
-         "print the maintainer's name"),
-        ('maintainer-email', None,
-         "print the maintainer's email address"),
-        ('contact', None,
-         "print the maintainer's name if known, else the author's"),
-        ('contact-email', None,
-         "print the maintainer's email address if known, else the author's"),
-        ('url', None,
-         "print the URL for this package"),
-        ('license', None,
-         "print the license of the package"),
-        ('licence', None,
-         "alias for --license"),
-        ('description', None,
-         "print the package description"),
-        ('long-description', None,
-         "print the long package description"),
-        ('platforms', None,
-         "print the list of platforms"),
-        ('classifier', None,
-         "print the list of classifiers"),
-        ('keywords', None,
-         "print the list of keywords"),
-        ('provides', None,
-         "print the list of packages/modules provided"),
-        ('requires', None,
-         "print the list of packages/modules required"),
-        ('obsoletes', None,
-         "print the list of packages/modules made obsolete"),
         ('use-2to3', None,
          "use 2to3 to make source python 3.x compatible"),
         ('convert-2to3-doctests', None,
@@ -342,7 +301,6 @@
         self.commands = []
         parser = FancyGetopt(toplevel_options + self.display_options)
         parser.set_negative_aliases(self.negative_opt)
-        parser.set_aliases({'licence': 'license'})
         args = parser.getopt(args=self.script_args, object=self)
         option_order = parser.get_option_order()
 
@@ -491,7 +449,7 @@
 
         If 'global_options' is true, lists the global options:
         --dry-run, etc.  If 'display_options' is true, lists
-        the "display-only" options: --name, --version, etc.  Finally,
+        the "display-only" options: --help-commands.  Finally,
         lists per-command help for every command name or command class
         in 'commands'.
         """
@@ -528,9 +486,8 @@
 
     def handle_display_options(self, option_order):
         """If there were any non-global "display-only" options
-        (--help-commands or the metadata display options) on the command
-        line, display the requested info and return true; else return
-        false.
+        (--help-commands) on the command line, display the requested info and
+        return true; else return false.
         """
         # User just wants a list of commands -- we'll print it out and stop
         # processing now (ie. if they ran "setup --help-commands foo bar",
@@ -539,7 +496,7 @@
             self.print_commands()
             print
             print gen_usage(self.script_name)
-            return 1
+            return True
 
         # If user supplied any of the "display metadata" options, then
         # display that metadata in the order in which the user supplied the
@@ -627,14 +584,14 @@
             cmd_obj = self.command_obj[command] = cls(self)
             self.have_run[command] = 0
 
-            # Set any options that were supplied in config files
-            # or on the command line.  (NB. support for error
-            # reporting is lame here: any errors aren't reported
-            # until 'finalize_options()' is called, which means
-            # we won't report the source of the error.)
+            # Set any options that were supplied in config files or on the
+            # command line.  (XXX support for error reporting is suboptimal
+            # here: errors aren't reported until finalize_options is called,
+            # which means we won't report the source of the error.)
             options = self.command_options.get(command)
             if options:
                 self._set_command_options(cmd_obj, options)
+
         return cmd_obj
 
     def _set_command_options(self, command_obj, option_dict=None):
diff --git a/distutils2/tests/test_command_bdist_dumb.py b/distutils2/tests/test_command_bdist_dumb.py
--- a/distutils2/tests/test_command_bdist_dumb.py
+++ b/distutils2/tests/test_command_bdist_dumb.py
@@ -35,7 +35,7 @@
 
         dist = Distribution({'name': 'foo', 'version': '0.1',
                              'py_modules': ['foo'],
-                             'url': 'xxx', 'author': 'xxx',
+                             'home_page': 'xxx', 'author': 'xxx',
                              'author_email': 'xxx'})
         os.chdir(pkg_dir)
         cmd = bdist_dumb(dist)
diff --git a/distutils2/tests/test_command_register.py b/distutils2/tests/test_command_register.py
--- a/distutils2/tests/test_command_register.py
+++ b/distutils2/tests/test_command_register.py
@@ -97,7 +97,7 @@
 
     def _get_cmd(self, metadata=None):
         if metadata is None:
-            metadata = {'url': 'xxx', 'author': 'xxx',
+            metadata = {'home_page': 'xxx', 'author': 'xxx',
                         'author_email': 'xxx',
                         'name': 'xxx', 'version': 'xxx'}
         pkg_info, dist = self.create_dist(**metadata)
diff --git a/distutils2/tests/test_command_sdist.py b/distutils2/tests/test_command_sdist.py
--- a/distutils2/tests/test_command_sdist.py
+++ b/distutils2/tests/test_command_sdist.py
@@ -72,7 +72,7 @@
         """Returns a cmd"""
         if metadata is None:
             metadata = {'name': 'fake', 'version': '1.0',
-                        'url': 'xxx', 'author': 'xxx',
+                        'home_page': 'xxx', 'author': 'xxx',
                         'author_email': 'xxx'}
         dist = Distribution(metadata)
         dist.packages = ['somecode']
diff --git a/distutils2/tests/test_dist.py b/distutils2/tests/test_dist.py
--- a/distutils2/tests/test_dist.py
+++ b/distutils2/tests/test_dist.py
@@ -108,17 +108,17 @@
         Distribution(attrs={'author': 'xxx',
                             'name': 'xxx',
                             'version': '1.2',
-                            'url': 'xxxx',
+                            'home_page': 'xxxx',
                             'badoptname': 'xxx'})
         logs = self.get_logs(logging.WARNING)
-        self.assertEqual(1, len(logs))
+        self.assertEqual(len(logs), 1)
         self.assertIn('unknown argument', logs[0])
 
     def test_bad_version(self):
         Distribution(attrs={'author': 'xxx',
                             'name': 'xxx',
                             'version': 'xxx',
-                            'url': 'xxxx'})
+                            'home_page': 'xxxx'})
         logs = self.get_logs(logging.WARNING)
         self.assertEqual(1, len(logs))
         self.assertIn('not a valid version', logs[0])
@@ -126,13 +126,12 @@
     def test_empty_options(self):
         # an empty options dictionary should not stay in the
         # list of attributes
-        Distribution(attrs={'author': 'xxx',
-                            'name': 'xxx',
-                            'version': '1.2',
-                            'url': 'xxxx',
-                            'options': {}})
+        dist = Distribution(attrs={'author': 'xxx', 'name': 'xxx',
+                                   'version': '1.2', 'home_page': 'xxxx',
+                                   'options': {}})
 
         self.assertEqual([], self.get_logs(logging.WARNING))
+        self.assertNotIn('options', dir(dist))
 
     def test_non_empty_options(self):
         # TODO: how to actually use options is not documented except
@@ -145,7 +144,7 @@
         dist = Distribution(attrs={'author': 'xxx',
                                    'name': 'xxx',
                                    'version': 'xxx',
-                                   'url': 'xxxx',
+                                   'home_page': 'xxxx',
                                    'options': {'sdist': {'owner': 'root'}}})
 
         self.assertIn('owner', dist.get_option_dict('sdist'))

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Remove_obsolete_mentions?=
 =?utf8?q?_of_the_compress_program_and_=2EZ_archives=2E?=
Message-ID: <E1R5deJ-0007m6-N1@dinsdale.python.org>

http://hg.python.org/distutils2/rev/9381fa671bc2
changeset:   1165:9381fa671bc2
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 03:51:51 2011 +0200
summary:
  Remove obsolete mentions of the compress program and .Z archives.

d2 uses the shutil.make_archive function (moved from distutils),
which does not support compress.  There is no test to check that
?bdist --format whatever? works, so this slipped by.

files:
  distutils2/command/bdist.py            |   3 +-
  distutils2/command/bdist_dumb.py       |   4 +-
  distutils2/tests/test_command_bdist.py |  23 ++++++-------
  3 files changed, 13 insertions(+), 17 deletions(-)


diff --git a/distutils2/command/bdist.py b/distutils2/command/bdist.py
--- a/distutils2/command/bdist.py
+++ b/distutils2/command/bdist.py
@@ -64,13 +64,12 @@
                       'os2': 'zip'}
 
     # Establish the preferred order (for the --help-formats option).
-    format_commands = ['gztar', 'bztar', 'ztar', 'tar',
+    format_commands = ['gztar', 'bztar', 'tar',
                        'wininst', 'zip', 'msi']
 
     # And the real information.
     format_command = {'gztar': ('bdist_dumb', "gzip'ed tar file"),
                       'bztar': ('bdist_dumb', "bzip2'ed tar file"),
-                      'ztar':  ('bdist_dumb', "compressed tar file"),
                       'tar':   ('bdist_dumb', "tar file"),
                       'wininst': ('bdist_wininst',
                                   "Windows executable installer"),
diff --git a/distutils2/command/bdist_dumb.py b/distutils2/command/bdist_dumb.py
--- a/distutils2/command/bdist_dumb.py
+++ b/distutils2/command/bdist_dumb.py
@@ -5,8 +5,8 @@
 """
 
 import os
+from shutil import rmtree
 
-from shutil import rmtree
 from distutils2.util import get_platform
 from distutils2.command.cmd import Command
 from distutils2.errors import PackagingPlatformError
@@ -24,7 +24,7 @@
                      "platform name to embed in generated filenames "
                      "(default: %s)" % get_platform()),
                     ('format=', 'f',
-                     "archive format to create (tar, ztar, gztar, zip)"),
+                     "archive format to create (tar, gztar, bztar, zip)"),
                     ('keep-temp', 'k',
                      "keep the pseudo-installation tree around after " +
                      "creating the distribution archive"),
diff --git a/distutils2/tests/test_command_bdist.py b/distutils2/tests/test_command_bdist.py
--- a/distutils2/tests/test_command_bdist.py
+++ b/distutils2/tests/test_command_bdist.py
@@ -27,36 +27,33 @@
         util.get_platform = self._get_platform
 
     def test_formats(self):
-
         # let's create a command and make sure
-        # we can fix the format
-        pkg_pth, dist = self.create_dist()
+        # we can set the format
+        dist = self.create_dist()[1]
         cmd = bdist(dist)
         cmd.formats = ['msi']
         cmd.ensure_finalized()
         self.assertEqual(cmd.formats, ['msi'])
 
-        # what format bdist offers ?
-        # XXX an explicit list in bdist is
-        # not the best way to  bdist_* commands
-        # we should add a registry
-        formats = sorted(('zip', 'gztar', 'bztar', 'ztar',
-                          'tar', 'wininst', 'msi'))
+        # what formats does bdist offer?
+        # XXX hard-coded lists are not the best way to find available bdist_*
+        # commands; we should add a registry
+        formats = ['bztar', 'gztar', 'msi', 'tar', 'wininst', 'zip']
         found = sorted(cmd.format_command)
         self.assertEqual(found, formats)
 
     def test_skip_build(self):
-        pkg_pth, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = bdist(dist)
         cmd.skip_build = False
-        cmd.formats = ['ztar']
+        cmd.formats = ['zip']
         cmd.ensure_finalized()
         self.assertFalse(self._get_platform_called)
 
-        pkg_pth, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = bdist(dist)
         cmd.skip_build = True
-        cmd.formats = ['ztar']
+        cmd.formats = ['zip']
         cmd.ensure_finalized()
         self.assertTrue(self._get_platform_called)
 

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Update_list_of_trove_cla?=
	=?utf8?q?ssifiers_to_match_PyPI?=
Message-ID: <E1R5deJ-0007mE-OR@dinsdale.python.org>

http://hg.python.org/distutils2/rev/f18c5f224536
changeset:   1167:f18c5f224536
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 03:59:32 2011 +0200
summary:
  Update list of trove classifiers to match PyPI

files:
  distutils2/_trove.py |  7 +++++++
  1 files changed, 7 insertions(+), 0 deletions(-)


diff --git a/distutils2/_trove.py b/distutils2/_trove.py
--- a/distutils2/_trove.py
+++ b/distutils2/_trove.py
@@ -38,7 +38,10 @@
 'Environment :: X11 Applications :: Qt',
 'Framework :: BFG',
 'Framework :: Buildout',
+'Framework :: Buildout :: Extension',
+'Framework :: Buildout :: Recipe',
 'Framework :: Chandler',
+'Framework :: CherryPy',
 'Framework :: CubicWeb',
 'Framework :: Django',
 'Framework :: IDLE',
@@ -47,6 +50,7 @@
 'Framework :: Pylons',
 'Framework :: Setuptools Plugin',
 'Framework :: Trac',
+'Framework :: Tryton',
 'Framework :: TurboGears',
 'Framework :: TurboGears :: Applications',
 'Framework :: TurboGears :: Widgets',
@@ -69,6 +73,7 @@
 'Intended Audience :: System Administrators',
 'Intended Audience :: Telecommunications Industry',
 'License :: Aladdin Free Public License (AFPL)',
+'License :: CC0 1.0 Universal (CC0 1.0) Public Domain Dedication',
 'License :: DFSG approved',
 'License :: Eiffel Forum License (EFL)',
 'License :: Free For Educational Use',
@@ -376,6 +381,7 @@
 'Topic :: Internet :: WWW/HTTP :: Dynamic Content :: Page Counters',
 'Topic :: Internet :: WWW/HTTP :: HTTP Servers',
 'Topic :: Internet :: WWW/HTTP :: Indexing/Search',
+'Topic :: Internet :: WWW/HTTP :: Session',
 'Topic :: Internet :: WWW/HTTP :: Site Management',
 'Topic :: Internet :: WWW/HTTP :: Site Management :: Link Checking',
 'Topic :: Internet :: WWW/HTTP :: WSGI',
@@ -431,6 +437,7 @@
 'Topic :: Printing',
 'Topic :: Religion',
 'Topic :: Scientific/Engineering',
+'Topic :: Scientific/Engineering :: Artificial Life',
 'Topic :: Scientific/Engineering :: Artificial Intelligence',
 'Topic :: Scientific/Engineering :: Astronomy',
 'Topic :: Scientific/Engineering :: Atmospheric Science',

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Make_bdist=5F*_commands_?=
 =?utf8?q?respect_--skip-build_passed_to_bdist_=28=2310946=29=2E?=
Message-ID: <E1R5deJ-0007mB-Nw@dinsdale.python.org>

http://hg.python.org/distutils2/rev/95172b1c5498
changeset:   1166:95172b1c5498
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 03:58:52 2011 +0200
summary:
  Make bdist_* commands respect --skip-build passed to bdist (#10946).

There was already a test for this, but it was complicated and had a
subtle bug (custom command objects need to be put in dist.command_obj so
that other command objects may see them) that rendered it moot.

files:
  distutils2/command/bdist_dumb.py       |   5 +-
  distutils2/command/bdist_msi.py        |   6 +-
  distutils2/command/bdist_wininst.py    |   7 ++-
  distutils2/tests/test_command_bdist.py |  40 ++++---------
  4 files changed, 26 insertions(+), 32 deletions(-)


diff --git a/distutils2/command/bdist_dumb.py b/distutils2/command/bdist_dumb.py
--- a/distutils2/command/bdist_dumb.py
+++ b/distutils2/command/bdist_dumb.py
@@ -55,7 +55,7 @@
         self.format = None
         self.keep_temp = False
         self.dist_dir = None
-        self.skip_build = False
+        self.skip_build = None
         self.relative = False
         self.owner = None
         self.group = None
@@ -73,7 +73,8 @@
                     "don't know how to create dumb built distributions "
                     "on platform %s" % os.name)
 
-        self.set_undefined_options('bdist', 'dist_dir', 'plat_name')
+        self.set_undefined_options('bdist',
+                                   'dist_dir', 'plat_name', 'skip_build')
 
     def run(self):
         if not self.skip_build:
diff --git a/distutils2/command/bdist_msi.py b/distutils2/command/bdist_msi.py
--- a/distutils2/command/bdist_msi.py
+++ b/distutils2/command/bdist_msi.py
@@ -139,18 +139,22 @@
         self.no_target_optimize = False
         self.target_version = None
         self.dist_dir = None
-        self.skip_build = False
+        self.skip_build = None
         self.install_script = None
         self.pre_install_script = None
         self.versions = None
 
     def finalize_options(self):
+        self.set_undefined_options('bdist', 'skip_build')
+
         if self.bdist_dir is None:
             bdist_base = self.get_finalized_command('bdist').bdist_base
             self.bdist_dir = os.path.join(bdist_base, 'msi')
+
         short_version = get_python_version()
         if (not self.target_version) and self.distribution.has_ext_modules():
             self.target_version = short_version
+
         if self.target_version:
             self.versions = [self.target_version]
             if not self.skip_build and self.distribution.has_ext_modules()\
diff --git a/distutils2/command/bdist_wininst.py b/distutils2/command/bdist_wininst.py
--- a/distutils2/command/bdist_wininst.py
+++ b/distutils2/command/bdist_wininst.py
@@ -6,6 +6,7 @@
 import os
 
 from shutil import rmtree
+
 from distutils2.command.cmd import Command
 from distutils2.errors import PackagingOptionError, PackagingPlatformError
 from distutils2 import logger
@@ -67,13 +68,15 @@
         self.dist_dir = None
         self.bitmap = None
         self.title = None
-        self.skip_build = False
+        self.skip_build = None
         self.install_script = None
         self.pre_install_script = None
         self.user_access_control = None
 
 
     def finalize_options(self):
+        self.set_undefined_options('bdist', 'skip_build')
+
         if self.bdist_dir is None:
             if self.skip_build and self.plat_name:
                 # If build is skipped and plat_name is overridden, bdist will
@@ -83,8 +86,10 @@
                 # next the command will be initialized using that name
             bdist_base = self.get_finalized_command('bdist').bdist_base
             self.bdist_dir = os.path.join(bdist_base, 'wininst')
+
         if not self.target_version:
             self.target_version = ""
+
         if not self.skip_build and self.distribution.has_ext_modules():
             short_version = get_python_version()
             if self.target_version and self.target_version != short_version:
diff --git a/distutils2/tests/test_command_bdist.py b/distutils2/tests/test_command_bdist.py
--- a/distutils2/tests/test_command_bdist.py
+++ b/distutils2/tests/test_command_bdist.py
@@ -1,8 +1,6 @@
 """Tests for distutils.command.bdist."""
-
-from distutils2 import util
+import os
 from distutils2.command.bdist import bdist, show_formats
-
 from distutils2.tests import unittest, support, captured_stdout
 
 
@@ -10,22 +8,6 @@
                     support.LoggingCatcher,
                     unittest.TestCase):
 
-    def _mock_get_platform(self):
-        self._get_platform_called = True
-        return self._get_platform()
-
-    def setUp(self):
-        super(BuildTestCase, self).setUp()
-
-        # mock util.get_platform
-        self._get_platform_called = False
-        self._get_platform = util.get_platform
-        util.get_platform = self._mock_get_platform
-
-    def tearDown(self):
-        super(BuildTestCase, self).tearDown()
-        util.get_platform = self._get_platform
-
     def test_formats(self):
         # let's create a command and make sure
         # we can set the format
@@ -43,19 +25,21 @@
         self.assertEqual(found, formats)
 
     def test_skip_build(self):
-        dist = self.create_dist()[1]
-        cmd = bdist(dist)
-        cmd.skip_build = False
-        cmd.formats = ['zip']
-        cmd.ensure_finalized()
-        self.assertFalse(self._get_platform_called)
-
+        # bug #10946: bdist --skip-build should trickle down to subcommands
         dist = self.create_dist()[1]
         cmd = bdist(dist)
         cmd.skip_build = True
-        cmd.formats = ['zip']
         cmd.ensure_finalized()
-        self.assertTrue(self._get_platform_called)
+        dist.command_obj['bdist'] = cmd
+
+        names = ['bdist_dumb', 'bdist_wininst']
+        if os.name == 'nt':
+            names.append('bdist_msi')
+
+        for name in names:
+            subcmd = cmd.get_finalized_command(name)
+            self.assertTrue(subcmd.skip_build,
+                            '%s should take --skip-build from bdist' % name)
 
     def test_show_formats(self):
         __, stdout = captured_stdout(show_formats)

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_command_registry_to_?=
 =?utf8?q?let_Windows_machines_find_bdist=5Fmsi?=
Message-ID: <E1R5deJ-0007mI-PJ@dinsdale.python.org>

http://hg.python.org/distutils2/rev/447f2153f15f
changeset:   1168:447f2153f15f
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 04:00:39 2011 +0200
summary:
  Fix command registry to let Windows machines find bdist_msi

files:
  distutils2/command/__init__.py |  6 +++++-
  1 files changed, 5 insertions(+), 1 deletions(-)


diff --git a/distutils2/command/__init__.py b/distutils2/command/__init__.py
--- a/distutils2/command/__init__.py
+++ b/distutils2/command/__init__.py
@@ -1,5 +1,5 @@
 """Subpackage containing all standard commands."""
-
+import os
 from distutils2.errors import PackagingModuleError
 from distutils2.util import resolve_name
 
@@ -31,6 +31,10 @@
     'upload_docs': 'distutils2.command.upload_docs.upload_docs',
 }
 
+# XXX this is crappy
+if os.name == 'nt':
+    _COMMANDS['bdist_msi'] = 'distutils2.command.bdist_msi.bdist_msi'
+
 # XXX use OrderedDict to preserve the grouping (build-related, install-related,
 # distribution-related)
 STANDARD_COMMANDS = set(_COMMANDS)

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_stupid_bug_I_introdu?=
	=?utf8?q?ced?=
Message-ID: <E1R5deJ-0007mN-QF@dinsdale.python.org>

http://hg.python.org/distutils2/rev/893ba4040d5d
changeset:   1169:893ba4040d5d
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 04:08:10 2011 +0200
summary:
  Fix stupid bug I introduced

files:
  distutils2/util.py |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/distutils2/util.py b/distutils2/util.py
--- a/distutils2/util.py
+++ b/distutils2/util.py
@@ -1218,8 +1218,8 @@
 
 
 def _has_text(setup_py, installer):
-    installer_pattern = re.compile('import %s|from %s' % (
-        installer[0], installer[0]))
+    installer_pattern = re.compile('import %s|from %s' %
+                                   (installer, installer))
     setup = codecs.open(setup_py, 'r', encoding='utf-8')
     try:
         for line in setup:

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Minor_improvement_to_ext?=
 =?utf8?q?ensions_section_in_setup=2Ecfg=2E?=
Message-ID: <E1R5deJ-0007mQ-Qk@dinsdale.python.org>

http://hg.python.org/distutils2/rev/e3168d27daae
changeset:   1170:e3168d27daae
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 04:13:44 2011 +0200
summary:
  Minor improvement to extensions section in setup.cfg.

The right-hand part in [extension: foo] is now used as the name of the
extension module.  (I changed the separator from = to : and allowed
whitespace to make the sections look nicer.)

files:
  distutils2/config.py            |  11 +++++++----
  distutils2/tests/test_config.py |  17 ++++++++++++-----
  2 files changed, 19 insertions(+), 9 deletions(-)


diff --git a/distutils2/config.py b/distutils2/config.py
--- a/distutils2/config.py
+++ b/distutils2/config.py
@@ -257,13 +257,16 @@
 
         ext_modules = self.dist.ext_modules
         for section_key in content:
-            labels = section_key.split('=')
+            # no str.partition in 2.4 :(
+            labels = section_key.split(':')
             if len(labels) == 2 and labels[0] == 'extension':
-                # labels[1] not used from now but should be implemented
-                # for extension build dependency
                 values_dct = content[section_key]
+                if 'name' in values_dct:
+                    raise PackagingOptionError(
+                        'extension name should be given as [extension: name], '
+                        'not as key')
                 ext_modules.append(Extension(
-                    values_dct.pop('name'),
+                    labels[1].strip(),
                     _pop_values(values_dct, 'sources'),
                     _pop_values(values_dct, 'include_dirs'),
                     _pop_values(values_dct, 'define_macros'),
diff --git a/distutils2/tests/test_config.py b/distutils2/tests/test_config.py
--- a/distutils2/tests/test_config.py
+++ b/distutils2/tests/test_config.py
@@ -7,7 +7,7 @@
 
 from distutils2 import command
 from distutils2.dist import Distribution
-from distutils2.errors import PackagingFileError
+from distutils2.errors import PackagingFileError, PackagingOptionError
 from distutils2.compiler import new_compiler, _COMPILERS
 from distutils2.command.sdist import sdist
 
@@ -101,21 +101,20 @@
 
 # Can not be merged with SETUP_CFG else install_dist
 # command will fail when trying to compile C sources
+# TODO use a DummyCommand to mock build_ext
 EXT_SETUP_CFG = """
 [files]
 packages = one
            two
 
-[extension=speed_coconuts]
-name = one.speed_coconuts
+[extension:one.speed_coconuts]
 sources = c_src/speed_coconuts.c
 extra_link_args = "`gcc -print-file-name=libgcc.a`" -shared
 define_macros = HAVE_CAIRO HAVE_GTK2
 libraries = gecodeint gecodekernel -- sys.platform != 'win32'
     GecodeInt GecodeKernel -- sys.platform == 'win32'
 
-[extension=fast_taunt]
-name = two.fast_taunt
+[extension: two.fast_taunt]
 sources = cxx_src/utils_taunt.cxx
           cxx_src/python_module.cxx
 include_dirs = /usr/include/gecode
@@ -127,6 +126,11 @@
 
 """
 
+EXT_SETUP_CFG_BUGGY_1 = """
+[extension: realname]
+name = crash_here
+"""
+
 HOOKS_MODULE = """
 import logging
 
@@ -336,6 +340,9 @@
         self.assertEqual(ext.extra_compile_args, cargs)
         self.assertEqual(ext.language, 'cxx')
 
+        self.write_file('setup.cfg', EXT_SETUP_CFG_BUGGY_1)
+        self.assertRaises(PackagingOptionError, self.get_dist)
+
     def test_project_setup_hook_works(self):
         # Bug #11637: ensure the project directory is on sys.path to allow
         # project-specific hooks

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Minor_improvement_to_ext?=
 =?utf8?q?ensions_in_setup=2Ecfg=3A_check_parent_package?=
Message-ID: <E1R5deJ-0007mU-RK@dinsdale.python.org>

http://hg.python.org/distutils2/rev/8da735d33a06
changeset:   1171:8da735d33a06
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 04:17:33 2011 +0200
summary:
  Minor improvement to extensions in setup.cfg: check parent package

files:
  distutils2/config.py            |  17 +++++++++++-
  distutils2/tests/test_config.py |  30 +++++++++++++++++++-
  2 files changed, 44 insertions(+), 3 deletions(-)


diff --git a/distutils2/config.py b/distutils2/config.py
--- a/distutils2/config.py
+++ b/distutils2/config.py
@@ -17,6 +17,19 @@
 from distutils2.markers import interpret
 
 
+def _check_name(name, packages):
+    if '.' not in name:
+        return
+    parts = name.split('.')
+    modname = parts[-1]
+    parent = '.'.join(parts[:-1])
+    if parent not in packages:
+        # we could log a warning instead of raising, but what's the use
+        # of letting people build modules they can't import?
+        raise PackagingOptionError(
+            'parent package for extension %r not found' % name)
+
+
 def _pop_values(values_dct, key):
     """Remove values from the dictionary and convert them as a list"""
     vals_str = values_dct.pop(key, '')
@@ -265,8 +278,10 @@
                     raise PackagingOptionError(
                         'extension name should be given as [extension: name], '
                         'not as key')
+                name = labels[1].strip()
+                _check_name(name, self.dist.packages)
                 ext_modules.append(Extension(
-                    labels[1].strip(),
+                    name,
                     _pop_values(values_dct, 'sources'),
                     _pop_values(values_dct, 'include_dirs'),
                     _pop_values(values_dct, 'define_macros'),
diff --git a/distutils2/tests/test_config.py b/distutils2/tests/test_config.py
--- a/distutils2/tests/test_config.py
+++ b/distutils2/tests/test_config.py
@@ -106,6 +106,7 @@
 [files]
 packages = one
            two
+           parent.undeclared
 
 [extension:one.speed_coconuts]
 sources = c_src/speed_coconuts.c
@@ -124,6 +125,10 @@
     /DGECODE_VERSION='win32' -- sys.platform == 'win32'
 language = cxx
 
+# corner case: if the parent package of an extension is declared but
+# not its grandparent, it's legal
+[extension: parent.undeclared._speed]
+sources = parent/undeclared/_speed.c
 """
 
 EXT_SETUP_CFG_BUGGY_1 = """
@@ -131,6 +136,21 @@
 name = crash_here
 """
 
+EXT_SETUP_CFG_BUGGY_2 = """
+[files]
+packages = ham
+
+[extension: spam.eggs]
+"""
+
+EXT_SETUP_CFG_BUGGY_3 = """
+[files]
+packages = ok
+           ok.works
+
+[extension: ok.works.breaks._ext]
+"""
+
 HOOKS_MODULE = """
 import logging
 
@@ -316,7 +336,7 @@
         dist = self.get_dist()
 
         ext_modules = dict((mod.name, mod) for mod in dist.ext_modules)
-        self.assertEqual(len(ext_modules), 2)
+        self.assertEqual(len(ext_modules), 3)
         ext = ext_modules.get('one.speed_coconuts')
         self.assertEqual(ext.sources, ['c_src/speed_coconuts.c'])
         self.assertEqual(ext.define_macros, ['HAVE_CAIRO', 'HAVE_GTK2'])
@@ -343,6 +363,12 @@
         self.write_file('setup.cfg', EXT_SETUP_CFG_BUGGY_1)
         self.assertRaises(PackagingOptionError, self.get_dist)
 
+        self.write_file('setup.cfg', EXT_SETUP_CFG_BUGGY_2)
+        self.assertRaises(PackagingOptionError, self.get_dist)
+
+        self.write_file('setup.cfg', EXT_SETUP_CFG_BUGGY_3)
+        self.assertRaises(PackagingOptionError, self.get_dist)
+
     def test_project_setup_hook_works(self):
         # Bug #11637: ensure the project directory is on sys.path to allow
         # project-specific hooks
@@ -366,7 +392,7 @@
         self.write_setup({
             'setup-hooks': '\n  distutils2.tests.test_config.first_hook'
                            '\n  distutils2.tests.test_config.missing_hook'
-                           '\n  distutils2.tests.test_config.third_hook'
+                           '\n  distutils2.tests.test_config.third_hook',
         })
         self.write_file('README', 'yeah')
         dist = self.get_dist()

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Avoid_gratuitous_changes?=
	=?utf8?q?_in_method_names?=
Message-ID: <E1R5deJ-0007mY-Rz@dinsdale.python.org>

http://hg.python.org/distutils2/rev/346437f4b3c5
changeset:   1172:346437f4b3c5
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 04:19:56 2011 +0200
summary:
  Avoid gratuitous changes in method names

files:
  distutils2/tests/test_util.py |  10 +++++-----
  1 files changed, 5 insertions(+), 5 deletions(-)


diff --git a/distutils2/tests/test_util.py b/distutils2/tests/test_util.py
--- a/distutils2/tests/test_util.py
+++ b/distutils2/tests/test_util.py
@@ -842,13 +842,13 @@
     def test_setup_py_not_importing_distutils_is_not_distutils_based(self):
         self.assertFalse(is_distutils(self._random_setup_py_pkg()))
 
-    def test_setup_cfg_with_no_metadata_section_is_not_distutils2_based(self):
+    def test_setup_cfg_with_no_metadata_section_is_not_packaging_based(self):
         self.assertFalse(is_packaging(self._setup_cfg_with_no_metadata_pkg()))
 
     def test_setup_cfg_with_valid_metadata_section_is_packaging_based(self):
         self.assertTrue(is_packaging(self._valid_setup_cfg_pkg()))
 
-    def test_setup_cfg_and_invalid_setup_cfg_is_not_distutils2_based(self):
+    def test_setup_cfg_and_invalid_setup_cfg_is_not_packaging_based(self):
         self.assertFalse(is_packaging(self._invalid_setup_cfg_pkg()))
 
     def test_get_install_method_with_setuptools_pkg(self):
@@ -859,7 +859,7 @@
         path = self._distutils_pkg_info()
         self.assertEqual("distutils", get_install_method(path))
 
-    def test_get_install_method_with_distutils2_pkg(self):
+    def test_get_install_method_with_packaging_pkg(self):
         path = self._valid_setup_cfg_pkg()
         self.assertEqual("distutils2", get_install_method(path))
 
@@ -923,7 +923,7 @@
         self.write_file([directory, 'setup.py'],
                 "from distutils.core import setup")
 
-    def _write_distutils2_setup_cfg(self, directory):
+    def _write_packaging_setup_cfg(self, directory):
         self.write_file([directory, 'setup.cfg'],
                         ("[metadata]\n"
                          "name = mypackage\n"
@@ -941,7 +941,7 @@
 
     def _valid_setup_cfg_pkg(self):
         tmp = self.mkdtemp()
-        self._write_distutils2_setup_cfg(tmp)
+        self._write_packaging_setup_cfg(tmp)
         return tmp
 
     def _setuptools_egg_info_pkg(self):

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Write_a_UTF-8_empty_stri?=
 =?utf8?q?ng_to_make_haypo_happy_=3A=29_=28=239561=29?=
Message-ID: <E1R5deJ-0007mc-ST@dinsdale.python.org>

http://hg.python.org/distutils2/rev/f5a74b1f9473
changeset:   1173:f5a74b1f9473
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 04:20:56 2011 +0200
summary:
  Write a UTF-8 empty string to make haypo happy :) (#9561)

files:
  distutils2/tests/test_util.py |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/distutils2/tests/test_util.py b/distutils2/tests/test_util.py
--- a/distutils2/tests/test_util.py
+++ b/distutils2/tests/test_util.py
@@ -952,7 +952,7 @@
 
     def _distutils_pkg_info(self):
         tmp = self._distutils_setup_py_pkg()
-        self.write_file([tmp, 'PKG-INFO'], '')
+        self.write_file([tmp, 'PKG-INFO'], '', encoding='UTF-8')
         return tmp
 
     def _setup_cfg_with_no_metadata_pkg(self):
@@ -977,7 +977,7 @@
 
     def _pkg_info_with_no_distutils(self):
         tmp = self._random_setup_py_pkg()
-        self.write_file([tmp, 'PKG-INFO'], '')
+        self.write_file([tmp, 'PKG-INFO'], '', encoding='UTF-8')
         return tmp
 
     def _random_setup_py_pkg(self):

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_determination_of_Met?=
 =?utf8?q?adata_version_in_packaging_=28=238933=29=2E?=
Message-ID: <E1R5deJ-0007ml-Tt@dinsdale.python.org>

http://hg.python.org/distutils2/rev/b9ca25b3254e
changeset:   1175:b9ca25b3254e
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 04:47:20 2011 +0200
summary:
  Fix determination of Metadata version in packaging (#8933).

Original patch by Filip Gruszczy?ski.

files:
  distutils2/metadata.py            |  8 +++++---
  distutils2/tests/test_metadata.py |  4 ++--
  2 files changed, 7 insertions(+), 5 deletions(-)


diff --git a/distutils2/metadata.py b/distutils2/metadata.py
--- a/distutils2/metadata.py
+++ b/distutils2/metadata.py
@@ -62,7 +62,8 @@
                'License', 'Classifier', 'Download-URL', 'Obsoletes',
                'Provides', 'Requires')
 
-_314_MARKERS = ('Obsoletes', 'Provides', 'Requires')
+_314_MARKERS = ('Obsoletes', 'Provides', 'Requires', 'Classifier',
+                'Download-URL')
 
 _345_FIELDS = ('Metadata-Version', 'Name', 'Version', 'Platform',
                'Supported-Platform', 'Summary', 'Description',
@@ -558,6 +559,7 @@
         return data
 
     # Mapping API
+    # TODO could add iter* variants
 
     def keys(self):
         return list(_version2fieldlist(self['Metadata-Version']))
@@ -567,7 +569,7 @@
             yield key
 
     def values(self):
-        return [self[key] for key in list(self.keys())]
+        return [self[key] for key in self.keys()]
 
     def items(self):
-        return [(key, self[key]) for key in list(self.keys())]
+        return [(key, self[key]) for key in self.keys()]
diff --git a/distutils2/tests/test_metadata.py b/distutils2/tests/test_metadata.py
--- a/distutils2/tests/test_metadata.py
+++ b/distutils2/tests/test_metadata.py
@@ -267,11 +267,11 @@
         self.assertNotIn('Obsoletes', metadata)
 
         metadata['Classifier'] = ['ok']
-        self.assertEqual(metadata['Metadata-Version'], '1.2')
+        self.assertEqual(metadata['Metadata-Version'], '1.1')
 
         metadata = Metadata()
         metadata['Download-URL'] = 'ok'
-        self.assertEqual(metadata['Metadata-Version'], '1.2')
+        self.assertEqual(metadata['Metadata-Version'], '1.1')
 
         metadata = Metadata()
         metadata['Obsoletes'] = 'ok'

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Consolidate_tests_for_d2?=
	=?utf8?q?=2Emetadata=2E?=
Message-ID: <E1R5deJ-0007mh-TF@dinsdale.python.org>

http://hg.python.org/distutils2/rev/d6a26dfd2456
changeset:   1174:d6a26dfd2456
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 04:44:00 2011 +0200
summary:
  Consolidate tests for d2.metadata.

New tests were added in test_metadata and old tests inherited from
distutils were still in test_dist, so I moved them into test_metadata
(except for one which was more at home in test_run) and merged
duplicates.

I also added some skips to lure contributors <wink>, optimized the
Metadata.update method a trifle, and added notes about a number of
issues.

A note: The tests in test_dist used to dump the Metadata objects to a
file in the METADATA format and look for strings in its contents; I
updated them to use the mapping API of Metadata instead.  For some
fields with special writing rules, I have added tests to ensure my
conversion did not lose anything.

files:
  distutils2/metadata.py            |   16 +-
  distutils2/tests/test_dist.py     |  246 +---------
  distutils2/tests/test_metadata.py |  415 ++++++++++++-----
  distutils2/tests/test_run.py      |   19 +-
  4 files changed, 350 insertions(+), 346 deletions(-)


diff --git a/distutils2/metadata.py b/distutils2/metadata.py
--- a/distutils2/metadata.py
+++ b/distutils2/metadata.py
@@ -371,11 +371,20 @@
         Keys that don't match a metadata field or that have an empty value are
         dropped.
         """
+        # XXX the code should just use self.set, which does tbe same checks and
+        # conversions already, but that would break packaging.pypi: it uses the
+        # update method, which does not call _set_best_version (which set
+        # does), and thus allows having a Metadata object (as long as you don't
+        # modify or write it) with extra fields from PyPI that are not fields
+        # defined in Metadata PEPs.  to solve it, the best_version system
+        # should be reworked so that it's called only for writing, or in a new
+        # strict mode, or with a new, more lax Metadata subclass in p7g.pypi
         def _set(key, value):
             if key in _ATTR2FIELD and value:
                 self.set(self._convert_name(key), value)
 
-        if other is None:
+        if not other:
+            # other is None or empty container
             pass
         elif hasattr(other, 'keys'):
             for k in other.keys():
@@ -385,7 +394,8 @@
                 _set(k, v)
 
         if kwargs:
-            self.update(kwargs)
+            for k, v in kwargs.items():
+                _set(k, v)
 
     def set(self, name, value):
         """Control then set a metadata field."""
@@ -550,7 +560,7 @@
     # Mapping API
 
     def keys(self):
-        return _version2fieldlist(self['Metadata-Version'])
+        return list(_version2fieldlist(self['Metadata-Version']))
 
     def __iter__(self):
         for key in self.keys():
diff --git a/distutils2/tests/test_dist.py b/distutils2/tests/test_dist.py
--- a/distutils2/tests/test_dist.py
+++ b/distutils2/tests/test_dist.py
@@ -4,14 +4,12 @@
 import codecs
 import logging
 import textwrap
-from distutils2._backport import sysconfig
+
 import distutils2.dist
-from StringIO import StringIO
 
 from distutils2.dist import Distribution
 from distutils2.command import set_command
 from distutils2.command.cmd import Command
-from distutils2.metadata import Metadata
 from distutils2.errors import PackagingModuleError, PackagingOptionError
 from distutils2.tests import captured_stdout
 from distutils2.tests import support, unittest
@@ -72,37 +70,7 @@
             __, stdout = captured_stdout(create_distribution, files)
             self.assertEqual(stdout, '')
         finally:
-            distutils2.dist.DEBUG = False
-
-    def test_write_pkg_file(self):
-        # Check Metadata handling of Unicode fields
-        tmp_dir = self.mkdtemp()
-        my_file = os.path.join(tmp_dir, 'f')
-        cls = Distribution
-
-        dist = cls(attrs={'author': u'Mister Caf\xe9',
-                          'name': 'my.package',
-                          'maintainer': u'Caf\xe9 Junior',
-                          'summary': u'Caf\xe9 torr\xe9fi\xe9',
-                          'description': u'H\xe9h\xe9h\xe9'})
-
-        # let's make sure the file can be written
-        # with Unicode fields. they are encoded with
-        # PKG_INFO_ENCODING
-        fp = codecs.open(my_file, 'w', encoding='utf-8')
-        dist.metadata.write_file(fp)
-        fp.close()
-
-        # regular ascii is of course always usable
-        dist = cls(attrs={'author': 'Mister Cafe',
-                          'name': 'my.package',
-                          'maintainer': 'Cafe Junior',
-                          'summary': 'Cafe torrefie',
-                          'description': 'Hehehe'})
-
-        fp = open(my_file, 'w')
-        dist.metadata.write_file(fp)
-        fp.close()
+            packaging.dist.DEBUG = False
 
     def test_bad_attr(self):
         Distribution(attrs={'author': 'xxx',
@@ -114,15 +82,6 @@
         self.assertEqual(len(logs), 1)
         self.assertIn('unknown argument', logs[0])
 
-    def test_bad_version(self):
-        Distribution(attrs={'author': 'xxx',
-                            'name': 'xxx',
-                            'version': 'xxx',
-                            'home_page': 'xxxx'})
-        logs = self.get_logs(logging.WARNING)
-        self.assertEqual(1, len(logs))
-        self.assertIn('not a valid version', logs[0])
-
     def test_empty_options(self):
         # an empty options dictionary should not stay in the
         # list of attributes
@@ -160,6 +119,27 @@
         self.assertEqual(dist.metadata['platform'], ['one', 'two'])
         self.assertEqual(dist.metadata['keywords'], ['one', 'two'])
 
+    def test_custom_pydistutils(self):
+        # Bug #2166:  make sure pydistutils.cfg is found
+        if os.name == 'posix':
+            user_filename = ".pydistutils.cfg"
+        else:
+            user_filename = "pydistutils.cfg"
+
+        temp_dir = self.mkdtemp()
+        user_filename = os.path.join(temp_dir, user_filename)
+        f = open(user_filename, 'w')
+        try:
+            f.write('.')
+        finally:
+            f.close()
+
+        dist = Distribution()
+
+        os.environ['HOME'] = temp_dir
+        files = dist.find_config_files()
+        self.assertIn(user_filename, files)
+
     def test_find_config_files_disable(self):
         # Bug #1180: Allow users to disable their own config file.
         temp_home = self.mkdtemp()
@@ -281,186 +261,8 @@
         self.assertRaises(PackagingOptionError, d.run_command, 'test_dist')
 
 
-class MetadataTestCase(support.TempdirManager,
-                       support.LoggingCatcher,
-                       support.EnvironRestorer,
-                       unittest.TestCase):
-
-    restore_environ = ['HOME']
-
-    def setUp(self):
-        super(MetadataTestCase, self).setUp()
-        self.argv = sys.argv, sys.argv[:]
-
-    def tearDown(self):
-        sys.argv = self.argv[0]
-        sys.argv[:] = self.argv[1]
-        super(MetadataTestCase, self).tearDown()
-
-    def test_simple_metadata(self):
-        attrs = {"name": "package",
-                 "version": "1.0"}
-        dist = Distribution(attrs)
-        meta = self.format_metadata(dist)
-        self.assertIn("Metadata-Version: 1.0", meta)
-        self.assertNotIn("provides:", meta.lower())
-        self.assertNotIn("requires:", meta.lower())
-        self.assertNotIn("obsoletes:", meta.lower())
-
-    def test_provides_dist(self):
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "provides_dist": ["package", "package.sub"]}
-        dist = Distribution(attrs)
-        self.assertEqual(dist.metadata['Provides-Dist'],
-                         ["package", "package.sub"])
-        meta = self.format_metadata(dist)
-        self.assertIn("Metadata-Version: 1.2", meta)
-        self.assertNotIn("requires:", meta.lower())
-        self.assertNotIn("obsoletes:", meta.lower())
-
-    def _test_provides_illegal(self):
-        # XXX to do: check the versions
-        self.assertRaises(ValueError, Distribution,
-                          {"name": "package",
-                           "version": "1.0",
-                           "provides_dist": ["my.pkg (splat)"]})
-
-    def test_requires_dist(self):
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "requires_dist": ["other", "another (==1.0)"]}
-        dist = Distribution(attrs)
-        self.assertEqual(dist.metadata['Requires-Dist'],
-                         ["other", "another (==1.0)"])
-        meta = self.format_metadata(dist)
-        self.assertIn("Metadata-Version: 1.2", meta)
-        self.assertNotIn("provides:", meta.lower())
-        self.assertIn("Requires-Dist: other", meta)
-        self.assertIn("Requires-Dist: another (==1.0)", meta)
-        self.assertNotIn("obsoletes:", meta.lower())
-
-    def _test_requires_illegal(self):
-        # XXX
-        self.assertRaises(ValueError, Distribution,
-                          {"name": "package",
-                           "version": "1.0",
-                           "requires": ["my.pkg (splat)"]})
-
-    def test_obsoletes_dist(self):
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "obsoletes_dist": ["other", "another (<1.0)"]}
-        dist = Distribution(attrs)
-        self.assertEqual(dist.metadata['Obsoletes-Dist'],
-                         ["other", "another (<1.0)"])
-        meta = self.format_metadata(dist)
-        self.assertIn("Metadata-Version: 1.2", meta)
-        self.assertNotIn("provides:", meta.lower())
-        self.assertNotIn("requires:", meta.lower())
-        self.assertIn("Obsoletes-Dist: other", meta)
-        self.assertIn("Obsoletes-Dist: another (<1.0)", meta)
-
-    def _test_obsoletes_illegal(self):
-        # XXX
-        self.assertRaises(ValueError, Distribution,
-                          {"name": "package",
-                           "version": "1.0",
-                           "obsoletes": ["my.pkg (splat)"]})
-
-    def format_metadata(self, dist):
-        sio = StringIO()
-        dist.metadata.write_file(sio)
-        return sio.getvalue()
-
-    def test_custom_pydistutils(self):
-        # fixes #2166
-        # make sure pydistutils.cfg is found
-        if os.name == 'posix':
-            user_filename = ".pydistutils.cfg"
-        else:
-            user_filename = "pydistutils.cfg"
-
-        temp_dir = self.mkdtemp()
-        user_filename = os.path.join(temp_dir, user_filename)
-        f = open(user_filename, 'w')
-        f.write('.')
-        f.close()
-
-        dist = Distribution()
-
-        # linux-style
-        if sys.platform in ('linux', 'darwin'):
-            os.environ['HOME'] = temp_dir
-            files = dist.find_config_files()
-            self.assertIn(user_filename, files)
-
-        # win32-style
-        if sys.platform == 'win32':
-            # home drive should be found
-            os.environ['HOME'] = temp_dir
-            files = dist.find_config_files()
-            self.assertIn(user_filename, files)
-
-    def test_show_help(self):
-        # smoke test, just makes sure some help is displayed
-        dist = Distribution()
-        sys.argv = []
-        dist.help = True
-        dist.script_name = os.path.join(sysconfig.get_path('scripts'),
-                                        'pysetup')
-        __, stdout = captured_stdout(dist.parse_command_line)
-        output = [line for line in stdout.split('\n')
-                  if line.strip() != '']
-        self.assertGreater(len(output), 0)
-
-    def test_description(self):
-        desc = textwrap.dedent("""\
-        example::
-              We start here
-            and continue here
-          and end here.""")
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "description": desc}
-
-        dist = distutils2.dist.Distribution(attrs)
-        meta = self.format_metadata(dist)
-        meta = meta.replace('\n' + 7 * ' ' + '|', '\n')
-        self.assertIn(desc, meta)
-
-    def test_read_metadata(self):
-        attrs = {"name": "package",
-                 "version": "1.0",
-                 "description": "desc",
-                 "summary": "xxx",
-                 "download_url": "http://example.com",
-                 "keywords": ['one', 'two'],
-                 "requires_dist": ['foo']}
-
-        dist = Distribution(attrs)
-        PKG_INFO = StringIO()
-        dist.metadata.write_file(PKG_INFO)
-        PKG_INFO.seek(0)
-
-        metadata = Metadata()
-        metadata.read_file(PKG_INFO)
-
-        self.assertEqual(metadata['name'], "package")
-        self.assertEqual(metadata['version'], "1.0")
-        self.assertEqual(metadata['summary'], "xxx")
-        self.assertEqual(metadata['download_url'], 'http://example.com')
-        self.assertEqual(metadata['keywords'], ['one', 'two'])
-        self.assertEqual(metadata['platform'], [])
-        self.assertEqual(metadata['obsoletes'], [])
-        self.assertEqual(metadata['requires-dist'], ['foo'])
-
-
 def test_suite():
-    suite = unittest.TestSuite()
-    suite.addTest(unittest.makeSuite(DistributionTestCase))
-    suite.addTest(unittest.makeSuite(MetadataTestCase))
-    return suite
+    return unittest.makeSuite(DistributionTestCase)
 
 if __name__ == "__main__":
     unittest.main(defaultTest="test_suite")
diff --git a/distutils2/tests/test_metadata.py b/distutils2/tests/test_metadata.py
--- a/distutils2/tests/test_metadata.py
+++ b/distutils2/tests/test_metadata.py
@@ -1,21 +1,40 @@
+# encoding: utf-8
 """Tests for distutils2.metadata."""
 import os
 import sys
 import codecs
 import logging
+from textwrap import dedent
 from StringIO import StringIO
 
 from distutils2.errors import (MetadataConflictError, MetadataMissingError,
-                              MetadataUnrecognizedVersionError)
+                               MetadataUnrecognizedVersionError)
 from distutils2.metadata import Metadata, PKG_INFO_PREFERRED_VERSION
 
 from distutils2.tests import unittest
-from distutils2.tests.support import LoggingCatcher
+from distutils2.tests.support import (LoggingCatcher, TempdirManager,
+                                      EnvironRestorer)
 
 
 class MetadataTestCase(LoggingCatcher,
+                       TempdirManager,
+                       EnvironRestorer,
                        unittest.TestCase):
 
+    maxDiff = None
+    restore_environ = ['HOME']
+
+    def setUp(self):
+        super(MetadataTestCase, self).setUp()
+        self.argv = sys.argv, sys.argv[:]
+
+    def tearDown(self):
+        sys.argv = self.argv[0]
+        sys.argv[:] = self.argv[1]
+        super(MetadataTestCase, self).tearDown()
+
+    ####  Test various methods of the Metadata class
+
     def test_instantiation(self):
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
         f = codecs.open(PKG_INFO, 'r', encoding='utf-8')
@@ -48,17 +67,6 @@
         self.assertRaises(TypeError, Metadata,
                           PKG_INFO, mapping=m, fileobj=fp)
 
-    def test_metadata_read_write(self):
-        PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
-        metadata = Metadata(PKG_INFO)
-        out = StringIO()
-        metadata.write_file(out)
-        out.seek(0)
-        res = Metadata()
-        res.read_file(out)
-        for k in metadata:
-            self.assertEqual(metadata[k], res[k])
-
     def test_metadata_markers(self):
         # see if we can be platform-aware
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
@@ -82,32 +90,6 @@
         metadata.read_file(StringIO(content))
         self.assertEqual(metadata['Requires-Dist'], ['foo'])
 
-    def test_description(self):
-        PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
-        f = codecs.open(PKG_INFO, 'r', encoding='utf-8')
-        try:
-            content = f.read() % sys.platform
-        finally:
-            f.close()
-        metadata = Metadata()
-        metadata.read_file(StringIO(content))
-
-        # see if we can read the description now
-        DESC = os.path.join(os.path.dirname(__file__), 'LONG_DESC.txt')
-        f = open(DESC)
-        try:
-            wanted = f.read()
-        finally:
-            f.close()
-        self.assertEqual(wanted, metadata['Description'])
-
-        # save the file somewhere and make sure we can read it back
-        out = StringIO()
-        metadata.write_file(out)
-        out.seek(0)
-        metadata.read_file(out)
-        self.assertEqual(wanted, metadata['Description'])
-
     def test_mapping_api(self):
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
         f = codecs.open(PKG_INFO, 'r', encoding='utf-8')
@@ -125,80 +107,90 @@
         metadata.update([('version', '0.7')])
         self.assertEqual(metadata['Version'], '0.7')
 
-        self.assertEqual(list(metadata), list(metadata.keys()))
+        # make sure update method checks values like the set method does
+        metadata.update({'version': '1--2'})
+        self.assertEqual(len(self.get_logs()), 1)
 
-    def test_versions(self):
-        metadata = Metadata()
-        metadata['Obsoletes'] = 'ok'
-        self.assertEqual(metadata['Metadata-Version'], '1.1')
+        self.assertEqual(list(metadata), metadata.keys())
 
-        del metadata['Obsoletes']
-        metadata['Obsoletes-Dist'] = 'ok'
-        self.assertEqual(metadata['Metadata-Version'], '1.2')
+    def test_read_metadata(self):
+        fields = {'name': 'project',
+                  'version': '1.0',
+                  'description': 'desc',
+                  'summary': 'xxx',
+                  'download_url': 'http://example.com',
+                  'keywords': ['one', 'two'],
+                  'requires_dist': ['foo']}
 
-        self.assertRaises(MetadataConflictError, metadata.set,
-                          'Obsoletes', 'ok')
+        metadata = Metadata(mapping=fields)
+        PKG_INFO = StringIO()
+        metadata.write_file(PKG_INFO)
+        PKG_INFO.seek(0)
 
-        del metadata['Obsoletes']
-        del metadata['Obsoletes-Dist']
-        metadata['Version'] = '1'
-        self.assertEqual(metadata['Metadata-Version'], '1.0')
+        metadata = Metadata(fileobj=PKG_INFO)
 
-        PKG_INFO = os.path.join(os.path.dirname(__file__),
-                                'SETUPTOOLS-PKG-INFO')
-        f = codecs.open(PKG_INFO, 'r', encoding='utf-8')
+        self.assertEqual(metadata['name'], 'project')
+        self.assertEqual(metadata['version'], '1.0')
+        self.assertEqual(metadata['summary'], 'xxx')
+        self.assertEqual(metadata['download_url'], 'http://example.com')
+        self.assertEqual(metadata['keywords'], ['one', 'two'])
+        self.assertEqual(metadata['platform'], [])
+        self.assertEqual(metadata['obsoletes'], [])
+        self.assertEqual(metadata['requires-dist'], ['foo'])
+
+    def test_write_metadata(self):
+        # check support of non-ASCII values
+        tmp_dir = self.mkdtemp()
+        my_file = os.path.join(tmp_dir, 'f')
+
+        metadata = Metadata(mapping={'author': u'Mister Caf?',
+                                     'name': u'my.project',
+                                     'author': u'Caf? Junior',
+                                     'summary': u'Caf? torr?fi?',
+                                     'description': u'H?h?h?',
+                                     'keywords': [u'caf?', u'coffee']})
+        metadata.write(my_file)
+
+        # the file should use UTF-8
+        metadata2 = Metadata()
+        fp = codecs.open(my_file, encoding='utf-8')
         try:
-            content = f.read()
+            metadata2.read_file(fp)
         finally:
-            f.close()
-        metadata.read_file(StringIO(content))
-        self.assertEqual(metadata['Metadata-Version'], '1.0')
+            fp.close()
 
-        PKG_INFO = os.path.join(os.path.dirname(__file__),
-                                'SETUPTOOLS-PKG-INFO2')
-        f = codecs.open(PKG_INFO, 'r', encoding='utf-8')
+        # XXX when keywords are not defined, metadata will have
+        # 'Keywords': [] but metadata2 will have 'Keywords': ['']
+        # because of a value.split(',') in Metadata.get
+        self.assertEqual(metadata.items(), metadata2.items())
+
+        # ASCII also works, it's a subset of UTF-8
+        metadata = Metadata(mapping={'author': u'Mister Cafe',
+                                     'name': u'my.project',
+                                     'author': u'Cafe Junior',
+                                     'summary': u'Cafe torrefie',
+                                     'description': u'Hehehe'})
+        metadata.write(my_file)
+
+        metadata2 = Metadata()
+        fp = codecs.open(my_file, encoding='utf-8')
         try:
-            content = f.read()
+            metadata2.read_file(fp)
         finally:
-            f.close()
+            fp.close()
 
-        metadata.read_file(StringIO(content))
-        self.assertEqual(metadata['Metadata-Version'], '1.1')
+    def test_metadata_read_write(self):
+        PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
+        metadata = Metadata(PKG_INFO)
+        out = StringIO()
+        metadata.write_file(out)
 
-        # Update the _fields dict directly to prevent 'Metadata-Version'
-        # from being updated by the _set_best_version() method.
-        metadata._fields['Metadata-Version'] = '1.618'
-        self.assertRaises(MetadataUnrecognizedVersionError, metadata.keys)
+        out.seek(0)
+        res = Metadata()
+        res.read_file(out)
+        self.assertEqual(metadata.values(), res.values())
 
-    def test_warnings(self):
-        metadata = Metadata()
-
-        # these should raise a warning
-        values = (('Requires-Dist', 'Funky (Groovie)'),
-                  ('Requires-Python', '1-4'))
-
-        for name, value in values:
-            metadata.set(name, value)
-
-        # we should have a certain amount of warnings
-        self.assertEqual(len(self.get_logs()), 2)
-
-    def test_multiple_predicates(self):
-        metadata = Metadata()
-
-        # see for "3" instead of "3.0"  ???
-        # its seems like the MINOR VERSION can be omitted
-        metadata['Requires-Python'] = '>=2.6, <3.0'
-        metadata['Requires-Dist'] = ['Foo (>=2.6, <3.0)']
-
-        self.assertEqual([], self.get_logs(logging.WARNING))
-
-    def test_project_url(self):
-        metadata = Metadata()
-        metadata['Project-URL'] = [('one', 'http://ok')]
-        self.assertEqual(metadata['Project-URL'],
-                          [('one', 'http://ok')])
-        self.assertEqual(metadata['Metadata-Version'], '1.2')
+    ####  Test checks
 
     def test_check_version(self):
         metadata = Metadata()
@@ -261,38 +253,221 @@
         metadata['Requires-dist'] = ['Foo (a)']
         metadata['Obsoletes-dist'] = ['Foo (a)']
         metadata['Provides-dist'] = ['Foo (a)']
-        if metadata.docutils_support:
-            missing, warnings = metadata.check()
-            self.assertEqual(len(warnings), 4)
-            metadata.docutils_support = False
         missing, warnings = metadata.check()
         self.assertEqual(len(warnings), 4)
 
-    def test_best_choice(self):
-        metadata = Metadata()
-        metadata['Version'] = '1.0'
+    ####  Test fields and metadata versions
+
+    def test_metadata_versions(self):
+        metadata = Metadata(mapping={'name': 'project', 'version': '1.0'})
         self.assertEqual(metadata['Metadata-Version'],
                          PKG_INFO_PREFERRED_VERSION)
+        self.assertNotIn('Provides', metadata)
+        self.assertNotIn('Requires', metadata)
+        self.assertNotIn('Obsoletes', metadata)
+
         metadata['Classifier'] = ['ok']
         self.assertEqual(metadata['Metadata-Version'], '1.2')
 
-    def test_project_urls(self):
-        # project-url is a bit specific, make sure we write it
-        # properly in PKG-INFO
         metadata = Metadata()
-        metadata['Version'] = '1.0'
-        metadata['Project-Url'] = [('one', 'http://ok')]
+        metadata['Download-URL'] = 'ok'
+        self.assertEqual(metadata['Metadata-Version'], '1.2')
+
+        metadata = Metadata()
+        metadata['Obsoletes'] = 'ok'
+        self.assertEqual(metadata['Metadata-Version'], '1.1')
+
+        del metadata['Obsoletes']
+        metadata['Obsoletes-Dist'] = 'ok'
+        self.assertEqual(metadata['Metadata-Version'], '1.2')
+
+        self.assertRaises(MetadataConflictError, metadata.set,
+                          'Obsoletes', 'ok')
+
+        del metadata['Obsoletes']
+        del metadata['Obsoletes-Dist']
+        metadata['Version'] = '1'
+        self.assertEqual(metadata['Metadata-Version'], '1.0')
+
+        # make sure the _best_version function works okay with
+        # non-conflicting fields from 1.1 and 1.2 (i.e. we want only the
+        # requires/requires-dist and co. pairs to cause a conflict, not all
+        # fields in _314_MARKERS)
+        metadata = Metadata()
+        metadata['Requires-Python'] = '3'
+        metadata['Classifier'] = ['Programming language :: Python :: 3']
+        self.assertEqual(metadata['Metadata-Version'], '1.2')
+
+        PKG_INFO = os.path.join(os.path.dirname(__file__),
+                                'SETUPTOOLS-PKG-INFO')
+        metadata = Metadata(PKG_INFO)
+        self.assertEqual(metadata['Metadata-Version'], '1.0')
+
+        PKG_INFO = os.path.join(os.path.dirname(__file__),
+                                'SETUPTOOLS-PKG-INFO2')
+        metadata = Metadata(PKG_INFO)
+        self.assertEqual(metadata['Metadata-Version'], '1.1')
+
+        # Update the _fields dict directly to prevent 'Metadata-Version'
+        # from being updated by the _set_best_version() method.
+        metadata._fields['Metadata-Version'] = '1.618'
+        self.assertRaises(MetadataUnrecognizedVersionError, metadata.keys)
+
+    def test_version(self):
+        Metadata(mapping={'author': 'xxx',
+                          'name': 'xxx',
+                          'version': 'xxx',
+                          'home_page': 'xxxx'})
+        logs = self.get_logs(logging.WARNING)
+        self.assertEqual(1, len(logs))
+        self.assertIn('not a valid version', logs[0])
+
+    def test_description(self):
+        PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
+        f = codecs.open(PKG_INFO, 'r', encoding='utf-8')
+        try:
+            content = f.read() % sys.platform
+        finally:
+            f.close()
+        metadata = Metadata()
+        metadata.read_file(StringIO(content))
+
+        # see if we can read the description now
+        DESC = os.path.join(os.path.dirname(__file__), 'LONG_DESC.txt')
+        f = open(DESC)
+        try:
+            wanted = f.read()
+        finally:
+            f.close()
+        self.assertEqual(wanted, metadata['Description'])
+
+        # save the file somewhere and make sure we can read it back
+        out = StringIO()
+        metadata.write_file(out)
+        out.seek(0)
+
+        out.seek(0)
+        metadata = Metadata()
+        metadata.read_file(out)
+        self.assertEqual(wanted, metadata['Description'])
+
+    def test_description_folding(self):
+        # make sure the indentation is preserved
+        out = StringIO()
+        desc = dedent("""\
+        example::
+              We start here
+            and continue here
+          and end here.
+        """)
+
+        metadata = Metadata()
+        metadata['description'] = desc
+        metadata.write_file(out)
+
+        folded_desc = desc.replace('\n', '\n' + (7 * ' ') + '|')
+        self.assertIn(folded_desc, out.getvalue())
+
+    def test_project_url(self):
+        metadata = Metadata()
+        metadata['Project-URL'] = [('one', 'http://ok')]
+        self.assertEqual(metadata['Project-URL'], [('one', 'http://ok')])
+        self.assertEqual(metadata['Metadata-Version'], '1.2')
+
+        # make sure this particular field is handled properly when written
+        fp = StringIO()
+        metadata.write_file(fp)
+        self.assertIn('Project-URL: one,http://ok', fp.getvalue().split('\n'))
+
+        fp.seek(0)
+        metadata = Metadata()
+        metadata.read_file(fp)
         self.assertEqual(metadata['Project-Url'], [('one', 'http://ok')])
-        file_ = StringIO()
-        metadata.write_file(file_)
-        file_.seek(0)
-        res = file_.read().split('\n')
-        self.assertIn('Project-URL: one,http://ok', res)
 
-        file_.seek(0)
+    # TODO copy tests for v1.1 requires, obsoletes and provides from distutils
+    # (they're useless but we support them so we should test them anyway)
+
+    def test_provides_dist(self):
+        fields = {'name': 'project',
+                  'version': '1.0',
+                  'provides_dist': ['project', 'my.project']}
+        metadata = Metadata(mapping=fields)
+        self.assertEqual(metadata['Provides-Dist'],
+                         ['project', 'my.project'])
+        self.assertEqual(metadata['Metadata-Version'], '1.2', metadata)
+        self.assertNotIn('Requires', metadata)
+        self.assertNotIn('Obsoletes', metadata)
+
+    @unittest.skip('needs to be implemented')
+    def test_provides_illegal(self):
+        # TODO check the versions (like distutils does for old provides field)
+        self.assertRaises(ValueError, Metadata,
+                          mapping={'name': 'project',
+                                   'version': '1.0',
+                                   'provides_dist': ['my.pkg (splat)']})
+
+    def test_requires_dist(self):
+        fields = {'name': 'project',
+                  'version': '1.0',
+                  'requires_dist': ['other', 'another (==1.0)']}
+        metadata = Metadata(mapping=fields)
+        self.assertEqual(metadata['Requires-Dist'],
+                         ['other', 'another (==1.0)'])
+        self.assertEqual(metadata['Metadata-Version'], '1.2')
+        self.assertNotIn('Provides', metadata)
+        self.assertEqual(metadata['Requires-Dist'],
+                         ['other', 'another (==1.0)'])
+        self.assertNotIn('Obsoletes', metadata)
+
+        # make sure write_file uses one RFC 822 header per item
+        fp = StringIO()
+        metadata.write_file(fp)
+        lines = fp.getvalue().split('\n')
+        self.assertIn('Requires-Dist: other', lines)
+        self.assertIn('Requires-Dist: another (==1.0)', lines)
+
+        # test warnings for invalid version predicates
+        # XXX this would cause no warnings if we used update (or the mapping
+        # argument of the constructor), see comment in Metadata.update
         metadata = Metadata()
-        metadata.read_file(file_)
-        self.assertEqual(metadata['Project-Url'], [('one', 'http://ok')])
+        metadata['Requires-Dist'] = 'Funky (Groovie)'
+        metadata['Requires-Python'] = '1-4'
+        self.assertEqual(len(self.get_logs()), 2)
+
+        # test multiple version predicates
+        metadata = Metadata()
+
+        # XXX check PEP and see if 3 == 3.0
+        metadata['Requires-Python'] = '>=2.6, <3.0'
+        metadata['Requires-Dist'] = ['Foo (>=2.6, <3.0)']
+        self.assertEqual([], self.get_logs(logging.WARNING))
+
+    @unittest.skip('needs to be implemented')
+    def test_requires_illegal(self):
+        self.assertRaises(ValueError, Metadata,
+                          mapping={'name': 'project',
+                                   'version': '1.0',
+                                   'requires': ['my.pkg (splat)']})
+
+    def test_obsoletes_dist(self):
+        fields = {'name': 'project',
+                  'version': '1.0',
+                  'obsoletes_dist': ['other', 'another (<1.0)']}
+        metadata = Metadata(mapping=fields)
+        self.assertEqual(metadata['Obsoletes-Dist'],
+                         ['other', 'another (<1.0)'])
+        self.assertEqual(metadata['Metadata-Version'], '1.2')
+        self.assertNotIn('Provides', metadata)
+        self.assertNotIn('Requires', metadata)
+        self.assertEqual(metadata['Obsoletes-Dist'],
+                         ['other', 'another (<1.0)'])
+
+    @unittest.skip('needs to be implemented')
+    def test_obsoletes_illegal(self):
+        self.assertRaises(ValueError, Metadata,
+                          mapping={'name': 'project',
+                                   'version': '1.0',
+                                   'obsoletes': ['my.pkg (splat)']})
 
 
 def test_suite():
diff --git a/distutils2/tests/test_run.py b/distutils2/tests/test_run.py
--- a/distutils2/tests/test_run.py
+++ b/distutils2/tests/test_run.py
@@ -2,13 +2,14 @@
 
 import os
 import sys
-import shutil
 from StringIO import StringIO
 
 from distutils2 import install
 from distutils2.tests import unittest, support
 from distutils2.run import main
 
+from distutils2.tests.support import assert_python_ok
+
 # setup script that uses __file__
 setup_using___file__ = """\
 __file__
@@ -61,6 +62,22 @@
             os.chmod(install_path, old_mod)
             install.get_path = old_get_path
 
+    def test_show_help(self):
+        # smoke test, just makes sure some help is displayed
+        pythonpath = os.environ.get('PYTHONPATH')
+        d2parent = os.path.dirname(os.path.dirname(__file__))
+        if pythonpath is not None:
+            pythonpath =  os.pathsep.join((pythonpath, d2parent))
+        else:
+            pythonpath = d2parent
+
+        status, out, err = assert_python_ok(
+            '-c', 'from distutils2.run import main; main()', '--help',
+            PYTHONPATH=pythonpath)
+        self.assertEqual(status, 0)
+        self.assertGreater(out, '')
+        self.assertEqual(err, '')
+
 
 def test_suite():
     return unittest.makeSuite(RunTestCase)

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_usage_of_dry-run_in_?=
 =?utf8?q?bdist=5Fwininst_and_install=5Fdistinfo=2E?=
Message-ID: <E1R5deJ-0007mo-V2@dinsdale.python.org>

http://hg.python.org/distutils2/rev/8695472d3009
changeset:   1176:8695472d3009
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 04:53:51 2011 +0200
summary:
  Fix usage of dry-run in bdist_wininst and install_distinfo.

In dry-run mode, commands should log the same info as in real operation
and should collect the same files in self.outputs, so that users can run
a command in verbose and dry-run mode to see exactly what operations
will be done in the real run.

files:
  distutils2/command/bdist_wininst.py    |    5 +-
  distutils2/command/install_distinfo.py |  148 ++++++------
  2 files changed, 80 insertions(+), 73 deletions(-)


diff --git a/distutils2/command/bdist_wininst.py b/distutils2/command/bdist_wininst.py
--- a/distutils2/command/bdist_wininst.py
+++ b/distutils2/command/bdist_wininst.py
@@ -187,9 +187,8 @@
         os.remove(arcname)
 
         if not self.keep_temp:
-            if self.dry_run:
-                logger.info('removing %s', self.bdist_dir)
-            else:
+            logger.info('removing %s', self.bdist_dir)
+            if not self.dry_run:
                 rmtree(self.bdist_dir)
 
     def get_inidata(self):
diff --git a/distutils2/command/install_distinfo.py b/distutils2/command/install_distinfo.py
--- a/distutils2/command/install_distinfo.py
+++ b/distutils2/command/install_distinfo.py
@@ -2,18 +2,17 @@
 
 # Forked from the former install_egg_info command by Josip Djolonga
 
+import os
+import csv
 import codecs
-import csv
-import os
-import re
+from shutil import rmtree
 try:
     import hashlib
 except ImportError:
     from distutils2._backport import hashlib
 
+from distutils2 import logger
 from distutils2.command.cmd import Command
-from distutils2 import logger
-from shutil import rmtree
 
 
 class install_distinfo(Command):
@@ -72,81 +71,90 @@
         self.distinfo_dir = os.path.join(self.distinfo_dir, basename)
 
     def run(self):
-        # FIXME dry-run should be used at a finer level, so that people get
-        # useful logging output and can have an idea of what the command would
-        # have done
+        target = self.distinfo_dir
+
+        if os.path.isdir(target) and not os.path.islink(target):
+            if not self.dry_run:
+                rmtree(target)
+        elif os.path.exists(target):
+            self.execute(os.unlink, (self.distinfo_dir,),
+                         "removing " + target)
+
+        self.execute(os.makedirs, (target,), "creating " + target)
+
+        metadata_path = os.path.join(self.distinfo_dir, 'METADATA')
+        self.execute(self.distribution.metadata.write, (metadata_path,),
+                     "creating " + metadata_path)
+        self.outfiles.append(metadata_path)
+
+        installer_path = os.path.join(self.distinfo_dir, 'INSTALLER')
+        logger.info('creating %s', installer_path)
         if not self.dry_run:
-            target = self.distinfo_dir
+            f = open(installer_path, 'w')
+            try:
+                f.write(self.installer)
+            finally:
+                f.close()
+        self.outfiles.append(installer_path)
 
-            if os.path.isdir(target) and not os.path.islink(target):
-                rmtree(target)
-            elif os.path.exists(target):
-                self.execute(os.unlink, (self.distinfo_dir,),
-                             "removing " + target)
+        if self.requested:
+            requested_path = os.path.join(self.distinfo_dir, 'REQUESTED')
+            logger.info('creating %s', requested_path)
+            if not self.dry_run:
+                open(requested_path, 'wb').close()
+            self.outfiles.append(requested_path)
 
-            self.execute(os.makedirs, (target,), "creating " + target)
+        if not self.no_resources:
+            install_data = self.get_finalized_command('install_data')
+            if install_data.get_resources_out() != []:
+                resources_path = os.path.join(self.distinfo_dir,
+                                              'RESOURCES')
+                logger.info('creating %s', resources_path)
+                if not self.dry_run:
+                    f = open(resources_path, 'wb')
+                    try:
+                        writer = csv.writer(f, delimiter=',',
+                                            lineterminator='\n',
+                                            quotechar='"')
+                        for row in install_data.get_resources_out():
+                            writer.writerow(row)
+                    finally:
+                        f.close()
 
-            metadata_path = os.path.join(self.distinfo_dir, 'METADATA')
-            logger.info('creating %s', metadata_path)
-            self.distribution.metadata.write(metadata_path)
-            self.outfiles.append(metadata_path)
+                self.outfiles.append(resources_path)
 
-            installer_path = os.path.join(self.distinfo_dir, 'INSTALLER')
-            logger.info('creating %s', installer_path)
-            f = open(installer_path, 'w')
-            f.write(self.installer)
-            f.close()
-            self.outfiles.append(installer_path)
-
-            if self.requested:
-                requested_path = os.path.join(self.distinfo_dir, 'REQUESTED')
-                logger.info('creating %s', requested_path)
-                open(requested_path, 'wb').close()
-                self.outfiles.append(requested_path)
-
-
-            if not self.no_resources:
-                install_data = self.get_finalized_command('install_data')
-                if install_data.get_resources_out() != []:
-                    resources_path = os.path.join(self.distinfo_dir,
-                                                  'RESOURCES')
-                    logger.info('creating %s', resources_path)
-                    f = open(resources_path, 'wb')
+        if not self.no_record:
+            record_path = os.path.join(self.distinfo_dir, 'RECORD')
+            logger.info('creating %s', record_path)
+            if not self.dry_run:
+                f = codecs.open(record_path, 'w', encoding='utf-8')
+                try:
                     writer = csv.writer(f, delimiter=',',
                                         lineterminator='\n',
                                         quotechar='"')
-                    for tuple in install_data.get_resources_out():
-                        writer.writerow(tuple)
 
+                    install = self.get_finalized_command('install_dist')
+
+                    for fpath in install.get_outputs():
+                        if fpath.endswith('.pyc') or fpath.endswith('.pyo'):
+                            # do not put size and md5 hash, as in PEP-376
+                            writer.writerow((fpath, '', ''))
+                        else:
+                            size = os.path.getsize(fpath)
+                            fp = open(fpath, 'rb')
+                            try:
+                                hash = hashlib.md5()
+                                hash.update(fp.read())
+                            finally:
+                                fp.close()
+                            md5sum = hash.hexdigest()
+                            writer.writerow((fpath, md5sum, size))
+
+                    # add the RECORD file itself
+                    writer.writerow((record_path, '', ''))
+                finally:
                     f.close()
-                    self.outfiles.append(resources_path)
-
-            if not self.no_record:
-                record_path = os.path.join(self.distinfo_dir, 'RECORD')
-                logger.info('creating %s', record_path)
-                f = codecs.open(record_path, 'w', encoding='utf-8')
-                writer = csv.writer(f, delimiter=',',
-                                        lineterminator='\n',
-                                        quotechar='"')
-
-                install = self.get_finalized_command('install_dist')
-
-                for fpath in install.get_outputs():
-                    if fpath.endswith('.pyc') or fpath.endswith('.pyo'):
-                        # do not put size and md5 hash, as in PEP-376
-                        writer.writerow((fpath, '', ''))
-                    else:
-                        size = os.path.getsize(fpath)
-                        fp = open(fpath, 'rb')
-                        hash = hashlib.md5()
-                        hash.update(fp.read())
-                        fp.close()
-                        md5sum = hash.hexdigest()
-                        writer.writerow((fpath, md5sum, size))
-
-                # add the RECORD file itself
-                writer.writerow((record_path, '', ''))
-                self.outfiles.append(record_path)
+            self.outfiles.append(record_path)
 
     def get_outputs(self):
         return self.outfiles

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:40 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:40 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Don=E2=80=99t_let_invali?=
 =?utf8?q?d_line_in_setup=2Ecfg_pass_silently?=
Message-ID: <E1R5deK-0007n0-0b@dinsdale.python.org>

http://hg.python.org/distutils2/rev/50cccbea5538
changeset:   1179:50cccbea5538
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 05:02:28 2011 +0200
summary:
  Don?t let invalid line in setup.cfg pass silently

files:
  distutils2/config.py |  7 ++++---
  1 files changed, 4 insertions(+), 3 deletions(-)


diff --git a/distutils2/config.py b/distutils2/config.py
--- a/distutils2/config.py
+++ b/distutils2/config.py
@@ -231,10 +231,11 @@
                 self.dist.scripts = [self.dist.scripts]
 
             self.dist.package_data = {}
-            for data in files.get('package_data', []):
-                data = data.split('=')
+            for line in files.get('package_data', []):
+                data = line.split('=')
                 if len(data) != 2:
-                    continue  # XXX error should never pass silently
+                    raise ValueError('invalid line for package_data: %s '
+                                     '(misses "=")' % line)
                 key, value = data
                 globs = self.dist.package_data.setdefault(key.strip(), [])
                 globs.extend([v.strip() for v in value.split(',')])

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:39 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Better_bytes/characters_?=
 =?utf8?q?separation_for_d2=2Epypi=2Esimple?=
Message-ID: <E1R5deJ-0007ms-Ve@dinsdale.python.org>

http://hg.python.org/distutils2/rev/3c0df18caa89
changeset:   1177:3c0df18caa89
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 04:57:47 2011 +0200
summary:
  Better bytes/characters separation for d2.pypi.simple

files:
  distutils2/pypi/simple.py |  22 ++++++++++------------
  1 files changed, 10 insertions(+), 12 deletions(-)


diff --git a/distutils2/pypi/simple.py b/distutils2/pypi/simple.py
--- a/distutils2/pypi/simple.py
+++ b/distutils2/pypi/simple.py
@@ -157,25 +157,23 @@
 
         Return a list of names.
         """
+        if u'*' in name:
+            name.replace(u'*', u'.*')
+        else:
+            name = u"%s%s%s" % (u'*.?', name, u'*.?')
+        name = name.replace(u'*', u'[^<]*')  # avoid matching end tag
+        pattern = (u'<a[^>]*>(%s)</a>' % name).encode('utf-8')
+        projectname = re.compile(pattern, re.I)
+        matching_projects = []
+
         index = self._open_url(self.index_url)
         try:
-            if '*' in name:
-                name.replace('*', '.*')
-            else:
-                name = "%s%s%s" % ('*.?', name, '*.?')
-            name = name.replace('*', '[^<]*')  # avoid matching end tag
-            projectname = re.compile('<a[^>]*>(%s)</a>' % name, re.I)
-            matching_projects = []
-
             index_content = index.read()
         finally:
             index.close()
 
-        # FIXME should use bytes I/O and regexes instead of decoding
-        index_content = index_content.decode()
-
         for match in projectname.finditer(index_content):
-            project_name = match.group(1)
+            project_name = match.group(1).decode('utf-8')
             matching_projects.append(self._get_project(project_name))
         return matching_projects
 

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:40 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:40 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_usage_of_bytes_in_bd?=
	=?utf8?q?ist=5Fwininst=2E?=
Message-ID: <E1R5deK-0007mw-09@dinsdale.python.org>

http://hg.python.org/distutils2/rev/62c7c062516b
changeset:   1178:62c7c062516b
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 05:00:50 2011 +0200
summary:
  Fix usage of bytes in bdist_wininst.

This is copied from the namesake distutils command; I can?t test the
change but it should be okay, as Python 3 users have tested the
distutils command.

files:
  distutils2/command/bdist_wininst.py |  10 ++++++----
  1 files changed, 6 insertions(+), 4 deletions(-)


diff --git a/distutils2/command/bdist_wininst.py b/distutils2/command/bdist_wininst.py
--- a/distutils2/command/bdist_wininst.py
+++ b/distutils2/command/bdist_wininst.py
@@ -1,9 +1,8 @@
 """Create an executable installer for Windows."""
 
-# FIXME synchronize bytes/str use with same file in distutils
-
 import sys
 import os
+import codecs
 
 from shutil import rmtree
 
@@ -271,9 +270,12 @@
             # Append the pre-install script
             cfgdata = cfgdata + "\0"
             if self.pre_install_script:
-                fp = open(self.pre_install_script)
+                # We need to normalize newlines, so we open in text mode and
+                # convert back to bytes. "latin-1" simply avoids any possible
+                # failures.
+                fp = codecs.open(self.pre_install_script, encoding="latin-1")
                 try:
-                    script_data = fp.read()
+                    script_data = fp.read().encode("latin-1")
                 finally:
                     fp.close()
                 cfgdata = cfgdata + script_data + "\n\0"

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:40 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:40 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_imports=3A_it_really?=
 =?utf8?q?_is_distutils=2Espawn=2C_not_d2?=
Message-ID: <E1R5deK-0007n4-1L@dinsdale.python.org>

http://hg.python.org/distutils2/rev/ead3e3c1bc06
changeset:   1180:ead3e3c1bc06
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 15:11:25 2011 +0200
summary:
  Fix imports: it really is distutils.spawn, not d2

files:
  distutils2/_backport/shutil.py |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/distutils2/_backport/shutil.py b/distutils2/_backport/shutil.py
--- a/distutils2/_backport/shutil.py
+++ b/distutils2/_backport/shutil.py
@@ -446,8 +446,8 @@
         zipoptions = "-r"
     else:
         zipoptions = "-rq"
-    from distutils2.errors import DistutilsExecError
-    from distutils2.spawn import spawn
+    from distutils.errors import DistutilsExecError
+    from distutils.spawn import spawn
     try:
         spawn(["zip", zipoptions, zip_filename, base_dir], dry_run=dry_run)
     except DistutilsExecError:

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 15:12:40 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 15:12:40 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_A_few_changes_to_match_p?=
	=?utf8?q?ackaging?=
Message-ID: <E1R5deK-0007n8-1v@dinsdale.python.org>

http://hg.python.org/distutils2/rev/c9b22c0b5f3e
changeset:   1181:c9b22c0b5f3e
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 15:11:51 2011 +0200
summary:
  A few changes to match packaging

files:
  distutils2/_backport/__init__.py |  2 +-
  distutils2/install.py            |  5 ++---
  distutils2/tests/support.py      |  1 +
  3 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/distutils2/_backport/__init__.py b/distutils2/_backport/__init__.py
--- a/distutils2/_backport/__init__.py
+++ b/distutils2/_backport/__init__.py
@@ -1,4 +1,4 @@
-"""Modules copied from the Python 3.2 standard library.
+"""Modules copied from Python 3 standard libraries.
 
 Individual classes and objects like the any function are in compat.
 """
diff --git a/distutils2/install.py b/distutils2/install.py
--- a/distutils2/install.py
+++ b/distutils2/install.py
@@ -10,7 +10,6 @@
 import sys
 import stat
 import errno
-import shutil
 import logging
 import tempfile
 
@@ -27,7 +26,7 @@
                                InstallationConflict, CCompilerError)
 from distutils2.pypi.errors import ProjectNotFound, ReleaseNotFound
 from distutils2 import database
-from distutils2._backport.shutil import unpack_archive
+from distutils2._backport import shutil
 from distutils2._backport.sysconfig import (get_config_var, get_path,
                                             is_python_build)
 
@@ -126,7 +125,7 @@
         logger.info('Installing from archive: %r', path)
         _unpacked_dir = tempfile.mkdtemp()
         try:
-            unpack_archive(path, _unpacked_dir)
+            shutil.unpack_archive(path, _unpacked_dir)
             return _run_install_from_archive(_unpacked_dir)
         finally:
             shutil.rmtree(_unpacked_dir)
diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -351,6 +351,7 @@
         if runshared is None:
             cmd.library_dirs = ['.']
         else:
+            # FIXME no partition in 2.4
             name, equals, value = runshared.partition('=')
             cmd.library_dirs = value.split(os.pathsep)
 

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Sep 19 16:10:49 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 16:10:49 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Packaging_cleanup=3A_remove?=
 =?utf8?q?_conditionals_for_=3C_2=2E6_support=2E?=
Message-ID: <E1R5eYb-0005Ej-RT@dinsdale.python.org>

http://hg.python.org/cpython/rev/60deed9538fa
changeset:   72401:60deed9538fa
parent:      72384:c6d52971dd2a
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 17 03:31:51 2011 +0200
summary:
  Packaging cleanup: remove conditionals for < 2.6 support.

PEP 370 features and sys.dont_write_bytecode are always available
in 3.3; the distutils2 backport still has the conditionals.

I also renamed an internal misnamed method and fixed a few things
(?packaging2? name, stray print, unused import, fd leak).

files:
  Lib/packaging/command/build_ext.py               |  17 +-
  Lib/packaging/command/build_py.py                |   2 +-
  Lib/packaging/command/install_dist.py            |  59 +++------
  Lib/packaging/command/install_distinfo.py        |   7 +-
  Lib/packaging/command/install_lib.py             |   2 +-
  Lib/packaging/compiler/bcppcompiler.py           |   1 -
  Lib/packaging/compiler/ccompiler.py              |   1 -
  Lib/packaging/compiler/unixccompiler.py          |   2 +-
  Lib/packaging/dist.py                            |   4 +-
  Lib/packaging/tests/test_command_build_ext.py    |   8 +-
  Lib/packaging/tests/test_command_build_py.py     |   2 -
  Lib/packaging/tests/test_command_install_dist.py |  12 +-
  Lib/packaging/tests/test_command_install_lib.py  |  11 +-
  Lib/packaging/tests/test_mixin2to3.py            |   3 -
  Lib/packaging/tests/test_util.py                 |   4 -
  Lib/packaging/util.py                            |   6 +-
  16 files changed, 46 insertions(+), 95 deletions(-)


diff --git a/Lib/packaging/command/build_ext.py b/Lib/packaging/command/build_ext.py
--- a/Lib/packaging/command/build_ext.py
+++ b/Lib/packaging/command/build_ext.py
@@ -3,6 +3,7 @@
 import os
 import re
 import sys
+import site
 import logging
 import sysconfig
 
@@ -15,9 +16,6 @@
 from packaging.compiler.extension import Extension
 from packaging import logger
 
-import site
-HAS_USER_SITE = True
-
 if os.name == 'nt':
     from packaging.compiler.msvccompiler import get_build_version
     MSVC_VERSION = int(get_build_version())
@@ -62,6 +60,8 @@
         ('inplace', 'i',
          "ignore build-lib and put compiled extensions into the source " +
          "directory alongside your pure Python modules"),
+        ('user', None,
+         "add user include, library and rpath"),
         ('include-dirs=', 'I',
          "list of directories to search for header files" + sep_by),
         ('define=', 'D',
@@ -88,12 +88,8 @@
          "path to the SWIG executable"),
         ]
 
-    boolean_options = ['inplace', 'debug', 'force']
+    boolean_options = ['inplace', 'debug', 'force', 'user']
 
-    if HAS_USER_SITE:
-        user_options.append(('user', None,
-                             "add user include, library and rpath"))
-        boolean_options.append('user')
 
     help_options = [
         ('help-compiler', None,
@@ -120,8 +116,7 @@
         self.compiler = None
         self.swig = None
         self.swig_opts = None
-        if HAS_USER_SITE:
-            self.user = None
+        self.user = None
 
     def finalize_options(self):
         self.set_undefined_options('build',
@@ -270,7 +265,7 @@
             self.swig_opts = self.swig_opts.split(' ')
 
         # Finally add the user include and library directories if requested
-        if HAS_USER_SITE and self.user:
+        if self.user:
             user_include = os.path.join(site.USER_BASE, "include")
             user_lib = os.path.join(site.USER_BASE, "lib")
             if os.path.isdir(user_include):
diff --git a/Lib/packaging/command/build_py.py b/Lib/packaging/command/build_py.py
--- a/Lib/packaging/command/build_py.py
+++ b/Lib/packaging/command/build_py.py
@@ -388,7 +388,7 @@
                 self.build_module(module, module_file, package)
 
     def byte_compile(self, files):
-        if hasattr(sys, 'dont_write_bytecode') and sys.dont_write_bytecode:
+        if sys.dont_write_bytecode:
             logger.warning('%s: byte-compiling is disabled, skipping.',
                            self.get_command_name())
             return
diff --git a/Lib/packaging/command/install_dist.py b/Lib/packaging/command/install_dist.py
--- a/Lib/packaging/command/install_dist.py
+++ b/Lib/packaging/command/install_dist.py
@@ -14,9 +14,6 @@
 from packaging.errors import PackagingOptionError
 
 
-HAS_USER_SITE = True
-
-
 class install_dist(Command):
 
     description = "install everything from build directory"
@@ -27,6 +24,9 @@
          "installation prefix"),
         ('exec-prefix=', None,
          "(Unix only) prefix for platform-specific files"),
+        ('user', None,
+         "install in user site-packages directory [%s]" %
+         get_path('purelib', '%s_user' % os.name)),
         ('home=', None,
          "(Unix only) home directory to install under"),
 
@@ -97,15 +97,7 @@
         ]
 
     boolean_options = ['compile', 'force', 'skip-build', 'no-distinfo',
-                       'requested', 'no-record']
-
-    if HAS_USER_SITE:
-        user_options.append(
-            ('user', None,
-             "install in user site-packages directory [%s]" %
-             get_path('purelib', '%s_user' % os.name)))
-
-        boolean_options.append('user')
+                       'requested', 'no-record', 'user']
 
     negative_opt = {'no-compile': 'compile', 'no-requested': 'requested'}
 
@@ -115,8 +107,7 @@
         self.prefix = None
         self.exec_prefix = None
         self.home = None
-        if HAS_USER_SITE:
-            self.user = False
+        self.user = False
 
         # These select only the installation base; it's up to the user to
         # specify the installation scheme (currently, that means supplying
@@ -135,9 +126,8 @@
         self.install_lib = None         # set to either purelib or platlib
         self.install_scripts = None
         self.install_data = None
-        if HAS_USER_SITE:
-            self.install_userbase = get_config_var('userbase')
-            self.install_usersite = get_path('purelib', '%s_user' % os.name)
+        self.install_userbase = get_config_var('userbase')
+        self.install_usersite = get_path('purelib', '%s_user' % os.name)
 
         self.compile = None
         self.optimize = None
@@ -219,9 +209,8 @@
             raise PackagingOptionError(
                 "must supply either home or prefix/exec-prefix -- not both")
 
-        if HAS_USER_SITE and self.user and (
-                self.prefix or self.exec_prefix or self.home or
-                self.install_base or self.install_platbase):
+        if self.user and (self.prefix or self.exec_prefix or self.home or
+                          self.install_base or self.install_platbase):
             raise PackagingOptionError(
                 "can't combine user with prefix/exec_prefix/home or "
                 "install_base/install_platbase")
@@ -274,11 +263,9 @@
             'exec_prefix': exec_prefix,
             'srcdir': srcdir,
             'projectbase': projectbase,
-            }
-
-        if HAS_USER_SITE:
-            self.config_vars['userbase'] = self.install_userbase
-            self.config_vars['usersite'] = self.install_usersite
+            'userbase': self.install_userbase,
+            'usersite': self.install_usersite,
+        }
 
         self.expand_basedirs()
 
@@ -295,9 +282,9 @@
 
         self.dump_dirs("post-expand_dirs()")
 
-        # Create directories in the home dir:
-        if HAS_USER_SITE and self.user:
-            self.create_home_path()
+        # Create directories under USERBASE
+        if self.user:
+            self.create_user_dirs()
 
         # Pick the actual directory to install all modules to: either
         # install_purelib or install_platlib, depending on whether this
@@ -311,10 +298,8 @@
 
         # Convert directories from Unix /-separated syntax to the local
         # convention.
-        self.convert_paths('lib', 'purelib', 'platlib',
-                           'scripts', 'data', 'headers')
-        if HAS_USER_SITE:
-            self.convert_paths('userbase', 'usersite')
+        self.convert_paths('lib', 'purelib', 'platlib', 'scripts',
+                           'data', 'headers', 'userbase', 'usersite')
 
         # Well, we're not actually fully completely finalized yet: we still
         # have to deal with 'extra_path', which is the hack for allowing
@@ -355,7 +340,7 @@
                     "installation scheme is incomplete")
             return
 
-        if HAS_USER_SITE and self.user:
+        if self.user:
             if self.install_userbase is None:
                 raise PackagingPlatformError(
                     "user base directory is not specified")
@@ -383,7 +368,7 @@
 
     def finalize_other(self):
         """Finalize options for non-posix platforms"""
-        if HAS_USER_SITE and self.user:
+        if self.user:
             if self.install_userbase is None:
                 raise PackagingPlatformError(
                     "user base directory is not specified")
@@ -494,10 +479,8 @@
             attr = "install_" + name
             setattr(self, attr, change_root(self.root, getattr(self, attr)))
 
-    def create_home_path(self):
-        """Create directories under ~."""
-        if HAS_USER_SITE and not self.user:
-            return
+    def create_user_dirs(self):
+        """Create directories under USERBASE as needed."""
         home = convert_path(os.path.expanduser("~"))
         for name, path in self.config_vars.items():
             if path.startswith(home) and not os.path.isdir(path):
diff --git a/Lib/packaging/command/install_distinfo.py b/Lib/packaging/command/install_distinfo.py
--- a/Lib/packaging/command/install_distinfo.py
+++ b/Lib/packaging/command/install_distinfo.py
@@ -2,14 +2,13 @@
 
 # Forked from the former install_egg_info command by Josip Djolonga
 
+import os
 import csv
-import os
-import re
 import hashlib
+from shutil import rmtree
 
+from packaging import logger
 from packaging.command.cmd import Command
-from packaging import logger
-from shutil import rmtree
 
 
 class install_distinfo(Command):
diff --git a/Lib/packaging/command/install_lib.py b/Lib/packaging/command/install_lib.py
--- a/Lib/packaging/command/install_lib.py
+++ b/Lib/packaging/command/install_lib.py
@@ -114,7 +114,7 @@
         return outfiles
 
     def byte_compile(self, files):
-        if getattr(sys, 'dont_write_bytecode'):
+        if sys.dont_write_bytecode:
             # XXX do we want this?  because a Python runs without bytecode
             # doesn't mean that the *dists should not contain bytecode
             #--or does it?
diff --git a/Lib/packaging/compiler/bcppcompiler.py b/Lib/packaging/compiler/bcppcompiler.py
--- a/Lib/packaging/compiler/bcppcompiler.py
+++ b/Lib/packaging/compiler/bcppcompiler.py
@@ -352,5 +352,4 @@
             try:
                 self.spawn(pp_args)
             except PackagingExecError as msg:
-                print(msg)
                 raise CompileError(msg)
diff --git a/Lib/packaging/compiler/ccompiler.py b/Lib/packaging/compiler/ccompiler.py
--- a/Lib/packaging/compiler/ccompiler.py
+++ b/Lib/packaging/compiler/ccompiler.py
@@ -5,7 +5,6 @@
 """
 
 import os
-import sys
 from shutil import move
 from packaging import logger
 from packaging.util import split_quoted, execute, newer_group, spawn
diff --git a/Lib/packaging/compiler/unixccompiler.py b/Lib/packaging/compiler/unixccompiler.py
--- a/Lib/packaging/compiler/unixccompiler.py
+++ b/Lib/packaging/compiler/unixccompiler.py
@@ -127,7 +127,7 @@
         executables['ranlib'] = ["ranlib"]
 
     # Needed for the filename generation methods provided by the base
-    # class, CCompiler.  NB. whoever instantiates/uses a particular
+    # class, CCompiler.  XXX whoever instantiates/uses a particular
     # UnixCCompiler instance should set 'shared_lib_ext' -- we set a
     # reasonable common default here, but it's not necessarily used on all
     # Unices!
diff --git a/Lib/packaging/dist.py b/Lib/packaging/dist.py
--- a/Lib/packaging/dist.py
+++ b/Lib/packaging/dist.py
@@ -537,7 +537,7 @@
     def _get_command_groups(self):
         """Helper function to retrieve all the command class names divided
         into standard commands (listed in
-        packaging2.command.STANDARD_COMMANDS) and extra commands (given in
+        packaging.command.STANDARD_COMMANDS) and extra commands (given in
         self.cmdclass and not standard commands).
         """
         extra_commands = [cmd for cmd in self.cmdclass
@@ -547,7 +547,7 @@
     def print_commands(self):
         """Print out a help message listing all available commands with a
         description of each.  The list is divided into standard commands
-        (listed in packaging2.command.STANDARD_COMMANDS) and extra commands
+        (listed in packaging.command.STANDARD_COMMANDS) and extra commands
         (given in self.cmdclass and not standard commands).  The
         descriptions come from the command class attribute
         'description'.
diff --git a/Lib/packaging/tests/test_command_build_ext.py b/Lib/packaging/tests/test_command_build_ext.py
--- a/Lib/packaging/tests/test_command_build_ext.py
+++ b/Lib/packaging/tests/test_command_build_ext.py
@@ -18,18 +18,13 @@
                        support.LoggingCatcher,
                        unittest.TestCase):
     def setUp(self):
-        # Create a simple test environment
-        # Note that we're making changes to sys.path
         super(BuildExtTestCase, self).setUp()
         self.tmp_dir = self.mkdtemp()
         self.old_user_base = site.USER_BASE
         site.USER_BASE = self.mkdtemp()
 
     def tearDown(self):
-        # Get everything back to normal
-        if sys.version > "2.6":
-            site.USER_BASE = self.old_user_base
-
+        site.USER_BASE = self.old_user_base
         super(BuildExtTestCase, self).tearDown()
 
     def test_build_ext(self):
@@ -94,7 +89,6 @@
         # make sure we get some library dirs under solaris
         self.assertGreater(len(cmd.library_dirs), 0)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_user_site(self):
         dist = Distribution({'name': 'xx'})
         cmd = build_ext(dist)
diff --git a/Lib/packaging/tests/test_command_build_py.py b/Lib/packaging/tests/test_command_build_py.py
--- a/Lib/packaging/tests/test_command_build_py.py
+++ b/Lib/packaging/tests/test_command_build_py.py
@@ -99,8 +99,6 @@
             os.chdir(cwd)
             sys.stdout = old_stdout
 
-    @unittest.skipUnless(hasattr(sys, 'dont_write_bytecode'),
-                         'sys.dont_write_bytecode not supported')
     def test_dont_write_bytecode(self):
         # makes sure byte_compile is not used
         pkg_dir, dist = self.create_dist()
diff --git a/Lib/packaging/tests/test_command_install_dist.py b/Lib/packaging/tests/test_command_install_dist.py
--- a/Lib/packaging/tests/test_command_install_dist.py
+++ b/Lib/packaging/tests/test_command_install_dist.py
@@ -72,7 +72,6 @@
         check_path(cmd.install_scripts, os.path.join(destination, "bin"))
         check_path(cmd.install_data, destination)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_user_site(self):
         # test install with --user
         # preparing the environment for the test
@@ -173,12 +172,11 @@
         cmd.home = 'home'
         self.assertRaises(PackagingOptionError, cmd.finalize_options)
 
-        if sys.version >= '2.6':
-            # can't combine user with with prefix/exec_prefix/home or
-            # install_(plat)base
-            cmd.prefix = None
-            cmd.user = 'user'
-            self.assertRaises(PackagingOptionError, cmd.finalize_options)
+        # can't combine user with with prefix/exec_prefix/home or
+        # install_(plat)base
+        cmd.prefix = None
+        cmd.user = 'user'
+        self.assertRaises(PackagingOptionError, cmd.finalize_options)
 
     def test_old_record(self):
         # test pre-PEP 376 --record option (outside dist-info dir)
diff --git a/Lib/packaging/tests/test_command_install_lib.py b/Lib/packaging/tests/test_command_install_lib.py
--- a/Lib/packaging/tests/test_command_install_lib.py
+++ b/Lib/packaging/tests/test_command_install_lib.py
@@ -7,13 +7,6 @@
 from packaging.compiler.extension import Extension
 from packaging.errors import PackagingOptionError
 
-try:
-    no_bytecode = sys.dont_write_bytecode
-    bytecode_support = True
-except AttributeError:
-    no_bytecode = False
-    bytecode_support = False
-
 
 class InstallLibTestCase(support.TempdirManager,
                          support.LoggingCatcher,
@@ -40,7 +33,7 @@
         cmd.finalize_options()
         self.assertEqual(cmd.optimize, 2)
 
-    @unittest.skipIf(no_bytecode, 'byte-compile not supported')
+    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile not supported')
     def test_byte_compile(self):
         pkg_dir, dist = self.create_dist()
         cmd = install_lib(dist)
@@ -89,8 +82,6 @@
         # get_input should return 2 elements
         self.assertEqual(len(cmd.get_inputs()), 2)
 
-    @unittest.skipUnless(bytecode_support,
-                         'sys.dont_write_bytecode not supported')
     def test_dont_write_bytecode(self):
         # makes sure byte_compile is not used
         pkg_dir, dist = self.create_dist()
diff --git a/Lib/packaging/tests/test_mixin2to3.py b/Lib/packaging/tests/test_mixin2to3.py
--- a/Lib/packaging/tests/test_mixin2to3.py
+++ b/Lib/packaging/tests/test_mixin2to3.py
@@ -9,7 +9,6 @@
                         support.LoggingCatcher,
                         unittest.TestCase):
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_convert_code_only(self):
         # used to check if code gets converted properly.
         code = "print 'test'"
@@ -26,7 +25,6 @@
 
         self.assertEqual(expected, converted)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_doctests_only(self):
         # used to check if doctests gets converted properly.
         doctest = textwrap.dedent('''\
@@ -57,7 +55,6 @@
 
         self.assertEqual(expected, converted)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_additional_fixers(self):
         # used to check if use_2to3_fixers works
         code = 'type(x) is not T'
diff --git a/Lib/packaging/tests/test_util.py b/Lib/packaging/tests/test_util.py
--- a/Lib/packaging/tests/test_util.py
+++ b/Lib/packaging/tests/test_util.py
@@ -319,8 +319,6 @@
         res = get_compiler_versions()
         self.assertEqual(res[2], None)
 
-    @unittest.skipUnless(hasattr(sys, 'dont_write_bytecode'),
-                         'sys.dont_write_bytecode not supported')
     def test_dont_write_bytecode(self):
         # makes sure byte_compile raise a PackagingError
         # if sys.dont_write_bytecode is True
@@ -407,7 +405,6 @@
         finally:
             sys.path.remove(tmp_dir)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_run_2to3_on_code(self):
         content = "print 'test'"
         converted_content = "print('test')"
@@ -422,7 +419,6 @@
         file_handle.close()
         self.assertEqual(new_content, converted_content)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_run_2to3_on_doctests(self):
         # to check if text files containing doctests only get converted.
         content = ">>> print 'test'\ntest\n"
diff --git a/Lib/packaging/util.py b/Lib/packaging/util.py
--- a/Lib/packaging/util.py
+++ b/Lib/packaging/util.py
@@ -326,7 +326,7 @@
     """
     # nothing is done if sys.dont_write_bytecode is True
     # FIXME this should not raise an error
-    if hasattr(sys, 'dont_write_bytecode') and sys.dont_write_bytecode:
+    if sys.dont_write_bytecode:
         raise PackagingByteCompileError('byte-compiling is disabled.')
 
     # First, if the caller didn't force us into direct or indirect mode,
@@ -346,8 +346,10 @@
     # run it with the appropriate flags.
     if not direct:
         from tempfile import mkstemp
-        # XXX script_fd may leak, use something better than mkstemp
+        # XXX use something better than mkstemp
         script_fd, script_name = mkstemp(".py")
+        os.close(script_fd)
+        script_fd = None
         logger.info("writing byte-compilation script '%s'", script_name)
         if not dry_run:
             if script_fd is not None:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 16:10:50 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 16:10:50 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Use_a_local_name_=28it_was_?=
 =?utf8?q?intended_to_be_used_but_overlooked=29=2E?=
Message-ID: <E1R5eYc-0005En-I0@dinsdale.python.org>

http://hg.python.org/cpython/rev/f18e9db83483
changeset:   72402:f18e9db83483
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 17 03:35:57 2011 +0200
summary:
  Use a local name (it was intended to be used but overlooked).

This was caught in the distutils2 repo by pyflakes.

files:
  Lib/pkgutil.py |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Lib/pkgutil.py b/Lib/pkgutil.py
--- a/Lib/pkgutil.py
+++ b/Lib/pkgutil.py
@@ -307,9 +307,9 @@
     def get_filename(self, fullname=None):
         fullname = self._fix_name(fullname)
         mod_type = self.etc[2]
-        if self.etc[2]==imp.PKG_DIRECTORY:
+        if mod_type==imp.PKG_DIRECTORY:
             return self._get_delegate().get_filename()
-        elif self.etc[2] in (imp.PY_SOURCE, imp.PY_COMPILED, imp.C_EXTENSION):
+        elif mod_type in (imp.PY_SOURCE, imp.PY_COMPILED, imp.C_EXTENSION):
             return self.filename
         return None
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 16:10:51 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 16:10:51 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_fallback_base_class_whe?=
 =?utf8?q?n_tests_run_without_threading?=
Message-ID: <E1R5eYd-0005Er-B1@dinsdale.python.org>

http://hg.python.org/cpython/rev/6feda7ae35c3
changeset:   72403:6feda7ae35c3
parent:      72391:d3e072083ff3
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Sep 18 17:00:38 2011 +0200
summary:
  Fix fallback base class when tests run without threading

files:
  Lib/packaging/tests/test_command_upload_docs.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/packaging/tests/test_command_upload_docs.py b/Lib/packaging/tests/test_command_upload_docs.py
--- a/Lib/packaging/tests/test_command_upload_docs.py
+++ b/Lib/packaging/tests/test_command_upload_docs.py
@@ -19,7 +19,7 @@
     from packaging.tests.pypi_server import PyPIServerTestCase
 except ImportError:
     threading = None
-    PyPIServerTestCase = object
+    PyPIServerTestCase = unittest.TestCase
 
 
 PYPIRC = """\

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 16:10:52 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 16:10:52 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Make_a_number_of_small_chan?=
 =?utf8?q?ges_to_ease_the_backport_to_distutils2?=
Message-ID: <E1R5eYe-0005Ev-2S@dinsdale.python.org>

http://hg.python.org/cpython/rev/a400c1c087dd
changeset:   72404:a400c1c087dd
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Sep 18 20:11:48 2011 +0200
summary:
  Make a number of small changes to ease the backport to distutils2

files:
  Lib/packaging/command/build_clib.py             |   3 +-
  Lib/packaging/command/upload_docs.py            |   1 +
  Lib/packaging/compiler/unixccompiler.py         |   2 +-
  Lib/packaging/database.py                       |   7 +-
  Lib/packaging/tests/__init__.py                 |  13 +-
  Lib/packaging/tests/__main__.py                 |   2 +
  Lib/packaging/tests/pypi_server.py              |   7 +-
  Lib/packaging/tests/test_command_build_ext.py   |   2 +-
  Lib/packaging/tests/test_command_install_lib.py |   2 +-
  Lib/packaging/tests/test_command_upload.py      |  24 +--
  Lib/packaging/tests/test_command_upload_docs.py |  65 ++++-----
  Lib/packaging/tests/test_create.py              |  13 +-
  Lib/packaging/tests/test_database.py            |   5 +-
  Lib/packaging/tests/test_depgraph.py            |  33 ++--
  Lib/packaging/tests/test_dist.py                |  10 +-
  Lib/packaging/tests/test_install.py             |   2 +-
  Lib/packaging/tests/test_pypi_server.py         |  15 +-
  Lib/packaging/tests/test_run.py                 |  17 +--
  Lib/packaging/tests/test_util.py                |   6 +-
  19 files changed, 104 insertions(+), 125 deletions(-)


diff --git a/Lib/packaging/command/build_clib.py b/Lib/packaging/command/build_clib.py
--- a/Lib/packaging/command/build_clib.py
+++ b/Lib/packaging/command/build_clib.py
@@ -16,7 +16,7 @@
 import os
 from packaging.command.cmd import Command
 from packaging.errors import PackagingSetupError
-from packaging.compiler import customize_compiler
+from packaging.compiler import customize_compiler, new_compiler
 from packaging import logger
 
 
@@ -93,7 +93,6 @@
             return
 
         # Yech -- this is cut 'n pasted from build_ext.py!
-        from packaging.compiler import new_compiler
         self.compiler = new_compiler(compiler=self.compiler,
                                      dry_run=self.dry_run,
                                      force=self.force)
diff --git a/Lib/packaging/command/upload_docs.py b/Lib/packaging/command/upload_docs.py
--- a/Lib/packaging/command/upload_docs.py
+++ b/Lib/packaging/command/upload_docs.py
@@ -87,6 +87,7 @@
         content_type, body = encode_multipart(fields, files)
 
         credentials = self.username + ':' + self.password
+        # FIXME should use explicit encoding
         auth = b"Basic " + base64.encodebytes(credentials.encode()).strip()
 
         logger.info("Submitting documentation to %s", self.repository)
diff --git a/Lib/packaging/compiler/unixccompiler.py b/Lib/packaging/compiler/unixccompiler.py
--- a/Lib/packaging/compiler/unixccompiler.py
+++ b/Lib/packaging/compiler/unixccompiler.py
@@ -127,7 +127,7 @@
         executables['ranlib'] = ["ranlib"]
 
     # Needed for the filename generation methods provided by the base
-    # class, CCompiler.  NB. whoever instantiates/uses a particular
+    # class, CCompiler.  XXX whoever instantiates/uses a particular
     # UnixCCompiler instance should set 'shared_lib_ext' -- we set a
     # reasonable common default here, but it's not necessarily used on all
     # Unices!
diff --git a/Lib/packaging/database.py b/Lib/packaging/database.py
--- a/Lib/packaging/database.py
+++ b/Lib/packaging/database.py
@@ -1,12 +1,13 @@
 """PEP 376 implementation."""
 
-import io
 import os
 import re
 import csv
 import sys
 import zipimport
+from io import StringIO
 from hashlib import md5
+
 from packaging import logger
 from packaging.errors import PackagingError
 from packaging.version import suggest_normalized_version, VersionPredicate
@@ -173,7 +174,7 @@
     def get_resource_path(self, relative_path):
         with self.get_distinfo_file('RESOURCES') as resources_file:
             resources_reader = csv.reader(resources_file, delimiter=',',
-                                           lineterminator='\n')
+                                          lineterminator='\n')
             for relative, destination in resources_reader:
                 if relative == relative_path:
                     return destination
@@ -334,7 +335,7 @@
             else:
                 # FIXME handle the case where zipfile is not available
                 zipf = zipimport.zipimporter(path)
-                fileobj = io.StringIO(
+                fileobj = StringIO(
                     zipf.get_data('EGG-INFO/PKG-INFO').decode('utf8'))
                 self.metadata = Metadata(fileobj=fileobj)
                 try:
diff --git a/Lib/packaging/tests/__init__.py b/Lib/packaging/tests/__init__.py
--- a/Lib/packaging/tests/__init__.py
+++ b/Lib/packaging/tests/__init__.py
@@ -6,17 +6,15 @@
 to return an initialized unittest.TestSuite instance.
 
 Utility code is included in packaging.tests.support.
+
+Always import unittest from this module: it will be unittest from the
+standard library for packaging tests and unittest2 for distutils2 tests.
 """
 
-# Put this text back for the backport
-#Always import unittest from this module, it will be the right version
-#(standard library unittest for 3.2 and higher, third-party unittest2
-#elease for older versions).
-
 import os
 import sys
 import unittest
-from test.support import TESTFN
+from io import StringIO
 
 # XXX move helpers to support, add tests for them, remove things that
 # duplicate test.support (or keep them for the backport; needs thinking)
@@ -115,9 +113,8 @@
 
 
 def captured_stdout(func, *args, **kw):
-    import io
     orig_stdout = getattr(sys, 'stdout')
-    setattr(sys, 'stdout', io.StringIO())
+    setattr(sys, 'stdout', StringIO())
     try:
         res = func(*args, **kw)
         sys.stdout.seek(0)
diff --git a/Lib/packaging/tests/__main__.py b/Lib/packaging/tests/__main__.py
--- a/Lib/packaging/tests/__main__.py
+++ b/Lib/packaging/tests/__main__.py
@@ -14,6 +14,8 @@
         start_dir = os.path.dirname(__file__)
         top_dir = os.path.dirname(os.path.dirname(start_dir))
         test_loader = unittest.TestLoader()
+        # XXX find out how to use unittest.main, to get command-line options
+        # (failfast, catch, etc.)
         run_unittest(test_loader.discover(start_dir, top_level_dir=top_dir))
     finally:
         reap_children()
diff --git a/Lib/packaging/tests/pypi_server.py b/Lib/packaging/tests/pypi_server.py
--- a/Lib/packaging/tests/pypi_server.py
+++ b/Lib/packaging/tests/pypi_server.py
@@ -40,6 +40,7 @@
 
 from packaging.tests import unittest
 
+
 PYPI_DEFAULT_STATIC_PATH = os.path.join(
     os.path.dirname(os.path.abspath(__file__)), 'pypiserver')
 
@@ -219,7 +220,7 @@
                         relative_path += "index.html"
 
                     if relative_path.endswith('.tar.gz'):
-                        with open(fs_path + relative_path, 'br') as file:
+                        with open(fs_path + relative_path, 'rb') as file:
                             data = file.read()
                         headers = [('Content-type', 'application/x-gtar')]
                     else:
@@ -260,8 +261,8 @@
             self.send_header(header, value)
         self.end_headers()
 
-        if type(data) is str:
-            data = data.encode()
+        if isinstance(data, str):
+            data = data.encode('utf-8')
 
         self.wfile.write(data)
 
diff --git a/Lib/packaging/tests/test_command_build_ext.py b/Lib/packaging/tests/test_command_build_ext.py
--- a/Lib/packaging/tests/test_command_build_ext.py
+++ b/Lib/packaging/tests/test_command_build_ext.py
@@ -4,6 +4,7 @@
 import sysconfig
 import textwrap
 from io import StringIO
+from sysconfig import _CONFIG_VARS
 from packaging.dist import Distribution
 from packaging.errors import (UnknownFileError, CompileError,
                               PackagingPlatformError)
@@ -78,7 +79,6 @@
         old = sys.platform
 
         sys.platform = 'sunos'  # fooling finalize_options
-        from sysconfig import _CONFIG_VARS
 
         old_var = _CONFIG_VARS.get('Py_ENABLE_SHARED')
         _CONFIG_VARS['Py_ENABLE_SHARED'] = 1
diff --git a/Lib/packaging/tests/test_command_install_lib.py b/Lib/packaging/tests/test_command_install_lib.py
--- a/Lib/packaging/tests/test_command_install_lib.py
+++ b/Lib/packaging/tests/test_command_install_lib.py
@@ -40,7 +40,7 @@
         cmd.finalize_options()
         self.assertEqual(cmd.optimize, 2)
 
-    @unittest.skipIf(no_bytecode, 'byte-compile not supported')
+    @unittest.skipIf(no_bytecode, 'byte-compile disabled')
     def test_byte_compile(self):
         pkg_dir, dist = self.create_dist()
         cmd = install_lib(dist)
diff --git a/Lib/packaging/tests/test_command_upload.py b/Lib/packaging/tests/test_command_upload.py
--- a/Lib/packaging/tests/test_command_upload.py
+++ b/Lib/packaging/tests/test_command_upload.py
@@ -1,6 +1,5 @@
 """Tests for packaging.command.upload."""
 import os
-import sys
 
 from packaging.command.upload import upload
 from packaging.dist import Distribution
@@ -103,22 +102,23 @@
         command, pyversion, filename = 'xxx', '3.3', path
         dist_files = [(command, pyversion, filename)]
 
-        # lets run it
-        pkg_dir, dist = self.create_dist(dist_files=dist_files, author='d?d?')
+        # let's run it
+        dist = self.create_dist(dist_files=dist_files, author='d?d?')[1]
         cmd = upload(dist)
         cmd.ensure_finalized()
         cmd.repository = self.pypi.full_address
         cmd.run()
 
-        # what did we send ?
+        # what did we send?
         handler, request_data = self.pypi.requests[-1]
         headers = handler.headers
-        #self.assertIn('d?d?', str(request_data))
+        self.assertIn('d?d?'.encode('utf-8'), request_data)
         self.assertIn(b'xxx', request_data)
 
         self.assertEqual(int(headers['content-length']), len(request_data))
         self.assertLess(int(headers['content-length']), 2500)
-        self.assertTrue(headers['content-type'].startswith('multipart/form-data'))
+        self.assertTrue(headers['content-type'].startswith(
+            'multipart/form-data'))
         self.assertEqual(handler.command, 'POST')
         self.assertNotIn('\n', headers['authorization'])
 
@@ -132,20 +132,16 @@
         self.write_file(os.path.join(docs_path, "index.html"), "yellow")
         self.write_file(self.rc, PYPIRC)
 
-        # lets run it
-        pkg_dir, dist = self.create_dist(dist_files=dist_files, author='d?d?')
+        # let's run it
+        dist = self.create_dist(dist_files=dist_files, author='d?d?')[1]
 
         cmd = upload(dist)
         cmd.get_finalized_command("build").run()
         cmd.upload_docs = True
         cmd.ensure_finalized()
         cmd.repository = self.pypi.full_address
-        prev_dir = os.getcwd()
-        try:
-            os.chdir(self.tmp_dir)
-            cmd.run()
-        finally:
-            os.chdir(prev_dir)
+        os.chdir(self.tmp_dir)
+        cmd.run()
 
         handler, request_data = self.pypi.requests[-1]
         action, name, content = request_data.split(
diff --git a/Lib/packaging/tests/test_command_upload_docs.py b/Lib/packaging/tests/test_command_upload_docs.py
--- a/Lib/packaging/tests/test_command_upload_docs.py
+++ b/Lib/packaging/tests/test_command_upload_docs.py
@@ -1,6 +1,5 @@
 """Tests for packaging.command.upload_docs."""
 import os
-import sys
 import shutil
 import zipfile
 try:
@@ -52,34 +51,27 @@
 
     def test_default_uploaddir(self):
         sandbox = self.mkdtemp()
-        previous = os.getcwd()
         os.chdir(sandbox)
-        try:
-            os.mkdir("build")
-            self.prepare_sample_dir("build")
-            self.cmd.ensure_finalized()
-            self.assertEqual(self.cmd.upload_dir, os.path.join("build", "docs"))
-        finally:
-            os.chdir(previous)
+        os.mkdir("build")
+        self.prepare_sample_dir("build")
+        self.cmd.ensure_finalized()
+        self.assertEqual(self.cmd.upload_dir, os.path.join("build", "docs"))
 
     def test_default_uploaddir_looks_for_doc_also(self):
         sandbox = self.mkdtemp()
-        previous = os.getcwd()
         os.chdir(sandbox)
-        try:
-            os.mkdir("build")
-            self.prepare_sample_dir("build")
-            os.rename(os.path.join("build", "docs"), os.path.join("build", "doc"))
-            self.cmd.ensure_finalized()
-            self.assertEqual(self.cmd.upload_dir, os.path.join("build", "doc"))
-        finally:
-            os.chdir(previous)
+        os.mkdir("build")
+        self.prepare_sample_dir("build")
+        os.rename(os.path.join("build", "docs"), os.path.join("build", "doc"))
+        self.cmd.ensure_finalized()
+        self.assertEqual(self.cmd.upload_dir, os.path.join("build", "doc"))
 
     def prepare_sample_dir(self, sample_dir=None):
         if sample_dir is None:
             sample_dir = self.mkdtemp()
         os.mkdir(os.path.join(sample_dir, "docs"))
-        self.write_file(os.path.join(sample_dir, "docs", "index.html"), "Ce mortel ennui")
+        self.write_file(os.path.join(sample_dir, "docs", "index.html"),
+                        "Ce mortel ennui")
         self.write_file(os.path.join(sample_dir, "index.html"), "Oh la la")
         return sample_dir
 
@@ -108,9 +100,8 @@
         self.assertTrue(handler.headers['content-type']
             .startswith('multipart/form-data;'))
 
-        action, name, version, content =\
-            request_data.split("----------------GHSKFJDLGDS7543FJKLFHRE75642756743254".encode())[1:5]
-
+        action, name, version, content = request_data.split(
+            b'----------------GHSKFJDLGDS7543FJKLFHRE75642756743254')[1:5]
 
         # check that we picked the right chunks
         self.assertIn(b'name=":action"', action)
@@ -126,27 +117,25 @@
 
     @unittest.skipIf(_ssl is None, 'Needs SSL support')
     def test_https_connection(self):
-        https_called = False
-
-        orig_https = upload_docs_mod.http.client.HTTPSConnection
+        self.https_called = False
+        self.addCleanup(
+            setattr, upload_docs_mod.http.client, 'HTTPSConnection',
+            upload_docs_mod.http.client.HTTPSConnection)
 
         def https_conn_wrapper(*args):
-            nonlocal https_called
-            https_called = True
+            self.https_called = True
             # the testing server is http
             return upload_docs_mod.http.client.HTTPConnection(*args)
 
         upload_docs_mod.http.client.HTTPSConnection = https_conn_wrapper
-        try:
-            self.prepare_command()
-            self.cmd.run()
-            self.assertFalse(https_called)
 
-            self.cmd.repository = self.cmd.repository.replace("http", "https")
-            self.cmd.run()
-            self.assertTrue(https_called)
-        finally:
-            upload_docs_mod.http.client.HTTPSConnection = orig_https
+        self.prepare_command()
+        self.cmd.run()
+        self.assertFalse(self.https_called)
+
+        self.cmd.repository = self.cmd.repository.replace("http", "https")
+        self.cmd.run()
+        self.assertTrue(self.https_called)
 
     def test_handling_response(self):
         self.pypi.default_response_status = '403 Forbidden'
@@ -155,7 +144,8 @@
         self.assertIn('Upload failed (403): Forbidden', self.get_logs()[-1])
 
         self.pypi.default_response_status = '301 Moved Permanently'
-        self.pypi.default_response_headers.append(("Location", "brand_new_location"))
+        self.pypi.default_response_headers.append(
+            ("Location", "brand_new_location"))
         self.cmd.run()
         self.assertIn('brand_new_location', self.get_logs()[-1])
 
@@ -185,6 +175,7 @@
         self.assertTrue(record, "should report the response")
         self.assertIn(self.pypi.default_response_data, record)
 
+
 def test_suite():
     return unittest.makeSuite(UploadDocsTestCase)
 
diff --git a/Lib/packaging/tests/test_create.py b/Lib/packaging/tests/test_create.py
--- a/Lib/packaging/tests/test_create.py
+++ b/Lib/packaging/tests/test_create.py
@@ -1,8 +1,8 @@
 """Tests for packaging.create."""
-import io
 import os
 import sys
 import sysconfig
+from io import StringIO
 from textwrap import dedent
 from packaging.create import MainProgram, ask_yn, ask, main
 
@@ -20,8 +20,8 @@
         super(CreateTestCase, self).setUp()
         self._stdin = sys.stdin  # TODO use Inputs
         self._stdout = sys.stdout
-        sys.stdin = io.StringIO()
-        sys.stdout = io.StringIO()
+        sys.stdin = StringIO()
+        sys.stdout = StringIO()
         self._cwd = os.getcwd()
         self.wdir = self.mkdtemp()
         os.chdir(self.wdir)
@@ -135,7 +135,8 @@
         sys.stdin.seek(0)
         main()
 
-        with open(os.path.join(self.wdir, 'setup.cfg'), encoding='utf-8') as fp:
+        path = os.path.join(self.wdir, 'setup.cfg')
+        with open(path, encoding='utf-8') as fp:
             contents = fp.read()
 
         self.assertEqual(contents, dedent("""\
@@ -210,7 +211,9 @@
         sys.stdin.seek(0)
         # FIXME Out of memory error.
         main()
-        with open(os.path.join(self.wdir, 'setup.cfg'), encoding='utf-8') as fp:
+
+        path = os.path.join(self.wdir, 'setup.cfg')
+        with open(path, encoding='utf-8') as fp:
             contents = fp.read()
 
         self.assertEqual(contents, dedent("""\
diff --git a/Lib/packaging/tests/test_database.py b/Lib/packaging/tests/test_database.py
--- a/Lib/packaging/tests/test_database.py
+++ b/Lib/packaging/tests/test_database.py
@@ -49,8 +49,8 @@
         # distributions
         tmpdir = tempfile.mkdtemp()
         self.addCleanup(shutil.rmtree, tmpdir)
-        self.fake_dists_path = os.path.join(tmpdir, 'fake_dists')
-        self.fake_dists_path = os.path.realpath(self.fake_dists_path)
+        self.fake_dists_path = os.path.realpath(
+            os.path.join(tmpdir, 'fake_dists'))
         fake_dists_src = os.path.abspath(
             os.path.join(os.path.dirname(__file__), 'fake_dists'))
         shutil.copytree(fake_dists_src, self.fake_dists_path)
@@ -58,6 +58,7 @@
         # back (to avoid getting a read-only copy of a read-only file).  we
         # could pass a custom copy_function to change the mode of files, but
         # shutil gives no control over the mode of directories :(
+        # see http://bugs.python.org/issue1666318
         for root, dirs, files in os.walk(self.fake_dists_path):
             os.chmod(root, 0o755)
             for f in files:
diff --git a/Lib/packaging/tests/test_depgraph.py b/Lib/packaging/tests/test_depgraph.py
--- a/Lib/packaging/tests/test_depgraph.py
+++ b/Lib/packaging/tests/test_depgraph.py
@@ -1,10 +1,11 @@
 """Tests for packaging.depgraph """
-import io
 import os
 import re
 import sys
-import packaging.database
+from io import StringIO
+
 from packaging import depgraph
+from packaging.database import get_distribution, enable_cache, disable_cache
 
 from packaging.tests import unittest, support
 from packaging.tests.support import requires_zlib
@@ -30,13 +31,13 @@
         path = os.path.abspath(path)
         sys.path.insert(0, path)
         self.addCleanup(sys.path.remove, path)
-        self.addCleanup(packaging.database.enable_cache)
-        packaging.database.disable_cache()
+        self.addCleanup(enable_cache)
+        disable_cache()
 
     def test_generate_graph(self):
         dists = []
         for name in self.DISTROS_DIST:
-            dist = packaging.database.get_distribution(name)
+            dist = get_distribution(name)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
@@ -61,7 +62,7 @@
     def test_generate_graph_egg(self):
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG:
-            dist = packaging.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
@@ -104,7 +105,7 @@
     def test_dependent_dists(self):
         dists = []
         for name in self.DISTROS_DIST:
-            dist = packaging.database.get_distribution(name)
+            dist = get_distribution(name)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
@@ -123,7 +124,7 @@
     def test_dependent_dists_egg(self):
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG:
-            dist = packaging.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
@@ -158,12 +159,12 @@
 
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG:
-            dist = packaging.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
         graph = depgraph.generate_graph(dists)
-        buf = io.StringIO()
+        buf = StringIO()
         depgraph.graph_to_dot(graph, buf)
         buf.seek(0)
         matches = []
@@ -189,12 +190,12 @@
 
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG:
-            dist = packaging.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
         graph = depgraph.generate_graph(dists)
-        buf = io.StringIO()
+        buf = StringIO()
         depgraph.graph_to_dot(graph, buf, skip_disconnected=False)
         buf.seek(0)
         lines = buf.readlines()
@@ -250,12 +251,12 @@
 
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG + self.BAD_EGGS:
-            dist = packaging.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
         graph = depgraph.generate_graph(dists)
-        buf = io.StringIO()
+        buf = StringIO()
         depgraph.graph_to_dot(graph, buf)
         buf.seek(0)
         matches = []
@@ -273,7 +274,7 @@
     def test_repr(self):
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG + self.BAD_EGGS:
-            dist = packaging.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
@@ -282,7 +283,7 @@
 
     @requires_zlib
     def test_main(self):
-        tempout = io.StringIO()
+        tempout = StringIO()
         old = sys.stdout
         sys.stdout = tempout
         oldargv = sys.argv[:]
diff --git a/Lib/packaging/tests/test_dist.py b/Lib/packaging/tests/test_dist.py
--- a/Lib/packaging/tests/test_dist.py
+++ b/Lib/packaging/tests/test_dist.py
@@ -3,13 +3,14 @@
 import sys
 import logging
 import textwrap
+
 import packaging.dist
 
 from packaging.dist import Distribution
 from packaging.command import set_command
 from packaging.command.cmd import Command
 from packaging.errors import PackagingModuleError, PackagingOptionError
-from packaging.tests import TESTFN, captured_stdout
+from packaging.tests import captured_stdout
 from packaging.tests import support, unittest
 from packaging.tests.support import create_distribution
 from test.support import unload
@@ -48,12 +49,13 @@
 
     @unittest.skip('needs to be updated')
     def test_debug_mode(self):
-        self.addCleanup(os.unlink, TESTFN)
-        with open(TESTFN, "w") as f:
+        tmpdir = self.mkdtemp()
+        setupcfg = os.path.join(tmpdir, 'setup.cfg')
+        with open(setupcfg, "w") as f:
             f.write("[global]\n")
             f.write("command_packages = foo.bar, splat")
 
-        files = [TESTFN]
+        files = [setupcfg]
         sys.argv.append("build")
         __, stdout = captured_stdout(create_distribution, files)
         self.assertEqual(stdout, '')
diff --git a/Lib/packaging/tests/test_install.py b/Lib/packaging/tests/test_install.py
--- a/Lib/packaging/tests/test_install.py
+++ b/Lib/packaging/tests/test_install.py
@@ -1,8 +1,8 @@
 """Tests for the packaging.install module."""
 import os
 import logging
+from tempfile import mkstemp
 from sysconfig import is_python_build
-from tempfile import mkstemp
 
 from packaging import install
 from packaging.pypi.xmlrpc import Client
diff --git a/Lib/packaging/tests/test_pypi_server.py b/Lib/packaging/tests/test_pypi_server.py
--- a/Lib/packaging/tests/test_pypi_server.py
+++ b/Lib/packaging/tests/test_pypi_server.py
@@ -1,13 +1,12 @@
 """Tests for packaging.command.bdist."""
-import sys
-
 import urllib.request
 import urllib.parse
 import urllib.error
 
 try:
     import threading
-    from packaging.tests.pypi_server import PyPIServer, PYPI_DEFAULT_STATIC_PATH
+    from packaging.tests.pypi_server import (
+        PyPIServer, PYPI_DEFAULT_STATIC_PATH)
 except ImportError:
     threading = None
     PyPIServer = None
@@ -32,18 +31,19 @@
 
             headers = {"X-test-header": "Mister Iceberg"}
 
-            request = urllib.request.Request(server.full_address, data, headers)
+            request = urllib.request.Request(
+                server.full_address, data, headers)
             urllib.request.urlopen(request)
             self.assertEqual(len(server.requests), 1)
             handler, request_data = server.requests[-1]
             self.assertIn(data, request_data)
             self.assertIn("x-test-header", handler.headers)
-            self.assertEqual(handler.headers["x-test-header"], "Mister Iceberg")
+            self.assertEqual(handler.headers["x-test-header"],
+                             "Mister Iceberg")
 
         finally:
             server.stop()
 
-
     def test_serve_static_content(self):
         # PYPI Mocked server can serve static content from disk.
 
@@ -74,7 +74,8 @@
             self.assertTrue(uses_local_files_for(server, "/simple/index.html"))
 
             # and another one in another root path
-            self.assertTrue(uses_local_files_for(server, "/external/index.html"))
+            self.assertTrue(uses_local_files_for(server,
+                                                 "/external/index.html"))
 
         finally:
             server.stop()
diff --git a/Lib/packaging/tests/test_run.py b/Lib/packaging/tests/test_run.py
--- a/Lib/packaging/tests/test_run.py
+++ b/Lib/packaging/tests/test_run.py
@@ -2,11 +2,10 @@
 
 import os
 import sys
-import shutil
 from io import StringIO
 
 from packaging import install
-from packaging.tests import unittest, support, TESTFN
+from packaging.tests import unittest, support
 from packaging.run import main
 
 from test.script_helper import assert_python_ok
@@ -35,28 +34,14 @@
     def setUp(self):
         super(RunTestCase, self).setUp()
         self.old_stdout = sys.stdout
-        self.cleanup_testfn()
         self.old_argv = sys.argv, sys.argv[:]
 
     def tearDown(self):
         sys.stdout = self.old_stdout
-        self.cleanup_testfn()
         sys.argv = self.old_argv[0]
         sys.argv[:] = self.old_argv[1]
         super(RunTestCase, self).tearDown()
 
-    def cleanup_testfn(self):
-        path = TESTFN
-        if os.path.isfile(path):
-            os.remove(path)
-        elif os.path.isdir(path):
-            shutil.rmtree(path)
-
-    def write_setup(self, text, path=TESTFN):
-        with open(path, "w") as fp:
-            fp.write(text)
-        return path
-
     # TODO restore the tests removed six months ago and port them to pysetup
 
     def test_install(self):
diff --git a/Lib/packaging/tests/test_util.py b/Lib/packaging/tests/test_util.py
--- a/Lib/packaging/tests/test_util.py
+++ b/Lib/packaging/tests/test_util.py
@@ -15,7 +15,7 @@
 from packaging import util
 from packaging.dist import Distribution
 from packaging.util import (
-    convert_path, change_root, split_quoted, strtobool,
+    convert_path, change_root, split_quoted, strtobool, run_2to3,
     get_compiler_versions, _MAC_OS_X_LD_VERSION, byte_compile, find_packages,
     spawn, get_pypirc_path, generate_pypirc, read_pypirc, resolve_name, iglob,
     RICH_GLOB, egginfo_to_distinfo, is_setuptools, is_distutils, is_packaging,
@@ -374,7 +374,7 @@
 
         res = find_packages([root], ['pkg1.pkg2'])
         self.assertEqual(set(res), set(['pkg1', 'pkg5', 'pkg1.pkg3',
-                                         'pkg1.pkg3.pkg6']))
+                                        'pkg1.pkg3.pkg6']))
 
     def test_resolve_name(self):
         self.assertIs(str, resolve_name('builtins.str'))
@@ -416,7 +416,6 @@
         file_handle.write(content)
         file_handle.flush()
         file_handle.seek(0)
-        from packaging.util import run_2to3
         run_2to3([file_name])
         new_content = "".join(file_handle.read())
         file_handle.close()
@@ -432,7 +431,6 @@
         file_handle.write(content)
         file_handle.flush()
         file_handle.seek(0)
-        from packaging.util import run_2to3
         run_2to3([file_name], doctests_only=True)
         new_content = "".join(file_handle.readlines())
         file_handle.close()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 16:10:52 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 16:10:52 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Branch_merge?=
Message-ID: <E1R5eYe-0005Ez-QX@dinsdale.python.org>

http://hg.python.org/cpython/rev/d92b17b1dce6
changeset:   72405:d92b17b1dce6
parent:      72404:a400c1c087dd
parent:      72402:f18e9db83483
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Sep 18 20:24:27 2011 +0200
summary:
  Branch merge

files:
  Lib/packaging/command/build_ext.py               |  17 +-
  Lib/packaging/command/build_py.py                |   2 +-
  Lib/packaging/command/install_dist.py            |  59 +++------
  Lib/packaging/command/install_distinfo.py        |   7 +-
  Lib/packaging/command/install_lib.py             |   2 +-
  Lib/packaging/compiler/bcppcompiler.py           |   1 -
  Lib/packaging/compiler/ccompiler.py              |   1 -
  Lib/packaging/dist.py                            |   4 +-
  Lib/packaging/tests/test_command_build_ext.py    |   8 +-
  Lib/packaging/tests/test_command_build_py.py     |   2 -
  Lib/packaging/tests/test_command_install_dist.py |  12 +-
  Lib/packaging/tests/test_command_install_lib.py  |  11 +-
  Lib/packaging/tests/test_mixin2to3.py            |   3 -
  Lib/packaging/tests/test_util.py                 |   4 -
  Lib/packaging/util.py                            |   6 +-
  Lib/pkgutil.py                                   |   4 +-
  16 files changed, 47 insertions(+), 96 deletions(-)


diff --git a/Lib/packaging/command/build_ext.py b/Lib/packaging/command/build_ext.py
--- a/Lib/packaging/command/build_ext.py
+++ b/Lib/packaging/command/build_ext.py
@@ -3,6 +3,7 @@
 import os
 import re
 import sys
+import site
 import logging
 import sysconfig
 
@@ -15,9 +16,6 @@
 from packaging.compiler.extension import Extension
 from packaging import logger
 
-import site
-HAS_USER_SITE = True
-
 if os.name == 'nt':
     from packaging.compiler.msvccompiler import get_build_version
     MSVC_VERSION = int(get_build_version())
@@ -62,6 +60,8 @@
         ('inplace', 'i',
          "ignore build-lib and put compiled extensions into the source " +
          "directory alongside your pure Python modules"),
+        ('user', None,
+         "add user include, library and rpath"),
         ('include-dirs=', 'I',
          "list of directories to search for header files" + sep_by),
         ('define=', 'D',
@@ -88,12 +88,8 @@
          "path to the SWIG executable"),
         ]
 
-    boolean_options = ['inplace', 'debug', 'force']
+    boolean_options = ['inplace', 'debug', 'force', 'user']
 
-    if HAS_USER_SITE:
-        user_options.append(('user', None,
-                             "add user include, library and rpath"))
-        boolean_options.append('user')
 
     help_options = [
         ('help-compiler', None,
@@ -120,8 +116,7 @@
         self.compiler = None
         self.swig = None
         self.swig_opts = None
-        if HAS_USER_SITE:
-            self.user = None
+        self.user = None
 
     def finalize_options(self):
         self.set_undefined_options('build',
@@ -270,7 +265,7 @@
             self.swig_opts = self.swig_opts.split(' ')
 
         # Finally add the user include and library directories if requested
-        if HAS_USER_SITE and self.user:
+        if self.user:
             user_include = os.path.join(site.USER_BASE, "include")
             user_lib = os.path.join(site.USER_BASE, "lib")
             if os.path.isdir(user_include):
diff --git a/Lib/packaging/command/build_py.py b/Lib/packaging/command/build_py.py
--- a/Lib/packaging/command/build_py.py
+++ b/Lib/packaging/command/build_py.py
@@ -388,7 +388,7 @@
                 self.build_module(module, module_file, package)
 
     def byte_compile(self, files):
-        if hasattr(sys, 'dont_write_bytecode') and sys.dont_write_bytecode:
+        if sys.dont_write_bytecode:
             logger.warning('%s: byte-compiling is disabled, skipping.',
                            self.get_command_name())
             return
diff --git a/Lib/packaging/command/install_dist.py b/Lib/packaging/command/install_dist.py
--- a/Lib/packaging/command/install_dist.py
+++ b/Lib/packaging/command/install_dist.py
@@ -14,9 +14,6 @@
 from packaging.errors import PackagingOptionError
 
 
-HAS_USER_SITE = True
-
-
 class install_dist(Command):
 
     description = "install everything from build directory"
@@ -27,6 +24,9 @@
          "installation prefix"),
         ('exec-prefix=', None,
          "(Unix only) prefix for platform-specific files"),
+        ('user', None,
+         "install in user site-packages directory [%s]" %
+         get_path('purelib', '%s_user' % os.name)),
         ('home=', None,
          "(Unix only) home directory to install under"),
 
@@ -97,15 +97,7 @@
         ]
 
     boolean_options = ['compile', 'force', 'skip-build', 'no-distinfo',
-                       'requested', 'no-record']
-
-    if HAS_USER_SITE:
-        user_options.append(
-            ('user', None,
-             "install in user site-packages directory [%s]" %
-             get_path('purelib', '%s_user' % os.name)))
-
-        boolean_options.append('user')
+                       'requested', 'no-record', 'user']
 
     negative_opt = {'no-compile': 'compile', 'no-requested': 'requested'}
 
@@ -115,8 +107,7 @@
         self.prefix = None
         self.exec_prefix = None
         self.home = None
-        if HAS_USER_SITE:
-            self.user = False
+        self.user = False
 
         # These select only the installation base; it's up to the user to
         # specify the installation scheme (currently, that means supplying
@@ -135,9 +126,8 @@
         self.install_lib = None         # set to either purelib or platlib
         self.install_scripts = None
         self.install_data = None
-        if HAS_USER_SITE:
-            self.install_userbase = get_config_var('userbase')
-            self.install_usersite = get_path('purelib', '%s_user' % os.name)
+        self.install_userbase = get_config_var('userbase')
+        self.install_usersite = get_path('purelib', '%s_user' % os.name)
 
         self.compile = None
         self.optimize = None
@@ -219,9 +209,8 @@
             raise PackagingOptionError(
                 "must supply either home or prefix/exec-prefix -- not both")
 
-        if HAS_USER_SITE and self.user and (
-                self.prefix or self.exec_prefix or self.home or
-                self.install_base or self.install_platbase):
+        if self.user and (self.prefix or self.exec_prefix or self.home or
+                          self.install_base or self.install_platbase):
             raise PackagingOptionError(
                 "can't combine user with prefix/exec_prefix/home or "
                 "install_base/install_platbase")
@@ -274,11 +263,9 @@
             'exec_prefix': exec_prefix,
             'srcdir': srcdir,
             'projectbase': projectbase,
-            }
-
-        if HAS_USER_SITE:
-            self.config_vars['userbase'] = self.install_userbase
-            self.config_vars['usersite'] = self.install_usersite
+            'userbase': self.install_userbase,
+            'usersite': self.install_usersite,
+        }
 
         self.expand_basedirs()
 
@@ -295,9 +282,9 @@
 
         self.dump_dirs("post-expand_dirs()")
 
-        # Create directories in the home dir:
-        if HAS_USER_SITE and self.user:
-            self.create_home_path()
+        # Create directories under USERBASE
+        if self.user:
+            self.create_user_dirs()
 
         # Pick the actual directory to install all modules to: either
         # install_purelib or install_platlib, depending on whether this
@@ -311,10 +298,8 @@
 
         # Convert directories from Unix /-separated syntax to the local
         # convention.
-        self.convert_paths('lib', 'purelib', 'platlib',
-                           'scripts', 'data', 'headers')
-        if HAS_USER_SITE:
-            self.convert_paths('userbase', 'usersite')
+        self.convert_paths('lib', 'purelib', 'platlib', 'scripts',
+                           'data', 'headers', 'userbase', 'usersite')
 
         # Well, we're not actually fully completely finalized yet: we still
         # have to deal with 'extra_path', which is the hack for allowing
@@ -355,7 +340,7 @@
                     "installation scheme is incomplete")
             return
 
-        if HAS_USER_SITE and self.user:
+        if self.user:
             if self.install_userbase is None:
                 raise PackagingPlatformError(
                     "user base directory is not specified")
@@ -383,7 +368,7 @@
 
     def finalize_other(self):
         """Finalize options for non-posix platforms"""
-        if HAS_USER_SITE and self.user:
+        if self.user:
             if self.install_userbase is None:
                 raise PackagingPlatformError(
                     "user base directory is not specified")
@@ -494,10 +479,8 @@
             attr = "install_" + name
             setattr(self, attr, change_root(self.root, getattr(self, attr)))
 
-    def create_home_path(self):
-        """Create directories under ~."""
-        if HAS_USER_SITE and not self.user:
-            return
+    def create_user_dirs(self):
+        """Create directories under USERBASE as needed."""
         home = convert_path(os.path.expanduser("~"))
         for name, path in self.config_vars.items():
             if path.startswith(home) and not os.path.isdir(path):
diff --git a/Lib/packaging/command/install_distinfo.py b/Lib/packaging/command/install_distinfo.py
--- a/Lib/packaging/command/install_distinfo.py
+++ b/Lib/packaging/command/install_distinfo.py
@@ -2,14 +2,13 @@
 
 # Forked from the former install_egg_info command by Josip Djolonga
 
+import os
 import csv
-import os
-import re
 import hashlib
+from shutil import rmtree
 
+from packaging import logger
 from packaging.command.cmd import Command
-from packaging import logger
-from shutil import rmtree
 
 
 class install_distinfo(Command):
diff --git a/Lib/packaging/command/install_lib.py b/Lib/packaging/command/install_lib.py
--- a/Lib/packaging/command/install_lib.py
+++ b/Lib/packaging/command/install_lib.py
@@ -114,7 +114,7 @@
         return outfiles
 
     def byte_compile(self, files):
-        if getattr(sys, 'dont_write_bytecode'):
+        if sys.dont_write_bytecode:
             # XXX do we want this?  because a Python runs without bytecode
             # doesn't mean that the *dists should not contain bytecode
             #--or does it?
diff --git a/Lib/packaging/compiler/bcppcompiler.py b/Lib/packaging/compiler/bcppcompiler.py
--- a/Lib/packaging/compiler/bcppcompiler.py
+++ b/Lib/packaging/compiler/bcppcompiler.py
@@ -352,5 +352,4 @@
             try:
                 self.spawn(pp_args)
             except PackagingExecError as msg:
-                print(msg)
                 raise CompileError(msg)
diff --git a/Lib/packaging/compiler/ccompiler.py b/Lib/packaging/compiler/ccompiler.py
--- a/Lib/packaging/compiler/ccompiler.py
+++ b/Lib/packaging/compiler/ccompiler.py
@@ -5,7 +5,6 @@
 """
 
 import os
-import sys
 from shutil import move
 from packaging import logger
 from packaging.util import split_quoted, execute, newer_group, spawn
diff --git a/Lib/packaging/dist.py b/Lib/packaging/dist.py
--- a/Lib/packaging/dist.py
+++ b/Lib/packaging/dist.py
@@ -537,7 +537,7 @@
     def _get_command_groups(self):
         """Helper function to retrieve all the command class names divided
         into standard commands (listed in
-        packaging2.command.STANDARD_COMMANDS) and extra commands (given in
+        packaging.command.STANDARD_COMMANDS) and extra commands (given in
         self.cmdclass and not standard commands).
         """
         extra_commands = [cmd for cmd in self.cmdclass
@@ -547,7 +547,7 @@
     def print_commands(self):
         """Print out a help message listing all available commands with a
         description of each.  The list is divided into standard commands
-        (listed in packaging2.command.STANDARD_COMMANDS) and extra commands
+        (listed in packaging.command.STANDARD_COMMANDS) and extra commands
         (given in self.cmdclass and not standard commands).  The
         descriptions come from the command class attribute
         'description'.
diff --git a/Lib/packaging/tests/test_command_build_ext.py b/Lib/packaging/tests/test_command_build_ext.py
--- a/Lib/packaging/tests/test_command_build_ext.py
+++ b/Lib/packaging/tests/test_command_build_ext.py
@@ -19,18 +19,13 @@
                        support.LoggingCatcher,
                        unittest.TestCase):
     def setUp(self):
-        # Create a simple test environment
-        # Note that we're making changes to sys.path
         super(BuildExtTestCase, self).setUp()
         self.tmp_dir = self.mkdtemp()
         self.old_user_base = site.USER_BASE
         site.USER_BASE = self.mkdtemp()
 
     def tearDown(self):
-        # Get everything back to normal
-        if sys.version > "2.6":
-            site.USER_BASE = self.old_user_base
-
+        site.USER_BASE = self.old_user_base
         super(BuildExtTestCase, self).tearDown()
 
     def test_build_ext(self):
@@ -94,7 +89,6 @@
         # make sure we get some library dirs under solaris
         self.assertGreater(len(cmd.library_dirs), 0)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_user_site(self):
         dist = Distribution({'name': 'xx'})
         cmd = build_ext(dist)
diff --git a/Lib/packaging/tests/test_command_build_py.py b/Lib/packaging/tests/test_command_build_py.py
--- a/Lib/packaging/tests/test_command_build_py.py
+++ b/Lib/packaging/tests/test_command_build_py.py
@@ -99,8 +99,6 @@
             os.chdir(cwd)
             sys.stdout = old_stdout
 
-    @unittest.skipUnless(hasattr(sys, 'dont_write_bytecode'),
-                         'sys.dont_write_bytecode not supported')
     def test_dont_write_bytecode(self):
         # makes sure byte_compile is not used
         pkg_dir, dist = self.create_dist()
diff --git a/Lib/packaging/tests/test_command_install_dist.py b/Lib/packaging/tests/test_command_install_dist.py
--- a/Lib/packaging/tests/test_command_install_dist.py
+++ b/Lib/packaging/tests/test_command_install_dist.py
@@ -72,7 +72,6 @@
         check_path(cmd.install_scripts, os.path.join(destination, "bin"))
         check_path(cmd.install_data, destination)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_user_site(self):
         # test install with --user
         # preparing the environment for the test
@@ -173,12 +172,11 @@
         cmd.home = 'home'
         self.assertRaises(PackagingOptionError, cmd.finalize_options)
 
-        if sys.version >= '2.6':
-            # can't combine user with with prefix/exec_prefix/home or
-            # install_(plat)base
-            cmd.prefix = None
-            cmd.user = 'user'
-            self.assertRaises(PackagingOptionError, cmd.finalize_options)
+        # can't combine user with with prefix/exec_prefix/home or
+        # install_(plat)base
+        cmd.prefix = None
+        cmd.user = 'user'
+        self.assertRaises(PackagingOptionError, cmd.finalize_options)
 
     def test_old_record(self):
         # test pre-PEP 376 --record option (outside dist-info dir)
diff --git a/Lib/packaging/tests/test_command_install_lib.py b/Lib/packaging/tests/test_command_install_lib.py
--- a/Lib/packaging/tests/test_command_install_lib.py
+++ b/Lib/packaging/tests/test_command_install_lib.py
@@ -7,13 +7,6 @@
 from packaging.compiler.extension import Extension
 from packaging.errors import PackagingOptionError
 
-try:
-    no_bytecode = sys.dont_write_bytecode
-    bytecode_support = True
-except AttributeError:
-    no_bytecode = False
-    bytecode_support = False
-
 
 class InstallLibTestCase(support.TempdirManager,
                          support.LoggingCatcher,
@@ -40,7 +33,7 @@
         cmd.finalize_options()
         self.assertEqual(cmd.optimize, 2)
 
-    @unittest.skipIf(no_bytecode, 'byte-compile disabled')
+    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
     def test_byte_compile(self):
         pkg_dir, dist = self.create_dist()
         cmd = install_lib(dist)
@@ -89,8 +82,6 @@
         # get_input should return 2 elements
         self.assertEqual(len(cmd.get_inputs()), 2)
 
-    @unittest.skipUnless(bytecode_support,
-                         'sys.dont_write_bytecode not supported')
     def test_dont_write_bytecode(self):
         # makes sure byte_compile is not used
         pkg_dir, dist = self.create_dist()
diff --git a/Lib/packaging/tests/test_mixin2to3.py b/Lib/packaging/tests/test_mixin2to3.py
--- a/Lib/packaging/tests/test_mixin2to3.py
+++ b/Lib/packaging/tests/test_mixin2to3.py
@@ -9,7 +9,6 @@
                         support.LoggingCatcher,
                         unittest.TestCase):
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_convert_code_only(self):
         # used to check if code gets converted properly.
         code = "print 'test'"
@@ -26,7 +25,6 @@
 
         self.assertEqual(expected, converted)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_doctests_only(self):
         # used to check if doctests gets converted properly.
         doctest = textwrap.dedent('''\
@@ -57,7 +55,6 @@
 
         self.assertEqual(expected, converted)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_additional_fixers(self):
         # used to check if use_2to3_fixers works
         code = 'type(x) is not T'
diff --git a/Lib/packaging/tests/test_util.py b/Lib/packaging/tests/test_util.py
--- a/Lib/packaging/tests/test_util.py
+++ b/Lib/packaging/tests/test_util.py
@@ -319,8 +319,6 @@
         res = get_compiler_versions()
         self.assertEqual(res[2], None)
 
-    @unittest.skipUnless(hasattr(sys, 'dont_write_bytecode'),
-                         'sys.dont_write_bytecode not supported')
     def test_dont_write_bytecode(self):
         # makes sure byte_compile raise a PackagingError
         # if sys.dont_write_bytecode is True
@@ -407,7 +405,6 @@
         finally:
             sys.path.remove(tmp_dir)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_run_2to3_on_code(self):
         content = "print 'test'"
         converted_content = "print('test')"
@@ -421,7 +418,6 @@
         file_handle.close()
         self.assertEqual(new_content, converted_content)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_run_2to3_on_doctests(self):
         # to check if text files containing doctests only get converted.
         content = ">>> print 'test'\ntest\n"
diff --git a/Lib/packaging/util.py b/Lib/packaging/util.py
--- a/Lib/packaging/util.py
+++ b/Lib/packaging/util.py
@@ -326,7 +326,7 @@
     """
     # nothing is done if sys.dont_write_bytecode is True
     # FIXME this should not raise an error
-    if hasattr(sys, 'dont_write_bytecode') and sys.dont_write_bytecode:
+    if sys.dont_write_bytecode:
         raise PackagingByteCompileError('byte-compiling is disabled.')
 
     # First, if the caller didn't force us into direct or indirect mode,
@@ -346,8 +346,10 @@
     # run it with the appropriate flags.
     if not direct:
         from tempfile import mkstemp
-        # XXX script_fd may leak, use something better than mkstemp
+        # XXX use something better than mkstemp
         script_fd, script_name = mkstemp(".py")
+        os.close(script_fd)
+        script_fd = None
         logger.info("writing byte-compilation script '%s'", script_name)
         if not dry_run:
             if script_fd is not None:
diff --git a/Lib/pkgutil.py b/Lib/pkgutil.py
--- a/Lib/pkgutil.py
+++ b/Lib/pkgutil.py
@@ -307,9 +307,9 @@
     def get_filename(self, fullname=None):
         fullname = self._fix_name(fullname)
         mod_type = self.etc[2]
-        if self.etc[2]==imp.PKG_DIRECTORY:
+        if mod_type==imp.PKG_DIRECTORY:
             return self._get_delegate().get_filename()
-        elif self.etc[2] in (imp.PY_SOURCE, imp.PY_COMPILED, imp.C_EXTENSION):
+        elif mod_type in (imp.PY_SOURCE, imp.PY_COMPILED, imp.C_EXTENSION):
             return self.filename
         return None
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 16:10:53 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 16:10:53 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_typo_and_wording?=
Message-ID: <E1R5eYf-0005F4-HK@dinsdale.python.org>

http://hg.python.org/cpython/rev/cd35efeb8526
changeset:   72406:cd35efeb8526
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Sep 18 20:36:19 2011 +0200
summary:
  Fix typo and wording

files:
  Lib/packaging/command/install_distinfo.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/packaging/command/install_distinfo.py b/Lib/packaging/command/install_distinfo.py
--- a/Lib/packaging/command/install_distinfo.py
+++ b/Lib/packaging/command/install_distinfo.py
@@ -27,7 +27,7 @@
         ('no-record', None,
          "do not generate a RECORD file"),
         ('no-resources', None,
-         "do not generate a RESSOURCES list installed file"),
+         "do not generate a RESOURCES file"),
     ]
 
     boolean_options = ['requested', 'no-record', 'no-resources']

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 16:10:54 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 16:10:54 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Replace_cmp_function_with_k?=
	=?utf8?q?ey_function?=
Message-ID: <E1R5eYg-0005F9-AB@dinsdale.python.org>

http://hg.python.org/cpython/rev/76ff72904de4
changeset:   72407:76ff72904de4
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Sep 18 21:03:24 2011 +0200
summary:
  Replace cmp function with key function

files:
  Lib/packaging/create.py |  17 ++---------------
  1 files changed, 2 insertions(+), 15 deletions(-)


diff --git a/Lib/packaging/create.py b/Lib/packaging/create.py
--- a/Lib/packaging/create.py
+++ b/Lib/packaging/create.py
@@ -28,7 +28,6 @@
 import tokenize
 from hashlib import md5
 from textwrap import dedent
-from functools import cmp_to_key
 from configparser import RawConfigParser
 # importing this with an underscore as it should be replaced by the
 # dict form or another structures for all purposes
@@ -370,21 +369,9 @@
                     dist.data_files = [('', dist.data_files)]
                 # add tokens in the destination paths
                 vars = {'distribution.name': data['name']}
-                path_tokens = list(sysconfig.get_paths(vars=vars).items())
-
-                # TODO replace this with a key function
-                def length_comparison(x, y):
-                    len_x = len(x[1])
-                    len_y = len(y[1])
-                    if len_x == len_y:
-                        return 0
-                    elif len_x < len_y:
-                        return -1
-                    else:
-                        return 1
-
+                path_tokens = sysconfig.get_paths(vars=vars).items()
                 # sort tokens to use the longest one first
-                path_tokens.sort(key=cmp_to_key(length_comparison))
+                path_tokens = sorted(path_tokens, key=lambda x: len(x[1]))
                 for dest, srcs in (dist.data_files or []):
                     dest = os.path.join(sys.prefix, dest)
                     dest = dest.replace(os.path.sep, '/')

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 16:11:00 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 16:11:00 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_A_few_style_changes_origina?=
 =?utf8?q?lly_done_in_the_distutils2_repo?=
Message-ID: <E1R5eYm-0005Gk-4F@dinsdale.python.org>

http://hg.python.org/cpython/rev/cc5e3a65e452
changeset:   72408:cc5e3a65e452
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Sep 18 23:12:30 2011 +0200
summary:
  A few style changes originally done in the distutils2 repo

files:
  Lib/packaging/command/build_scripts.py |   4 +-
  Lib/packaging/create.py                |   5 ++-
  Lib/shutil.py                          |   8 +++---
  Lib/test/test_sysconfig.py             |  16 ++++++-------
  4 files changed, 16 insertions(+), 17 deletions(-)


diff --git a/Lib/packaging/command/build_scripts.py b/Lib/packaging/command/build_scripts.py
--- a/Lib/packaging/command/build_scripts.py
+++ b/Lib/packaging/command/build_scripts.py
@@ -3,7 +3,7 @@
 import os
 import re
 import sysconfig
-import tokenize
+from tokenize import detect_encoding
 
 from packaging.command.cmd import Command
 from packaging.util import convert_path, newer
@@ -83,7 +83,7 @@
                     raise
                 f = None
             else:
-                encoding, lines = tokenize.detect_encoding(f.readline)
+                encoding, lines = detect_encoding(f.readline)
                 f.seek(0)
                 first_line = f.readline()
                 if not first_line:
diff --git a/Lib/packaging/create.py b/Lib/packaging/create.py
--- a/Lib/packaging/create.py
+++ b/Lib/packaging/create.py
@@ -25,10 +25,11 @@
 import glob
 import shutil
 import sysconfig
-import tokenize
 from hashlib import md5
 from textwrap import dedent
+from tokenize import detect_encoding
 from configparser import RawConfigParser
+
 # importing this with an underscore as it should be replaced by the
 # dict form or another structures for all purposes
 from packaging._trove import all_classifiers as _CLASSIFIERS_LIST
@@ -111,7 +112,7 @@
     been loaded before, because we are monkey patching its setup function with
     a particular one"""
     with open("setup.py", "rb") as f:
-        encoding, lines = tokenize.detect_encoding(f.readline)
+        encoding, lines = detect_encoding(f.readline)
     with open("setup.py", encoding=encoding) as f:
         imp.load_module("setup", f, "setup.py", (".py", "r", imp.PY_SOURCE))
 
diff --git a/Lib/shutil.py b/Lib/shutil.py
--- a/Lib/shutil.py
+++ b/Lib/shutil.py
@@ -391,7 +391,7 @@
         compress_ext['bzip2'] = '.bz2'
 
     # flags for compression program, each element of list will be an argument
-    if compress is not None and compress not in compress_ext.keys():
+    if compress is not None and compress not in compress_ext:
         raise ValueError("bad value for 'compress', or compression format not "
                          "supported : {0}".format(compress))
 
@@ -497,7 +497,7 @@
     'gztar': (_make_tarball, [('compress', 'gzip')], "gzip'ed tar-file"),
     'bztar': (_make_tarball, [('compress', 'bzip2')], "bzip2'ed tar-file"),
     'tar':   (_make_tarball, [('compress', None)], "uncompressed tar file"),
-    'zip':   (_make_zipfile, [],"ZIP file")
+    'zip':   (_make_zipfile, [], "ZIP file")
     }
 
 if _BZ2_SUPPORTED:
@@ -530,7 +530,7 @@
     if not isinstance(extra_args, (tuple, list)):
         raise TypeError('extra_args needs to be a sequence')
     for element in extra_args:
-        if not isinstance(element, (tuple, list)) or len(element) !=2 :
+        if not isinstance(element, (tuple, list)) or len(element) !=2:
             raise TypeError('extra_args elements are : (arg_name, value)')
 
     _ARCHIVE_FORMATS[name] = (function, extra_args, description)
@@ -682,7 +682,7 @@
             if not name.endswith('/'):
                 # file
                 data = zip.read(info.filename)
-                f = open(target,'wb')
+                f = open(target, 'wb')
                 try:
                     f.write(data)
                 finally:
diff --git a/Lib/test/test_sysconfig.py b/Lib/test/test_sysconfig.py
--- a/Lib/test/test_sysconfig.py
+++ b/Lib/test/test_sysconfig.py
@@ -39,7 +39,7 @@
         self._config_vars = copy(sysconfig._CONFIG_VARS)
         self._added_envvars = []
         self._changed_envvars = []
-        for var in ('MACOSX_DEPLOYMENT_TARGET', 'Path'):
+        for var in ('MACOSX_DEPLOYMENT_TARGET', 'PATH'):
             if var in os.environ:
                 self._changed_envvars.append((var, os.environ[var]))
             else:
@@ -87,21 +87,19 @@
         scheme = get_paths()
         default_scheme = _get_default_scheme()
         wanted = _expand_vars(default_scheme, None)
-        wanted = list(wanted.items())
-        wanted.sort()
-        scheme = list(scheme.items())
-        scheme.sort()
+        wanted = sorted(wanted.items())
+        scheme = sorted(scheme.items())
         self.assertEqual(scheme, wanted)
 
     def test_get_path(self):
-        # xxx make real tests here
+        # XXX make real tests here
         for scheme in _SCHEMES:
             for name in _SCHEMES[scheme]:
                 res = get_path(name, scheme)
 
     def test_get_config_vars(self):
         cvars = get_config_vars()
-        self.assertTrue(isinstance(cvars, dict))
+        self.assertIsInstance(cvars, dict)
         self.assertTrue(cvars)
 
     def test_get_platform(self):
@@ -236,8 +234,8 @@
         # On Windows, the EXE needs to know where pythonXY.dll is at so we have
         # to add the directory to the path.
         if sys.platform == "win32":
-            os.environ["Path"] = "{};{}".format(
-                os.path.dirname(sys.executable), os.environ["Path"])
+            os.environ["PATH"] = "{};{}".format(
+                os.path.dirname(sys.executable), os.environ["PATH"])
 
         # Issue 7880
         def get(python):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 16:11:00 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 16:11:00 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Final_bag_of_small_changes_?=
	=?utf8?q?coming_from_distutils2=2E?=
Message-ID: <E1R5eYm-0005Gp-TR@dinsdale.python.org>

http://hg.python.org/cpython/rev/8aaf81d4ea66
changeset:   72409:8aaf81d4ea66
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 15:12:23 2011 +0200
summary:
  Final bag of small changes coming from distutils2.

- minor cleanup in Metadata
- trigger creation of the sysconfig._CONFIG_VARS dict
- home_page is used over home-page: it?s not a compound word, it?s an
  escaped space

Distutils2 is now synchronized with Packaging.

files:
  Lib/packaging/command/bdist_dumb.py            |   4 +-
  Lib/packaging/compat.py                        |   8 +------
  Lib/packaging/metadata.py                      |   7 +++--
  Lib/packaging/pypi/simple.py                   |   7 ++---
  Lib/packaging/tests/test_command_bdist_dumb.py |   2 +-
  Lib/packaging/tests/test_command_build_ext.py  |  11 ++++-----
  Lib/packaging/tests/test_command_register.py   |   2 +-
  Lib/packaging/tests/test_dist.py               |   6 ++--
  Lib/packaging/tests/test_metadata.py           |   4 +-
  9 files changed, 22 insertions(+), 29 deletions(-)


diff --git a/Lib/packaging/command/bdist_dumb.py b/Lib/packaging/command/bdist_dumb.py
--- a/Lib/packaging/command/bdist_dumb.py
+++ b/Lib/packaging/command/bdist_dumb.py
@@ -5,9 +5,9 @@
 """
 
 import os
-
 from shutil import rmtree
 from sysconfig import get_python_version
+
 from packaging.util import get_platform
 from packaging.command.cmd import Command
 from packaging.errors import PackagingPlatformError
@@ -24,7 +24,7 @@
                      "platform name to embed in generated filenames "
                      "(default: %s)" % get_platform()),
                     ('format=', 'f',
-                     "archive format to create (tar, gztar, zip)"),
+                     "archive format to create (tar, gztar, bztar, zip)"),
                     ('keep-temp', 'k',
                      "keep the pseudo-installation tree around after " +
                      "creating the distribution archive"),
diff --git a/Lib/packaging/compat.py b/Lib/packaging/compat.py
--- a/Lib/packaging/compat.py
+++ b/Lib/packaging/compat.py
@@ -1,8 +1,4 @@
-"""Compatibility helpers.
-
-This module provides classes, variables and imports which are used to
-support packaging across Python 2.x and 3.x.
-"""
+"""Compatibility helpers."""
 
 from packaging import logger
 
@@ -10,8 +6,6 @@
 # XXX Having two classes with the same name is not a good thing.
 # XXX 2to3-related code should move from util to this module
 
-# TODO Move common code here: PY3 (bool indicating if we're on 3.x), any, etc.
-
 try:
     from packaging.util import Mixin2to3 as _Mixin2to3
     _CONVERT = True
diff --git a/Lib/packaging/metadata.py b/Lib/packaging/metadata.py
--- a/Lib/packaging/metadata.py
+++ b/Lib/packaging/metadata.py
@@ -552,16 +552,17 @@
         return data
 
     # Mapping API
+    # XXX these methods should return views or sets in 3.x
 
     def keys(self):
-        return _version2fieldlist(self['Metadata-Version'])
+        return list(_version2fieldlist(self['Metadata-Version']))
 
     def __iter__(self):
         for key in self.keys():
             yield key
 
     def values(self):
-        return [self[key] for key in list(self.keys())]
+        return [self[key] for key in self.keys()]
 
     def items(self):
-        return [(key, self[key]) for key in list(self.keys())]
+        return [(key, self[key]) for key in self.keys()]
diff --git a/Lib/packaging/pypi/simple.py b/Lib/packaging/pypi/simple.py
--- a/Lib/packaging/pypi/simple.py
+++ b/Lib/packaging/pypi/simple.py
@@ -23,12 +23,11 @@
 from packaging import __version__ as packaging_version
 from packaging.pypi.base import BaseClient
 from packaging.pypi.dist import (ReleasesList, EXTENSIONS,
-                                  get_infos_from_url, MD5_HASH)
+                                 get_infos_from_url, MD5_HASH)
 from packaging.pypi.errors import (PackagingPyPIError, DownloadError,
-                                    UnableToDownload, CantParseArchiveName,
-                                    ReleaseNotFound, ProjectNotFound)
+                                   UnableToDownload, CantParseArchiveName,
+                                   ReleaseNotFound, ProjectNotFound)
 from packaging.pypi.mirrors import get_mirrors
-from packaging.metadata import Metadata
 
 __all__ = ['Crawler', 'DEFAULT_SIMPLE_INDEX_URL']
 
diff --git a/Lib/packaging/tests/test_command_bdist_dumb.py b/Lib/packaging/tests/test_command_bdist_dumb.py
--- a/Lib/packaging/tests/test_command_bdist_dumb.py
+++ b/Lib/packaging/tests/test_command_bdist_dumb.py
@@ -35,7 +35,7 @@
 
         dist = Distribution({'name': 'foo', 'version': '0.1',
                              'py_modules': ['foo'],
-                             'home-page': 'xxx', 'author': 'xxx',
+                             'home_page': 'xxx', 'author': 'xxx',
                              'author_email': 'xxx'})
         os.chdir(pkg_dir)
         cmd = bdist_dumb(dist)
diff --git a/Lib/packaging/tests/test_command_build_ext.py b/Lib/packaging/tests/test_command_build_ext.py
--- a/Lib/packaging/tests/test_command_build_ext.py
+++ b/Lib/packaging/tests/test_command_build_ext.py
@@ -4,14 +4,13 @@
 import sysconfig
 import textwrap
 from io import StringIO
-from sysconfig import _CONFIG_VARS
 from packaging.dist import Distribution
 from packaging.errors import (UnknownFileError, CompileError,
                               PackagingPlatformError)
 from packaging.command.build_ext import build_ext
 from packaging.compiler.extension import Extension
+
 from test.script_helper import assert_python_ok
-
 from packaging.tests import support, unittest, verbose
 
 
@@ -75,16 +74,16 @@
 
         sys.platform = 'sunos'  # fooling finalize_options
 
-        old_var = _CONFIG_VARS.get('Py_ENABLE_SHARED')
-        _CONFIG_VARS['Py_ENABLE_SHARED'] = 1
+        old_var = sysconfig.get_config_var('Py_ENABLE_SHARED')
+        sysconfig._CONFIG_VARS['Py_ENABLE_SHARED'] = 1
         try:
             cmd.ensure_finalized()
         finally:
             sys.platform = old
             if old_var is None:
-                del _CONFIG_VARS['Py_ENABLE_SHARED']
+                del sysconfig._CONFIG_VARS['Py_ENABLE_SHARED']
             else:
-                _CONFIG_VARS['Py_ENABLE_SHARED'] = old_var
+                sysconfig._CONFIG_VARS['Py_ENABLE_SHARED'] = old_var
 
         # make sure we get some library dirs under solaris
         self.assertGreater(len(cmd.library_dirs), 0)
diff --git a/Lib/packaging/tests/test_command_register.py b/Lib/packaging/tests/test_command_register.py
--- a/Lib/packaging/tests/test_command_register.py
+++ b/Lib/packaging/tests/test_command_register.py
@@ -99,7 +99,7 @@
 
     def _get_cmd(self, metadata=None):
         if metadata is None:
-            metadata = {'home-page': 'xxx', 'author': 'xxx',
+            metadata = {'home_page': 'xxx', 'author': 'xxx',
                         'author_email': 'xxx',
                         'name': 'xxx', 'version': 'xxx'}
         pkg_info, dist = self.create_dist(**metadata)
diff --git a/Lib/packaging/tests/test_dist.py b/Lib/packaging/tests/test_dist.py
--- a/Lib/packaging/tests/test_dist.py
+++ b/Lib/packaging/tests/test_dist.py
@@ -72,7 +72,7 @@
         Distribution(attrs={'author': 'xxx',
                             'name': 'xxx',
                             'version': '1.2',
-                            'home-page': 'xxxx',
+                            'home_page': 'xxxx',
                             'badoptname': 'xxx'})
         logs = self.get_logs(logging.WARNING)
         self.assertEqual(len(logs), 1)
@@ -82,7 +82,7 @@
         # an empty options dictionary should not stay in the
         # list of attributes
         dist = Distribution(attrs={'author': 'xxx', 'name': 'xxx',
-                                   'version': '1.2', 'home-page': 'xxxx',
+                                   'version': '1.2', 'home_page': 'xxxx',
                                    'options': {}})
 
         self.assertEqual([], self.get_logs(logging.WARNING))
@@ -99,7 +99,7 @@
         dist = Distribution(attrs={'author': 'xxx',
                                    'name': 'xxx',
                                    'version': 'xxx',
-                                   'home-page': 'xxxx',
+                                   'home_page': 'xxxx',
                                    'options': {'sdist': {'owner': 'root'}}})
 
         self.assertIn('owner', dist.get_option_dict('sdist'))
diff --git a/Lib/packaging/tests/test_metadata.py b/Lib/packaging/tests/test_metadata.py
--- a/Lib/packaging/tests/test_metadata.py
+++ b/Lib/packaging/tests/test_metadata.py
@@ -101,7 +101,7 @@
 
         # XXX caveat: the keys method and friends are not 3.x-style views
         # should be changed or documented
-        self.assertEqual(list(metadata), list(metadata.keys()))
+        self.assertEqual(list(metadata), metadata.keys())
 
     def test_read_metadata(self):
         fields = {'name': 'project',
@@ -301,7 +301,7 @@
         Metadata(mapping={'author': 'xxx',
                           'name': 'xxx',
                           'version': 'xxx',
-                          'home-page': 'xxxx'})
+                          'home_page': 'xxxx'})
         logs = self.get_logs(logging.WARNING)
         self.assertEqual(1, len(logs))
         self.assertIn('not a valid version', logs[0])

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 16:11:01 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 16:11:01 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Make_regrtest_look_at_inter?=
	=?utf8?q?nal_dicts_in_sysconfig=2E?=
Message-ID: <E1R5eYn-0005HQ-G6@dinsdale.python.org>

http://hg.python.org/cpython/rev/9c61f46ea6b4
changeset:   72410:9c61f46ea6b4
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 05:10:45 2011 +0200
summary:
  Make regrtest look at internal dicts in sysconfig.

This reveals problems in the packaging test suite, which I?ll look
into after the regrtest checks are made more usable (see #12314).

files:
  Lib/test/regrtest.py |  25 ++++++++++++++++++++++++-
  1 files changed, 24 insertions(+), 1 deletions(-)


diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -965,7 +965,9 @@
                  'warnings.filters', 'asyncore.socket_map',
                  'logging._handlers', 'logging._handlerList', 'sys.gettrace',
                  'sys.warnoptions', 'threading._dangling',
-                 'multiprocessing.process._dangling')
+                 'multiprocessing.process._dangling',
+                 'sysconfig._CONFIG_VARS', 'sysconfig._SCHEMES',
+                )
 
     def get_sys_argv(self):
         return id(sys.argv), sys.argv, sys.argv[:]
@@ -1083,6 +1085,27 @@
         multiprocessing.process._dangling.clear()
         multiprocessing.process._dangling.update(saved)
 
+    def get_sysconfig__CONFIG_VARS(self):
+        # make sure the dict is initialized
+        sysconfig.get_config_var('prefix')
+        return (id(sysconfig._CONFIG_VARS), sysconfig._CONFIG_VARS,
+                dict(sysconfig._CONFIG_VARS))
+    def restore_sysconfig__CONFIG_VARS(self, saved):
+        sysconfig._CONFIG_VARS = saved[1]
+        sysconfig._CONFIG_VARS.clear()
+        sysconfig._CONFIG_VARS.update(saved[2])
+
+    def get_sysconfig__SCHEMES(self):
+        # it's mildly evil to look at the internal attribute, but it's easier
+        # than copying a RawConfigParser object
+        return (id(sysconfig._SCHEMES), sysconfig._SCHEMES._sections,
+                sysconfig._SCHEMES._sections.copy())
+    def restore_sysconfig__SCHEMES(self, saved):
+        sysconfig._SCHEMES._sections = saved[1]
+        sysconfig._SCHEMES._sections.clear()
+        sysconfig._SCHEMES._sections.update(saved[2])
+
+
     def resource_info(self):
         for name in self.resources:
             method_suffix = name.replace('.', '_')

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 16:11:02 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 16:11:02 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Branch_merge?=
Message-ID: <E1R5eYo-0005HV-55@dinsdale.python.org>

http://hg.python.org/cpython/rev/cb56ab0cb9c5
changeset:   72411:cb56ab0cb9c5
parent:      72400:a4e4facad164
parent:      72410:9c61f46ea6b4
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 16:10:26 2011 +0200
summary:
  Branch merge

files:
  Lib/packaging/command/bdist_dumb.py              |   4 +-
  Lib/packaging/command/build_clib.py              |   3 +-
  Lib/packaging/command/build_ext.py               |  17 +-
  Lib/packaging/command/build_py.py                |   2 +-
  Lib/packaging/command/build_scripts.py           |   4 +-
  Lib/packaging/command/install_dist.py            |  59 +++-----
  Lib/packaging/command/install_distinfo.py        |   9 +-
  Lib/packaging/command/install_lib.py             |   2 +-
  Lib/packaging/command/upload_docs.py             |   1 +
  Lib/packaging/compat.py                          |   8 +-
  Lib/packaging/compiler/bcppcompiler.py           |   1 -
  Lib/packaging/compiler/ccompiler.py              |   1 -
  Lib/packaging/compiler/unixccompiler.py          |   2 +-
  Lib/packaging/create.py                          |  22 +--
  Lib/packaging/database.py                        |   7 +-
  Lib/packaging/dist.py                            |   4 +-
  Lib/packaging/metadata.py                        |   7 +-
  Lib/packaging/pypi/simple.py                     |   7 +-
  Lib/packaging/tests/__init__.py                  |  13 +-
  Lib/packaging/tests/__main__.py                  |   2 +
  Lib/packaging/tests/pypi_server.py               |   7 +-
  Lib/packaging/tests/test_command_bdist_dumb.py   |   2 +-
  Lib/packaging/tests/test_command_build_ext.py    |  19 +-
  Lib/packaging/tests/test_command_build_py.py     |   2 -
  Lib/packaging/tests/test_command_install_dist.py |  12 +-
  Lib/packaging/tests/test_command_install_lib.py  |  11 +-
  Lib/packaging/tests/test_command_register.py     |   2 +-
  Lib/packaging/tests/test_command_upload.py       |  24 +--
  Lib/packaging/tests/test_command_upload_docs.py  |  67 ++++-----
  Lib/packaging/tests/test_create.py               |  13 +-
  Lib/packaging/tests/test_database.py             |   5 +-
  Lib/packaging/tests/test_depgraph.py             |  33 ++--
  Lib/packaging/tests/test_dist.py                 |  16 +-
  Lib/packaging/tests/test_install.py              |   2 +-
  Lib/packaging/tests/test_metadata.py             |   4 +-
  Lib/packaging/tests/test_mixin2to3.py            |   3 -
  Lib/packaging/tests/test_pypi_server.py          |  15 +-
  Lib/packaging/tests/test_run.py                  |  17 +--
  Lib/packaging/tests/test_util.py                 |  10 +-
  Lib/packaging/util.py                            |   6 +-
  Lib/pkgutil.py                                   |   4 +-
  Lib/shutil.py                                    |   8 +-
  Lib/test/regrtest.py                             |  25 +++-
  Lib/test/test_sysconfig.py                       |  16 +-
  44 files changed, 215 insertions(+), 283 deletions(-)


diff --git a/Lib/packaging/command/bdist_dumb.py b/Lib/packaging/command/bdist_dumb.py
--- a/Lib/packaging/command/bdist_dumb.py
+++ b/Lib/packaging/command/bdist_dumb.py
@@ -5,9 +5,9 @@
 """
 
 import os
-
 from shutil import rmtree
 from sysconfig import get_python_version
+
 from packaging.util import get_platform
 from packaging.command.cmd import Command
 from packaging.errors import PackagingPlatformError
@@ -24,7 +24,7 @@
                      "platform name to embed in generated filenames "
                      "(default: %s)" % get_platform()),
                     ('format=', 'f',
-                     "archive format to create (tar, gztar, zip)"),
+                     "archive format to create (tar, gztar, bztar, zip)"),
                     ('keep-temp', 'k',
                      "keep the pseudo-installation tree around after " +
                      "creating the distribution archive"),
diff --git a/Lib/packaging/command/build_clib.py b/Lib/packaging/command/build_clib.py
--- a/Lib/packaging/command/build_clib.py
+++ b/Lib/packaging/command/build_clib.py
@@ -16,7 +16,7 @@
 import os
 from packaging.command.cmd import Command
 from packaging.errors import PackagingSetupError
-from packaging.compiler import customize_compiler
+from packaging.compiler import customize_compiler, new_compiler
 from packaging import logger
 
 
@@ -93,7 +93,6 @@
             return
 
         # Yech -- this is cut 'n pasted from build_ext.py!
-        from packaging.compiler import new_compiler
         self.compiler = new_compiler(compiler=self.compiler,
                                      dry_run=self.dry_run,
                                      force=self.force)
diff --git a/Lib/packaging/command/build_ext.py b/Lib/packaging/command/build_ext.py
--- a/Lib/packaging/command/build_ext.py
+++ b/Lib/packaging/command/build_ext.py
@@ -3,6 +3,7 @@
 import os
 import re
 import sys
+import site
 import logging
 import sysconfig
 
@@ -15,9 +16,6 @@
 from packaging.compiler.extension import Extension
 from packaging import logger
 
-import site
-HAS_USER_SITE = True
-
 if os.name == 'nt':
     from packaging.compiler.msvccompiler import get_build_version
     MSVC_VERSION = int(get_build_version())
@@ -62,6 +60,8 @@
         ('inplace', 'i',
          "ignore build-lib and put compiled extensions into the source " +
          "directory alongside your pure Python modules"),
+        ('user', None,
+         "add user include, library and rpath"),
         ('include-dirs=', 'I',
          "list of directories to search for header files" + sep_by),
         ('define=', 'D',
@@ -88,12 +88,8 @@
          "path to the SWIG executable"),
         ]
 
-    boolean_options = ['inplace', 'debug', 'force']
+    boolean_options = ['inplace', 'debug', 'force', 'user']
 
-    if HAS_USER_SITE:
-        user_options.append(('user', None,
-                             "add user include, library and rpath"))
-        boolean_options.append('user')
 
     help_options = [
         ('help-compiler', None,
@@ -120,8 +116,7 @@
         self.compiler = None
         self.swig = None
         self.swig_opts = None
-        if HAS_USER_SITE:
-            self.user = None
+        self.user = None
 
     def finalize_options(self):
         self.set_undefined_options('build',
@@ -270,7 +265,7 @@
             self.swig_opts = self.swig_opts.split(' ')
 
         # Finally add the user include and library directories if requested
-        if HAS_USER_SITE and self.user:
+        if self.user:
             user_include = os.path.join(site.USER_BASE, "include")
             user_lib = os.path.join(site.USER_BASE, "lib")
             if os.path.isdir(user_include):
diff --git a/Lib/packaging/command/build_py.py b/Lib/packaging/command/build_py.py
--- a/Lib/packaging/command/build_py.py
+++ b/Lib/packaging/command/build_py.py
@@ -388,7 +388,7 @@
                 self.build_module(module, module_file, package)
 
     def byte_compile(self, files):
-        if hasattr(sys, 'dont_write_bytecode') and sys.dont_write_bytecode:
+        if sys.dont_write_bytecode:
             logger.warning('%s: byte-compiling is disabled, skipping.',
                            self.get_command_name())
             return
diff --git a/Lib/packaging/command/build_scripts.py b/Lib/packaging/command/build_scripts.py
--- a/Lib/packaging/command/build_scripts.py
+++ b/Lib/packaging/command/build_scripts.py
@@ -3,7 +3,7 @@
 import os
 import re
 import sysconfig
-import tokenize
+from tokenize import detect_encoding
 
 from packaging.command.cmd import Command
 from packaging.util import convert_path, newer
@@ -83,7 +83,7 @@
                     raise
                 f = None
             else:
-                encoding, lines = tokenize.detect_encoding(f.readline)
+                encoding, lines = detect_encoding(f.readline)
                 f.seek(0)
                 first_line = f.readline()
                 if not first_line:
diff --git a/Lib/packaging/command/install_dist.py b/Lib/packaging/command/install_dist.py
--- a/Lib/packaging/command/install_dist.py
+++ b/Lib/packaging/command/install_dist.py
@@ -14,9 +14,6 @@
 from packaging.errors import PackagingOptionError
 
 
-HAS_USER_SITE = True
-
-
 class install_dist(Command):
 
     description = "install everything from build directory"
@@ -27,6 +24,9 @@
          "installation prefix"),
         ('exec-prefix=', None,
          "(Unix only) prefix for platform-specific files"),
+        ('user', None,
+         "install in user site-packages directory [%s]" %
+         get_path('purelib', '%s_user' % os.name)),
         ('home=', None,
          "(Unix only) home directory to install under"),
 
@@ -97,15 +97,7 @@
         ]
 
     boolean_options = ['compile', 'force', 'skip-build', 'no-distinfo',
-                       'requested', 'no-record']
-
-    if HAS_USER_SITE:
-        user_options.append(
-            ('user', None,
-             "install in user site-packages directory [%s]" %
-             get_path('purelib', '%s_user' % os.name)))
-
-        boolean_options.append('user')
+                       'requested', 'no-record', 'user']
 
     negative_opt = {'no-compile': 'compile', 'no-requested': 'requested'}
 
@@ -115,8 +107,7 @@
         self.prefix = None
         self.exec_prefix = None
         self.home = None
-        if HAS_USER_SITE:
-            self.user = False
+        self.user = False
 
         # These select only the installation base; it's up to the user to
         # specify the installation scheme (currently, that means supplying
@@ -135,9 +126,8 @@
         self.install_lib = None         # set to either purelib or platlib
         self.install_scripts = None
         self.install_data = None
-        if HAS_USER_SITE:
-            self.install_userbase = get_config_var('userbase')
-            self.install_usersite = get_path('purelib', '%s_user' % os.name)
+        self.install_userbase = get_config_var('userbase')
+        self.install_usersite = get_path('purelib', '%s_user' % os.name)
 
         self.compile = None
         self.optimize = None
@@ -219,9 +209,8 @@
             raise PackagingOptionError(
                 "must supply either home or prefix/exec-prefix -- not both")
 
-        if HAS_USER_SITE and self.user and (
-                self.prefix or self.exec_prefix or self.home or
-                self.install_base or self.install_platbase):
+        if self.user and (self.prefix or self.exec_prefix or self.home or
+                          self.install_base or self.install_platbase):
             raise PackagingOptionError(
                 "can't combine user with prefix/exec_prefix/home or "
                 "install_base/install_platbase")
@@ -274,11 +263,9 @@
             'exec_prefix': exec_prefix,
             'srcdir': srcdir,
             'projectbase': projectbase,
-            }
-
-        if HAS_USER_SITE:
-            self.config_vars['userbase'] = self.install_userbase
-            self.config_vars['usersite'] = self.install_usersite
+            'userbase': self.install_userbase,
+            'usersite': self.install_usersite,
+        }
 
         self.expand_basedirs()
 
@@ -295,9 +282,9 @@
 
         self.dump_dirs("post-expand_dirs()")
 
-        # Create directories in the home dir:
-        if HAS_USER_SITE and self.user:
-            self.create_home_path()
+        # Create directories under USERBASE
+        if self.user:
+            self.create_user_dirs()
 
         # Pick the actual directory to install all modules to: either
         # install_purelib or install_platlib, depending on whether this
@@ -311,10 +298,8 @@
 
         # Convert directories from Unix /-separated syntax to the local
         # convention.
-        self.convert_paths('lib', 'purelib', 'platlib',
-                           'scripts', 'data', 'headers')
-        if HAS_USER_SITE:
-            self.convert_paths('userbase', 'usersite')
+        self.convert_paths('lib', 'purelib', 'platlib', 'scripts',
+                           'data', 'headers', 'userbase', 'usersite')
 
         # Well, we're not actually fully completely finalized yet: we still
         # have to deal with 'extra_path', which is the hack for allowing
@@ -355,7 +340,7 @@
                     "installation scheme is incomplete")
             return
 
-        if HAS_USER_SITE and self.user:
+        if self.user:
             if self.install_userbase is None:
                 raise PackagingPlatformError(
                     "user base directory is not specified")
@@ -383,7 +368,7 @@
 
     def finalize_other(self):
         """Finalize options for non-posix platforms"""
-        if HAS_USER_SITE and self.user:
+        if self.user:
             if self.install_userbase is None:
                 raise PackagingPlatformError(
                     "user base directory is not specified")
@@ -494,10 +479,8 @@
             attr = "install_" + name
             setattr(self, attr, change_root(self.root, getattr(self, attr)))
 
-    def create_home_path(self):
-        """Create directories under ~."""
-        if HAS_USER_SITE and not self.user:
-            return
+    def create_user_dirs(self):
+        """Create directories under USERBASE as needed."""
         home = convert_path(os.path.expanduser("~"))
         for name, path in self.config_vars.items():
             if path.startswith(home) and not os.path.isdir(path):
diff --git a/Lib/packaging/command/install_distinfo.py b/Lib/packaging/command/install_distinfo.py
--- a/Lib/packaging/command/install_distinfo.py
+++ b/Lib/packaging/command/install_distinfo.py
@@ -2,14 +2,13 @@
 
 # Forked from the former install_egg_info command by Josip Djolonga
 
+import os
 import csv
-import os
-import re
 import hashlib
+from shutil import rmtree
 
+from packaging import logger
 from packaging.command.cmd import Command
-from packaging import logger
-from shutil import rmtree
 
 
 class install_distinfo(Command):
@@ -28,7 +27,7 @@
         ('no-record', None,
          "do not generate a RECORD file"),
         ('no-resources', None,
-         "do not generate a RESSOURCES list installed file"),
+         "do not generate a RESOURCES file"),
     ]
 
     boolean_options = ['requested', 'no-record', 'no-resources']
diff --git a/Lib/packaging/command/install_lib.py b/Lib/packaging/command/install_lib.py
--- a/Lib/packaging/command/install_lib.py
+++ b/Lib/packaging/command/install_lib.py
@@ -114,7 +114,7 @@
         return outfiles
 
     def byte_compile(self, files):
-        if getattr(sys, 'dont_write_bytecode'):
+        if sys.dont_write_bytecode:
             # XXX do we want this?  because a Python runs without bytecode
             # doesn't mean that the *dists should not contain bytecode
             #--or does it?
diff --git a/Lib/packaging/command/upload_docs.py b/Lib/packaging/command/upload_docs.py
--- a/Lib/packaging/command/upload_docs.py
+++ b/Lib/packaging/command/upload_docs.py
@@ -87,6 +87,7 @@
         content_type, body = encode_multipart(fields, files)
 
         credentials = self.username + ':' + self.password
+        # FIXME should use explicit encoding
         auth = b"Basic " + base64.encodebytes(credentials.encode()).strip()
 
         logger.info("Submitting documentation to %s", self.repository)
diff --git a/Lib/packaging/compat.py b/Lib/packaging/compat.py
--- a/Lib/packaging/compat.py
+++ b/Lib/packaging/compat.py
@@ -1,8 +1,4 @@
-"""Compatibility helpers.
-
-This module provides classes, variables and imports which are used to
-support packaging across Python 2.x and 3.x.
-"""
+"""Compatibility helpers."""
 
 from packaging import logger
 
@@ -10,8 +6,6 @@
 # XXX Having two classes with the same name is not a good thing.
 # XXX 2to3-related code should move from util to this module
 
-# TODO Move common code here: PY3 (bool indicating if we're on 3.x), any, etc.
-
 try:
     from packaging.util import Mixin2to3 as _Mixin2to3
     _CONVERT = True
diff --git a/Lib/packaging/compiler/bcppcompiler.py b/Lib/packaging/compiler/bcppcompiler.py
--- a/Lib/packaging/compiler/bcppcompiler.py
+++ b/Lib/packaging/compiler/bcppcompiler.py
@@ -352,5 +352,4 @@
             try:
                 self.spawn(pp_args)
             except PackagingExecError as msg:
-                print(msg)
                 raise CompileError(msg)
diff --git a/Lib/packaging/compiler/ccompiler.py b/Lib/packaging/compiler/ccompiler.py
--- a/Lib/packaging/compiler/ccompiler.py
+++ b/Lib/packaging/compiler/ccompiler.py
@@ -5,7 +5,6 @@
 """
 
 import os
-import sys
 from shutil import move
 from packaging import logger
 from packaging.util import split_quoted, execute, newer_group, spawn
diff --git a/Lib/packaging/compiler/unixccompiler.py b/Lib/packaging/compiler/unixccompiler.py
--- a/Lib/packaging/compiler/unixccompiler.py
+++ b/Lib/packaging/compiler/unixccompiler.py
@@ -127,7 +127,7 @@
         executables['ranlib'] = ["ranlib"]
 
     # Needed for the filename generation methods provided by the base
-    # class, CCompiler.  NB. whoever instantiates/uses a particular
+    # class, CCompiler.  XXX whoever instantiates/uses a particular
     # UnixCCompiler instance should set 'shared_lib_ext' -- we set a
     # reasonable common default here, but it's not necessarily used on all
     # Unices!
diff --git a/Lib/packaging/create.py b/Lib/packaging/create.py
--- a/Lib/packaging/create.py
+++ b/Lib/packaging/create.py
@@ -25,11 +25,11 @@
 import glob
 import shutil
 import sysconfig
-import tokenize
 from hashlib import md5
 from textwrap import dedent
-from functools import cmp_to_key
+from tokenize import detect_encoding
 from configparser import RawConfigParser
+
 # importing this with an underscore as it should be replaced by the
 # dict form or another structures for all purposes
 from packaging._trove import all_classifiers as _CLASSIFIERS_LIST
@@ -112,7 +112,7 @@
     been loaded before, because we are monkey patching its setup function with
     a particular one"""
     with open("setup.py", "rb") as f:
-        encoding, lines = tokenize.detect_encoding(f.readline)
+        encoding, lines = detect_encoding(f.readline)
     with open("setup.py", encoding=encoding) as f:
         imp.load_module("setup", f, "setup.py", (".py", "r", imp.PY_SOURCE))
 
@@ -370,21 +370,9 @@
                     dist.data_files = [('', dist.data_files)]
                 # add tokens in the destination paths
                 vars = {'distribution.name': data['name']}
-                path_tokens = list(sysconfig.get_paths(vars=vars).items())
-
-                # TODO replace this with a key function
-                def length_comparison(x, y):
-                    len_x = len(x[1])
-                    len_y = len(y[1])
-                    if len_x == len_y:
-                        return 0
-                    elif len_x < len_y:
-                        return -1
-                    else:
-                        return 1
-
+                path_tokens = sysconfig.get_paths(vars=vars).items()
                 # sort tokens to use the longest one first
-                path_tokens.sort(key=cmp_to_key(length_comparison))
+                path_tokens = sorted(path_tokens, key=lambda x: len(x[1]))
                 for dest, srcs in (dist.data_files or []):
                     dest = os.path.join(sys.prefix, dest)
                     dest = dest.replace(os.path.sep, '/')
diff --git a/Lib/packaging/database.py b/Lib/packaging/database.py
--- a/Lib/packaging/database.py
+++ b/Lib/packaging/database.py
@@ -1,12 +1,13 @@
 """PEP 376 implementation."""
 
-import io
 import os
 import re
 import csv
 import sys
 import zipimport
+from io import StringIO
 from hashlib import md5
+
 from packaging import logger
 from packaging.errors import PackagingError
 from packaging.version import suggest_normalized_version, VersionPredicate
@@ -173,7 +174,7 @@
     def get_resource_path(self, relative_path):
         with self.get_distinfo_file('RESOURCES') as resources_file:
             resources_reader = csv.reader(resources_file, delimiter=',',
-                                           lineterminator='\n')
+                                          lineterminator='\n')
             for relative, destination in resources_reader:
                 if relative == relative_path:
                     return destination
@@ -334,7 +335,7 @@
             else:
                 # FIXME handle the case where zipfile is not available
                 zipf = zipimport.zipimporter(path)
-                fileobj = io.StringIO(
+                fileobj = StringIO(
                     zipf.get_data('EGG-INFO/PKG-INFO').decode('utf8'))
                 self.metadata = Metadata(fileobj=fileobj)
                 try:
diff --git a/Lib/packaging/dist.py b/Lib/packaging/dist.py
--- a/Lib/packaging/dist.py
+++ b/Lib/packaging/dist.py
@@ -537,7 +537,7 @@
     def _get_command_groups(self):
         """Helper function to retrieve all the command class names divided
         into standard commands (listed in
-        packaging2.command.STANDARD_COMMANDS) and extra commands (given in
+        packaging.command.STANDARD_COMMANDS) and extra commands (given in
         self.cmdclass and not standard commands).
         """
         extra_commands = [cmd for cmd in self.cmdclass
@@ -547,7 +547,7 @@
     def print_commands(self):
         """Print out a help message listing all available commands with a
         description of each.  The list is divided into standard commands
-        (listed in packaging2.command.STANDARD_COMMANDS) and extra commands
+        (listed in packaging.command.STANDARD_COMMANDS) and extra commands
         (given in self.cmdclass and not standard commands).  The
         descriptions come from the command class attribute
         'description'.
diff --git a/Lib/packaging/metadata.py b/Lib/packaging/metadata.py
--- a/Lib/packaging/metadata.py
+++ b/Lib/packaging/metadata.py
@@ -552,16 +552,17 @@
         return data
 
     # Mapping API
+    # XXX these methods should return views or sets in 3.x
 
     def keys(self):
-        return _version2fieldlist(self['Metadata-Version'])
+        return list(_version2fieldlist(self['Metadata-Version']))
 
     def __iter__(self):
         for key in self.keys():
             yield key
 
     def values(self):
-        return [self[key] for key in list(self.keys())]
+        return [self[key] for key in self.keys()]
 
     def items(self):
-        return [(key, self[key]) for key in list(self.keys())]
+        return [(key, self[key]) for key in self.keys()]
diff --git a/Lib/packaging/pypi/simple.py b/Lib/packaging/pypi/simple.py
--- a/Lib/packaging/pypi/simple.py
+++ b/Lib/packaging/pypi/simple.py
@@ -23,12 +23,11 @@
 from packaging import __version__ as packaging_version
 from packaging.pypi.base import BaseClient
 from packaging.pypi.dist import (ReleasesList, EXTENSIONS,
-                                  get_infos_from_url, MD5_HASH)
+                                 get_infos_from_url, MD5_HASH)
 from packaging.pypi.errors import (PackagingPyPIError, DownloadError,
-                                    UnableToDownload, CantParseArchiveName,
-                                    ReleaseNotFound, ProjectNotFound)
+                                   UnableToDownload, CantParseArchiveName,
+                                   ReleaseNotFound, ProjectNotFound)
 from packaging.pypi.mirrors import get_mirrors
-from packaging.metadata import Metadata
 
 __all__ = ['Crawler', 'DEFAULT_SIMPLE_INDEX_URL']
 
diff --git a/Lib/packaging/tests/__init__.py b/Lib/packaging/tests/__init__.py
--- a/Lib/packaging/tests/__init__.py
+++ b/Lib/packaging/tests/__init__.py
@@ -6,17 +6,15 @@
 to return an initialized unittest.TestSuite instance.
 
 Utility code is included in packaging.tests.support.
+
+Always import unittest from this module: it will be unittest from the
+standard library for packaging tests and unittest2 for distutils2 tests.
 """
 
-# Put this text back for the backport
-#Always import unittest from this module, it will be the right version
-#(standard library unittest for 3.2 and higher, third-party unittest2
-#elease for older versions).
-
 import os
 import sys
 import unittest
-from test.support import TESTFN
+from io import StringIO
 
 # XXX move helpers to support, add tests for them, remove things that
 # duplicate test.support (or keep them for the backport; needs thinking)
@@ -115,9 +113,8 @@
 
 
 def captured_stdout(func, *args, **kw):
-    import io
     orig_stdout = getattr(sys, 'stdout')
-    setattr(sys, 'stdout', io.StringIO())
+    setattr(sys, 'stdout', StringIO())
     try:
         res = func(*args, **kw)
         sys.stdout.seek(0)
diff --git a/Lib/packaging/tests/__main__.py b/Lib/packaging/tests/__main__.py
--- a/Lib/packaging/tests/__main__.py
+++ b/Lib/packaging/tests/__main__.py
@@ -14,6 +14,8 @@
         start_dir = os.path.dirname(__file__)
         top_dir = os.path.dirname(os.path.dirname(start_dir))
         test_loader = unittest.TestLoader()
+        # XXX find out how to use unittest.main, to get command-line options
+        # (failfast, catch, etc.)
         run_unittest(test_loader.discover(start_dir, top_level_dir=top_dir))
     finally:
         reap_children()
diff --git a/Lib/packaging/tests/pypi_server.py b/Lib/packaging/tests/pypi_server.py
--- a/Lib/packaging/tests/pypi_server.py
+++ b/Lib/packaging/tests/pypi_server.py
@@ -40,6 +40,7 @@
 
 from packaging.tests import unittest
 
+
 PYPI_DEFAULT_STATIC_PATH = os.path.join(
     os.path.dirname(os.path.abspath(__file__)), 'pypiserver')
 
@@ -219,7 +220,7 @@
                         relative_path += "index.html"
 
                     if relative_path.endswith('.tar.gz'):
-                        with open(fs_path + relative_path, 'br') as file:
+                        with open(fs_path + relative_path, 'rb') as file:
                             data = file.read()
                         headers = [('Content-type', 'application/x-gtar')]
                     else:
@@ -260,8 +261,8 @@
             self.send_header(header, value)
         self.end_headers()
 
-        if type(data) is str:
-            data = data.encode()
+        if isinstance(data, str):
+            data = data.encode('utf-8')
 
         self.wfile.write(data)
 
diff --git a/Lib/packaging/tests/test_command_bdist_dumb.py b/Lib/packaging/tests/test_command_bdist_dumb.py
--- a/Lib/packaging/tests/test_command_bdist_dumb.py
+++ b/Lib/packaging/tests/test_command_bdist_dumb.py
@@ -35,7 +35,7 @@
 
         dist = Distribution({'name': 'foo', 'version': '0.1',
                              'py_modules': ['foo'],
-                             'home-page': 'xxx', 'author': 'xxx',
+                             'home_page': 'xxx', 'author': 'xxx',
                              'author_email': 'xxx'})
         os.chdir(pkg_dir)
         cmd = bdist_dumb(dist)
diff --git a/Lib/packaging/tests/test_command_build_ext.py b/Lib/packaging/tests/test_command_build_ext.py
--- a/Lib/packaging/tests/test_command_build_ext.py
+++ b/Lib/packaging/tests/test_command_build_ext.py
@@ -9,8 +9,8 @@
                               PackagingPlatformError)
 from packaging.command.build_ext import build_ext
 from packaging.compiler.extension import Extension
+
 from test.script_helper import assert_python_ok
-
 from packaging.tests import support, unittest, verbose
 
 
@@ -18,18 +18,13 @@
                        support.LoggingCatcher,
                        unittest.TestCase):
     def setUp(self):
-        # Create a simple test environment
-        # Note that we're making changes to sys.path
         super(BuildExtTestCase, self).setUp()
         self.tmp_dir = self.mkdtemp()
         self.old_user_base = site.USER_BASE
         site.USER_BASE = self.mkdtemp()
 
     def tearDown(self):
-        # Get everything back to normal
-        if sys.version > "2.6":
-            site.USER_BASE = self.old_user_base
-
+        site.USER_BASE = self.old_user_base
         super(BuildExtTestCase, self).tearDown()
 
     def test_build_ext(self):
@@ -78,23 +73,21 @@
         old = sys.platform
 
         sys.platform = 'sunos'  # fooling finalize_options
-        from sysconfig import _CONFIG_VARS
 
-        old_var = _CONFIG_VARS.get('Py_ENABLE_SHARED')
-        _CONFIG_VARS['Py_ENABLE_SHARED'] = 1
+        old_var = sysconfig.get_config_var('Py_ENABLE_SHARED')
+        sysconfig._CONFIG_VARS['Py_ENABLE_SHARED'] = 1
         try:
             cmd.ensure_finalized()
         finally:
             sys.platform = old
             if old_var is None:
-                del _CONFIG_VARS['Py_ENABLE_SHARED']
+                del sysconfig._CONFIG_VARS['Py_ENABLE_SHARED']
             else:
-                _CONFIG_VARS['Py_ENABLE_SHARED'] = old_var
+                sysconfig._CONFIG_VARS['Py_ENABLE_SHARED'] = old_var
 
         # make sure we get some library dirs under solaris
         self.assertGreater(len(cmd.library_dirs), 0)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_user_site(self):
         dist = Distribution({'name': 'xx'})
         cmd = build_ext(dist)
diff --git a/Lib/packaging/tests/test_command_build_py.py b/Lib/packaging/tests/test_command_build_py.py
--- a/Lib/packaging/tests/test_command_build_py.py
+++ b/Lib/packaging/tests/test_command_build_py.py
@@ -99,8 +99,6 @@
             os.chdir(cwd)
             sys.stdout = old_stdout
 
-    @unittest.skipUnless(hasattr(sys, 'dont_write_bytecode'),
-                         'sys.dont_write_bytecode not supported')
     def test_dont_write_bytecode(self):
         # makes sure byte_compile is not used
         pkg_dir, dist = self.create_dist()
diff --git a/Lib/packaging/tests/test_command_install_dist.py b/Lib/packaging/tests/test_command_install_dist.py
--- a/Lib/packaging/tests/test_command_install_dist.py
+++ b/Lib/packaging/tests/test_command_install_dist.py
@@ -72,7 +72,6 @@
         check_path(cmd.install_scripts, os.path.join(destination, "bin"))
         check_path(cmd.install_data, destination)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_user_site(self):
         # test install with --user
         # preparing the environment for the test
@@ -173,12 +172,11 @@
         cmd.home = 'home'
         self.assertRaises(PackagingOptionError, cmd.finalize_options)
 
-        if sys.version >= '2.6':
-            # can't combine user with with prefix/exec_prefix/home or
-            # install_(plat)base
-            cmd.prefix = None
-            cmd.user = 'user'
-            self.assertRaises(PackagingOptionError, cmd.finalize_options)
+        # can't combine user with with prefix/exec_prefix/home or
+        # install_(plat)base
+        cmd.prefix = None
+        cmd.user = 'user'
+        self.assertRaises(PackagingOptionError, cmd.finalize_options)
 
     def test_old_record(self):
         # test pre-PEP 376 --record option (outside dist-info dir)
diff --git a/Lib/packaging/tests/test_command_install_lib.py b/Lib/packaging/tests/test_command_install_lib.py
--- a/Lib/packaging/tests/test_command_install_lib.py
+++ b/Lib/packaging/tests/test_command_install_lib.py
@@ -7,13 +7,6 @@
 from packaging.compiler.extension import Extension
 from packaging.errors import PackagingOptionError
 
-try:
-    no_bytecode = sys.dont_write_bytecode
-    bytecode_support = True
-except AttributeError:
-    no_bytecode = False
-    bytecode_support = False
-
 
 class InstallLibTestCase(support.TempdirManager,
                          support.LoggingCatcher,
@@ -40,7 +33,7 @@
         cmd.finalize_options()
         self.assertEqual(cmd.optimize, 2)
 
-    @unittest.skipIf(no_bytecode, 'byte-compile not supported')
+    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
     def test_byte_compile(self):
         pkg_dir, dist = self.create_dist()
         cmd = install_lib(dist)
@@ -89,8 +82,6 @@
         # get_input should return 2 elements
         self.assertEqual(len(cmd.get_inputs()), 2)
 
-    @unittest.skipUnless(bytecode_support,
-                         'sys.dont_write_bytecode not supported')
     def test_dont_write_bytecode(self):
         # makes sure byte_compile is not used
         pkg_dir, dist = self.create_dist()
diff --git a/Lib/packaging/tests/test_command_register.py b/Lib/packaging/tests/test_command_register.py
--- a/Lib/packaging/tests/test_command_register.py
+++ b/Lib/packaging/tests/test_command_register.py
@@ -99,7 +99,7 @@
 
     def _get_cmd(self, metadata=None):
         if metadata is None:
-            metadata = {'home-page': 'xxx', 'author': 'xxx',
+            metadata = {'home_page': 'xxx', 'author': 'xxx',
                         'author_email': 'xxx',
                         'name': 'xxx', 'version': 'xxx'}
         pkg_info, dist = self.create_dist(**metadata)
diff --git a/Lib/packaging/tests/test_command_upload.py b/Lib/packaging/tests/test_command_upload.py
--- a/Lib/packaging/tests/test_command_upload.py
+++ b/Lib/packaging/tests/test_command_upload.py
@@ -1,6 +1,5 @@
 """Tests for packaging.command.upload."""
 import os
-import sys
 
 from packaging.command.upload import upload
 from packaging.dist import Distribution
@@ -103,22 +102,23 @@
         command, pyversion, filename = 'xxx', '3.3', path
         dist_files = [(command, pyversion, filename)]
 
-        # lets run it
-        pkg_dir, dist = self.create_dist(dist_files=dist_files, author='d?d?')
+        # let's run it
+        dist = self.create_dist(dist_files=dist_files, author='d?d?')[1]
         cmd = upload(dist)
         cmd.ensure_finalized()
         cmd.repository = self.pypi.full_address
         cmd.run()
 
-        # what did we send ?
+        # what did we send?
         handler, request_data = self.pypi.requests[-1]
         headers = handler.headers
-        #self.assertIn('d?d?', str(request_data))
+        self.assertIn('d?d?'.encode('utf-8'), request_data)
         self.assertIn(b'xxx', request_data)
 
         self.assertEqual(int(headers['content-length']), len(request_data))
         self.assertLess(int(headers['content-length']), 2500)
-        self.assertTrue(headers['content-type'].startswith('multipart/form-data'))
+        self.assertTrue(headers['content-type'].startswith(
+            'multipart/form-data'))
         self.assertEqual(handler.command, 'POST')
         self.assertNotIn('\n', headers['authorization'])
 
@@ -132,20 +132,16 @@
         self.write_file(os.path.join(docs_path, "index.html"), "yellow")
         self.write_file(self.rc, PYPIRC)
 
-        # lets run it
-        pkg_dir, dist = self.create_dist(dist_files=dist_files, author='d?d?')
+        # let's run it
+        dist = self.create_dist(dist_files=dist_files, author='d?d?')[1]
 
         cmd = upload(dist)
         cmd.get_finalized_command("build").run()
         cmd.upload_docs = True
         cmd.ensure_finalized()
         cmd.repository = self.pypi.full_address
-        prev_dir = os.getcwd()
-        try:
-            os.chdir(self.tmp_dir)
-            cmd.run()
-        finally:
-            os.chdir(prev_dir)
+        os.chdir(self.tmp_dir)
+        cmd.run()
 
         handler, request_data = self.pypi.requests[-1]
         action, name, content = request_data.split(
diff --git a/Lib/packaging/tests/test_command_upload_docs.py b/Lib/packaging/tests/test_command_upload_docs.py
--- a/Lib/packaging/tests/test_command_upload_docs.py
+++ b/Lib/packaging/tests/test_command_upload_docs.py
@@ -1,6 +1,5 @@
 """Tests for packaging.command.upload_docs."""
 import os
-import sys
 import shutil
 import zipfile
 try:
@@ -19,7 +18,7 @@
     from packaging.tests.pypi_server import PyPIServerTestCase
 except ImportError:
     threading = None
-    PyPIServerTestCase = object
+    PyPIServerTestCase = unittest.TestCase
 
 
 PYPIRC = """\
@@ -52,34 +51,27 @@
 
     def test_default_uploaddir(self):
         sandbox = self.mkdtemp()
-        previous = os.getcwd()
         os.chdir(sandbox)
-        try:
-            os.mkdir("build")
-            self.prepare_sample_dir("build")
-            self.cmd.ensure_finalized()
-            self.assertEqual(self.cmd.upload_dir, os.path.join("build", "docs"))
-        finally:
-            os.chdir(previous)
+        os.mkdir("build")
+        self.prepare_sample_dir("build")
+        self.cmd.ensure_finalized()
+        self.assertEqual(self.cmd.upload_dir, os.path.join("build", "docs"))
 
     def test_default_uploaddir_looks_for_doc_also(self):
         sandbox = self.mkdtemp()
-        previous = os.getcwd()
         os.chdir(sandbox)
-        try:
-            os.mkdir("build")
-            self.prepare_sample_dir("build")
-            os.rename(os.path.join("build", "docs"), os.path.join("build", "doc"))
-            self.cmd.ensure_finalized()
-            self.assertEqual(self.cmd.upload_dir, os.path.join("build", "doc"))
-        finally:
-            os.chdir(previous)
+        os.mkdir("build")
+        self.prepare_sample_dir("build")
+        os.rename(os.path.join("build", "docs"), os.path.join("build", "doc"))
+        self.cmd.ensure_finalized()
+        self.assertEqual(self.cmd.upload_dir, os.path.join("build", "doc"))
 
     def prepare_sample_dir(self, sample_dir=None):
         if sample_dir is None:
             sample_dir = self.mkdtemp()
         os.mkdir(os.path.join(sample_dir, "docs"))
-        self.write_file(os.path.join(sample_dir, "docs", "index.html"), "Ce mortel ennui")
+        self.write_file(os.path.join(sample_dir, "docs", "index.html"),
+                        "Ce mortel ennui")
         self.write_file(os.path.join(sample_dir, "index.html"), "Oh la la")
         return sample_dir
 
@@ -108,9 +100,8 @@
         self.assertTrue(handler.headers['content-type']
             .startswith('multipart/form-data;'))
 
-        action, name, version, content =\
-            request_data.split("----------------GHSKFJDLGDS7543FJKLFHRE75642756743254".encode())[1:5]
-
+        action, name, version, content = request_data.split(
+            b'----------------GHSKFJDLGDS7543FJKLFHRE75642756743254')[1:5]
 
         # check that we picked the right chunks
         self.assertIn(b'name=":action"', action)
@@ -126,27 +117,25 @@
 
     @unittest.skipIf(_ssl is None, 'Needs SSL support')
     def test_https_connection(self):
-        https_called = False
-
-        orig_https = upload_docs_mod.http.client.HTTPSConnection
+        self.https_called = False
+        self.addCleanup(
+            setattr, upload_docs_mod.http.client, 'HTTPSConnection',
+            upload_docs_mod.http.client.HTTPSConnection)
 
         def https_conn_wrapper(*args):
-            nonlocal https_called
-            https_called = True
+            self.https_called = True
             # the testing server is http
             return upload_docs_mod.http.client.HTTPConnection(*args)
 
         upload_docs_mod.http.client.HTTPSConnection = https_conn_wrapper
-        try:
-            self.prepare_command()
-            self.cmd.run()
-            self.assertFalse(https_called)
 
-            self.cmd.repository = self.cmd.repository.replace("http", "https")
-            self.cmd.run()
-            self.assertTrue(https_called)
-        finally:
-            upload_docs_mod.http.client.HTTPSConnection = orig_https
+        self.prepare_command()
+        self.cmd.run()
+        self.assertFalse(self.https_called)
+
+        self.cmd.repository = self.cmd.repository.replace("http", "https")
+        self.cmd.run()
+        self.assertTrue(self.https_called)
 
     def test_handling_response(self):
         self.pypi.default_response_status = '403 Forbidden'
@@ -155,7 +144,8 @@
         self.assertIn('Upload failed (403): Forbidden', self.get_logs()[-1])
 
         self.pypi.default_response_status = '301 Moved Permanently'
-        self.pypi.default_response_headers.append(("Location", "brand_new_location"))
+        self.pypi.default_response_headers.append(
+            ("Location", "brand_new_location"))
         self.cmd.run()
         self.assertIn('brand_new_location', self.get_logs()[-1])
 
@@ -185,6 +175,7 @@
         self.assertTrue(record, "should report the response")
         self.assertIn(self.pypi.default_response_data, record)
 
+
 def test_suite():
     return unittest.makeSuite(UploadDocsTestCase)
 
diff --git a/Lib/packaging/tests/test_create.py b/Lib/packaging/tests/test_create.py
--- a/Lib/packaging/tests/test_create.py
+++ b/Lib/packaging/tests/test_create.py
@@ -1,8 +1,8 @@
 """Tests for packaging.create."""
-import io
 import os
 import sys
 import sysconfig
+from io import StringIO
 from textwrap import dedent
 from packaging.create import MainProgram, ask_yn, ask, main
 
@@ -20,8 +20,8 @@
         super(CreateTestCase, self).setUp()
         self._stdin = sys.stdin  # TODO use Inputs
         self._stdout = sys.stdout
-        sys.stdin = io.StringIO()
-        sys.stdout = io.StringIO()
+        sys.stdin = StringIO()
+        sys.stdout = StringIO()
         self._cwd = os.getcwd()
         self.wdir = self.mkdtemp()
         os.chdir(self.wdir)
@@ -135,7 +135,8 @@
         sys.stdin.seek(0)
         main()
 
-        with open(os.path.join(self.wdir, 'setup.cfg'), encoding='utf-8') as fp:
+        path = os.path.join(self.wdir, 'setup.cfg')
+        with open(path, encoding='utf-8') as fp:
             contents = fp.read()
 
         self.assertEqual(contents, dedent("""\
@@ -210,7 +211,9 @@
         sys.stdin.seek(0)
         # FIXME Out of memory error.
         main()
-        with open(os.path.join(self.wdir, 'setup.cfg'), encoding='utf-8') as fp:
+
+        path = os.path.join(self.wdir, 'setup.cfg')
+        with open(path, encoding='utf-8') as fp:
             contents = fp.read()
 
         self.assertEqual(contents, dedent("""\
diff --git a/Lib/packaging/tests/test_database.py b/Lib/packaging/tests/test_database.py
--- a/Lib/packaging/tests/test_database.py
+++ b/Lib/packaging/tests/test_database.py
@@ -49,8 +49,8 @@
         # distributions
         tmpdir = tempfile.mkdtemp()
         self.addCleanup(shutil.rmtree, tmpdir)
-        self.fake_dists_path = os.path.join(tmpdir, 'fake_dists')
-        self.fake_dists_path = os.path.realpath(self.fake_dists_path)
+        self.fake_dists_path = os.path.realpath(
+            os.path.join(tmpdir, 'fake_dists'))
         fake_dists_src = os.path.abspath(
             os.path.join(os.path.dirname(__file__), 'fake_dists'))
         shutil.copytree(fake_dists_src, self.fake_dists_path)
@@ -58,6 +58,7 @@
         # back (to avoid getting a read-only copy of a read-only file).  we
         # could pass a custom copy_function to change the mode of files, but
         # shutil gives no control over the mode of directories :(
+        # see http://bugs.python.org/issue1666318
         for root, dirs, files in os.walk(self.fake_dists_path):
             os.chmod(root, 0o755)
             for f in files:
diff --git a/Lib/packaging/tests/test_depgraph.py b/Lib/packaging/tests/test_depgraph.py
--- a/Lib/packaging/tests/test_depgraph.py
+++ b/Lib/packaging/tests/test_depgraph.py
@@ -1,10 +1,11 @@
 """Tests for packaging.depgraph """
-import io
 import os
 import re
 import sys
-import packaging.database
+from io import StringIO
+
 from packaging import depgraph
+from packaging.database import get_distribution, enable_cache, disable_cache
 
 from packaging.tests import unittest, support
 from packaging.tests.support import requires_zlib
@@ -30,13 +31,13 @@
         path = os.path.abspath(path)
         sys.path.insert(0, path)
         self.addCleanup(sys.path.remove, path)
-        self.addCleanup(packaging.database.enable_cache)
-        packaging.database.disable_cache()
+        self.addCleanup(enable_cache)
+        disable_cache()
 
     def test_generate_graph(self):
         dists = []
         for name in self.DISTROS_DIST:
-            dist = packaging.database.get_distribution(name)
+            dist = get_distribution(name)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
@@ -61,7 +62,7 @@
     def test_generate_graph_egg(self):
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG:
-            dist = packaging.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
@@ -104,7 +105,7 @@
     def test_dependent_dists(self):
         dists = []
         for name in self.DISTROS_DIST:
-            dist = packaging.database.get_distribution(name)
+            dist = get_distribution(name)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
@@ -123,7 +124,7 @@
     def test_dependent_dists_egg(self):
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG:
-            dist = packaging.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
@@ -158,12 +159,12 @@
 
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG:
-            dist = packaging.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
         graph = depgraph.generate_graph(dists)
-        buf = io.StringIO()
+        buf = StringIO()
         depgraph.graph_to_dot(graph, buf)
         buf.seek(0)
         matches = []
@@ -189,12 +190,12 @@
 
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG:
-            dist = packaging.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
         graph = depgraph.generate_graph(dists)
-        buf = io.StringIO()
+        buf = StringIO()
         depgraph.graph_to_dot(graph, buf, skip_disconnected=False)
         buf.seek(0)
         lines = buf.readlines()
@@ -250,12 +251,12 @@
 
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG + self.BAD_EGGS:
-            dist = packaging.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
         graph = depgraph.generate_graph(dists)
-        buf = io.StringIO()
+        buf = StringIO()
         depgraph.graph_to_dot(graph, buf)
         buf.seek(0)
         matches = []
@@ -273,7 +274,7 @@
     def test_repr(self):
         dists = []
         for name in self.DISTROS_DIST + self.DISTROS_EGG + self.BAD_EGGS:
-            dist = packaging.database.get_distribution(name, use_egg_info=True)
+            dist = get_distribution(name, use_egg_info=True)
             self.assertNotEqual(dist, None)
             dists.append(dist)
 
@@ -282,7 +283,7 @@
 
     @requires_zlib
     def test_main(self):
-        tempout = io.StringIO()
+        tempout = StringIO()
         old = sys.stdout
         sys.stdout = tempout
         oldargv = sys.argv[:]
diff --git a/Lib/packaging/tests/test_dist.py b/Lib/packaging/tests/test_dist.py
--- a/Lib/packaging/tests/test_dist.py
+++ b/Lib/packaging/tests/test_dist.py
@@ -3,13 +3,14 @@
 import sys
 import logging
 import textwrap
+
 import packaging.dist
 
 from packaging.dist import Distribution
 from packaging.command import set_command
 from packaging.command.cmd import Command
 from packaging.errors import PackagingModuleError, PackagingOptionError
-from packaging.tests import TESTFN, captured_stdout
+from packaging.tests import captured_stdout
 from packaging.tests import support, unittest
 from packaging.tests.support import create_distribution
 from test.support import unload
@@ -48,12 +49,13 @@
 
     @unittest.skip('needs to be updated')
     def test_debug_mode(self):
-        self.addCleanup(os.unlink, TESTFN)
-        with open(TESTFN, "w") as f:
+        tmpdir = self.mkdtemp()
+        setupcfg = os.path.join(tmpdir, 'setup.cfg')
+        with open(setupcfg, "w") as f:
             f.write("[global]\n")
             f.write("command_packages = foo.bar, splat")
 
-        files = [TESTFN]
+        files = [setupcfg]
         sys.argv.append("build")
         __, stdout = captured_stdout(create_distribution, files)
         self.assertEqual(stdout, '')
@@ -70,7 +72,7 @@
         Distribution(attrs={'author': 'xxx',
                             'name': 'xxx',
                             'version': '1.2',
-                            'home-page': 'xxxx',
+                            'home_page': 'xxxx',
                             'badoptname': 'xxx'})
         logs = self.get_logs(logging.WARNING)
         self.assertEqual(len(logs), 1)
@@ -80,7 +82,7 @@
         # an empty options dictionary should not stay in the
         # list of attributes
         dist = Distribution(attrs={'author': 'xxx', 'name': 'xxx',
-                                   'version': '1.2', 'home-page': 'xxxx',
+                                   'version': '1.2', 'home_page': 'xxxx',
                                    'options': {}})
 
         self.assertEqual([], self.get_logs(logging.WARNING))
@@ -97,7 +99,7 @@
         dist = Distribution(attrs={'author': 'xxx',
                                    'name': 'xxx',
                                    'version': 'xxx',
-                                   'home-page': 'xxxx',
+                                   'home_page': 'xxxx',
                                    'options': {'sdist': {'owner': 'root'}}})
 
         self.assertIn('owner', dist.get_option_dict('sdist'))
diff --git a/Lib/packaging/tests/test_install.py b/Lib/packaging/tests/test_install.py
--- a/Lib/packaging/tests/test_install.py
+++ b/Lib/packaging/tests/test_install.py
@@ -1,8 +1,8 @@
 """Tests for the packaging.install module."""
 import os
 import logging
+from tempfile import mkstemp
 from sysconfig import is_python_build
-from tempfile import mkstemp
 
 from packaging import install
 from packaging.pypi.xmlrpc import Client
diff --git a/Lib/packaging/tests/test_metadata.py b/Lib/packaging/tests/test_metadata.py
--- a/Lib/packaging/tests/test_metadata.py
+++ b/Lib/packaging/tests/test_metadata.py
@@ -101,7 +101,7 @@
 
         # XXX caveat: the keys method and friends are not 3.x-style views
         # should be changed or documented
-        self.assertEqual(list(metadata), list(metadata.keys()))
+        self.assertEqual(list(metadata), metadata.keys())
 
     def test_read_metadata(self):
         fields = {'name': 'project',
@@ -301,7 +301,7 @@
         Metadata(mapping={'author': 'xxx',
                           'name': 'xxx',
                           'version': 'xxx',
-                          'home-page': 'xxxx'})
+                          'home_page': 'xxxx'})
         logs = self.get_logs(logging.WARNING)
         self.assertEqual(1, len(logs))
         self.assertIn('not a valid version', logs[0])
diff --git a/Lib/packaging/tests/test_mixin2to3.py b/Lib/packaging/tests/test_mixin2to3.py
--- a/Lib/packaging/tests/test_mixin2to3.py
+++ b/Lib/packaging/tests/test_mixin2to3.py
@@ -9,7 +9,6 @@
                         support.LoggingCatcher,
                         unittest.TestCase):
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_convert_code_only(self):
         # used to check if code gets converted properly.
         code = "print 'test'"
@@ -26,7 +25,6 @@
 
         self.assertEqual(expected, converted)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_doctests_only(self):
         # used to check if doctests gets converted properly.
         doctest = textwrap.dedent('''\
@@ -57,7 +55,6 @@
 
         self.assertEqual(expected, converted)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_additional_fixers(self):
         # used to check if use_2to3_fixers works
         code = 'type(x) is not T'
diff --git a/Lib/packaging/tests/test_pypi_server.py b/Lib/packaging/tests/test_pypi_server.py
--- a/Lib/packaging/tests/test_pypi_server.py
+++ b/Lib/packaging/tests/test_pypi_server.py
@@ -1,13 +1,12 @@
 """Tests for packaging.command.bdist."""
-import sys
-
 import urllib.request
 import urllib.parse
 import urllib.error
 
 try:
     import threading
-    from packaging.tests.pypi_server import PyPIServer, PYPI_DEFAULT_STATIC_PATH
+    from packaging.tests.pypi_server import (
+        PyPIServer, PYPI_DEFAULT_STATIC_PATH)
 except ImportError:
     threading = None
     PyPIServer = None
@@ -32,18 +31,19 @@
 
             headers = {"X-test-header": "Mister Iceberg"}
 
-            request = urllib.request.Request(server.full_address, data, headers)
+            request = urllib.request.Request(
+                server.full_address, data, headers)
             urllib.request.urlopen(request)
             self.assertEqual(len(server.requests), 1)
             handler, request_data = server.requests[-1]
             self.assertIn(data, request_data)
             self.assertIn("x-test-header", handler.headers)
-            self.assertEqual(handler.headers["x-test-header"], "Mister Iceberg")
+            self.assertEqual(handler.headers["x-test-header"],
+                             "Mister Iceberg")
 
         finally:
             server.stop()
 
-
     def test_serve_static_content(self):
         # PYPI Mocked server can serve static content from disk.
 
@@ -74,7 +74,8 @@
             self.assertTrue(uses_local_files_for(server, "/simple/index.html"))
 
             # and another one in another root path
-            self.assertTrue(uses_local_files_for(server, "/external/index.html"))
+            self.assertTrue(uses_local_files_for(server,
+                                                 "/external/index.html"))
 
         finally:
             server.stop()
diff --git a/Lib/packaging/tests/test_run.py b/Lib/packaging/tests/test_run.py
--- a/Lib/packaging/tests/test_run.py
+++ b/Lib/packaging/tests/test_run.py
@@ -2,11 +2,10 @@
 
 import os
 import sys
-import shutil
 from io import StringIO
 
 from packaging import install
-from packaging.tests import unittest, support, TESTFN
+from packaging.tests import unittest, support
 from packaging.run import main
 
 from test.script_helper import assert_python_ok
@@ -35,28 +34,14 @@
     def setUp(self):
         super(RunTestCase, self).setUp()
         self.old_stdout = sys.stdout
-        self.cleanup_testfn()
         self.old_argv = sys.argv, sys.argv[:]
 
     def tearDown(self):
         sys.stdout = self.old_stdout
-        self.cleanup_testfn()
         sys.argv = self.old_argv[0]
         sys.argv[:] = self.old_argv[1]
         super(RunTestCase, self).tearDown()
 
-    def cleanup_testfn(self):
-        path = TESTFN
-        if os.path.isfile(path):
-            os.remove(path)
-        elif os.path.isdir(path):
-            shutil.rmtree(path)
-
-    def write_setup(self, text, path=TESTFN):
-        with open(path, "w") as fp:
-            fp.write(text)
-        return path
-
     # TODO restore the tests removed six months ago and port them to pysetup
 
     def test_install(self):
diff --git a/Lib/packaging/tests/test_util.py b/Lib/packaging/tests/test_util.py
--- a/Lib/packaging/tests/test_util.py
+++ b/Lib/packaging/tests/test_util.py
@@ -15,7 +15,7 @@
 from packaging import util
 from packaging.dist import Distribution
 from packaging.util import (
-    convert_path, change_root, split_quoted, strtobool,
+    convert_path, change_root, split_quoted, strtobool, run_2to3,
     get_compiler_versions, _MAC_OS_X_LD_VERSION, byte_compile, find_packages,
     spawn, get_pypirc_path, generate_pypirc, read_pypirc, resolve_name, iglob,
     RICH_GLOB, egginfo_to_distinfo, is_setuptools, is_distutils, is_packaging,
@@ -319,8 +319,6 @@
         res = get_compiler_versions()
         self.assertEqual(res[2], None)
 
-    @unittest.skipUnless(hasattr(sys, 'dont_write_bytecode'),
-                         'sys.dont_write_bytecode not supported')
     def test_dont_write_bytecode(self):
         # makes sure byte_compile raise a PackagingError
         # if sys.dont_write_bytecode is True
@@ -374,7 +372,7 @@
 
         res = find_packages([root], ['pkg1.pkg2'])
         self.assertEqual(set(res), set(['pkg1', 'pkg5', 'pkg1.pkg3',
-                                         'pkg1.pkg3.pkg6']))
+                                        'pkg1.pkg3.pkg6']))
 
     def test_resolve_name(self):
         self.assertIs(str, resolve_name('builtins.str'))
@@ -407,7 +405,6 @@
         finally:
             sys.path.remove(tmp_dir)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_run_2to3_on_code(self):
         content = "print 'test'"
         converted_content = "print('test')"
@@ -416,13 +413,11 @@
         file_handle.write(content)
         file_handle.flush()
         file_handle.seek(0)
-        from packaging.util import run_2to3
         run_2to3([file_name])
         new_content = "".join(file_handle.read())
         file_handle.close()
         self.assertEqual(new_content, converted_content)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_run_2to3_on_doctests(self):
         # to check if text files containing doctests only get converted.
         content = ">>> print 'test'\ntest\n"
@@ -432,7 +427,6 @@
         file_handle.write(content)
         file_handle.flush()
         file_handle.seek(0)
-        from packaging.util import run_2to3
         run_2to3([file_name], doctests_only=True)
         new_content = "".join(file_handle.readlines())
         file_handle.close()
diff --git a/Lib/packaging/util.py b/Lib/packaging/util.py
--- a/Lib/packaging/util.py
+++ b/Lib/packaging/util.py
@@ -326,7 +326,7 @@
     """
     # nothing is done if sys.dont_write_bytecode is True
     # FIXME this should not raise an error
-    if hasattr(sys, 'dont_write_bytecode') and sys.dont_write_bytecode:
+    if sys.dont_write_bytecode:
         raise PackagingByteCompileError('byte-compiling is disabled.')
 
     # First, if the caller didn't force us into direct or indirect mode,
@@ -346,8 +346,10 @@
     # run it with the appropriate flags.
     if not direct:
         from tempfile import mkstemp
-        # XXX script_fd may leak, use something better than mkstemp
+        # XXX use something better than mkstemp
         script_fd, script_name = mkstemp(".py")
+        os.close(script_fd)
+        script_fd = None
         logger.info("writing byte-compilation script '%s'", script_name)
         if not dry_run:
             if script_fd is not None:
diff --git a/Lib/pkgutil.py b/Lib/pkgutil.py
--- a/Lib/pkgutil.py
+++ b/Lib/pkgutil.py
@@ -307,9 +307,9 @@
     def get_filename(self, fullname=None):
         fullname = self._fix_name(fullname)
         mod_type = self.etc[2]
-        if self.etc[2]==imp.PKG_DIRECTORY:
+        if mod_type==imp.PKG_DIRECTORY:
             return self._get_delegate().get_filename()
-        elif self.etc[2] in (imp.PY_SOURCE, imp.PY_COMPILED, imp.C_EXTENSION):
+        elif mod_type in (imp.PY_SOURCE, imp.PY_COMPILED, imp.C_EXTENSION):
             return self.filename
         return None
 
diff --git a/Lib/shutil.py b/Lib/shutil.py
--- a/Lib/shutil.py
+++ b/Lib/shutil.py
@@ -391,7 +391,7 @@
         compress_ext['bzip2'] = '.bz2'
 
     # flags for compression program, each element of list will be an argument
-    if compress is not None and compress not in compress_ext.keys():
+    if compress is not None and compress not in compress_ext:
         raise ValueError("bad value for 'compress', or compression format not "
                          "supported : {0}".format(compress))
 
@@ -497,7 +497,7 @@
     'gztar': (_make_tarball, [('compress', 'gzip')], "gzip'ed tar-file"),
     'bztar': (_make_tarball, [('compress', 'bzip2')], "bzip2'ed tar-file"),
     'tar':   (_make_tarball, [('compress', None)], "uncompressed tar file"),
-    'zip':   (_make_zipfile, [],"ZIP file")
+    'zip':   (_make_zipfile, [], "ZIP file")
     }
 
 if _BZ2_SUPPORTED:
@@ -530,7 +530,7 @@
     if not isinstance(extra_args, (tuple, list)):
         raise TypeError('extra_args needs to be a sequence')
     for element in extra_args:
-        if not isinstance(element, (tuple, list)) or len(element) !=2 :
+        if not isinstance(element, (tuple, list)) or len(element) !=2:
             raise TypeError('extra_args elements are : (arg_name, value)')
 
     _ARCHIVE_FORMATS[name] = (function, extra_args, description)
@@ -682,7 +682,7 @@
             if not name.endswith('/'):
                 # file
                 data = zip.read(info.filename)
-                f = open(target,'wb')
+                f = open(target, 'wb')
                 try:
                     f.write(data)
                 finally:
diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -965,7 +965,9 @@
                  'warnings.filters', 'asyncore.socket_map',
                  'logging._handlers', 'logging._handlerList', 'sys.gettrace',
                  'sys.warnoptions', 'threading._dangling',
-                 'multiprocessing.process._dangling')
+                 'multiprocessing.process._dangling',
+                 'sysconfig._CONFIG_VARS', 'sysconfig._SCHEMES',
+                )
 
     def get_sys_argv(self):
         return id(sys.argv), sys.argv, sys.argv[:]
@@ -1083,6 +1085,27 @@
         multiprocessing.process._dangling.clear()
         multiprocessing.process._dangling.update(saved)
 
+    def get_sysconfig__CONFIG_VARS(self):
+        # make sure the dict is initialized
+        sysconfig.get_config_var('prefix')
+        return (id(sysconfig._CONFIG_VARS), sysconfig._CONFIG_VARS,
+                dict(sysconfig._CONFIG_VARS))
+    def restore_sysconfig__CONFIG_VARS(self, saved):
+        sysconfig._CONFIG_VARS = saved[1]
+        sysconfig._CONFIG_VARS.clear()
+        sysconfig._CONFIG_VARS.update(saved[2])
+
+    def get_sysconfig__SCHEMES(self):
+        # it's mildly evil to look at the internal attribute, but it's easier
+        # than copying a RawConfigParser object
+        return (id(sysconfig._SCHEMES), sysconfig._SCHEMES._sections,
+                sysconfig._SCHEMES._sections.copy())
+    def restore_sysconfig__SCHEMES(self, saved):
+        sysconfig._SCHEMES._sections = saved[1]
+        sysconfig._SCHEMES._sections.clear()
+        sysconfig._SCHEMES._sections.update(saved[2])
+
+
     def resource_info(self):
         for name in self.resources:
             method_suffix = name.replace('.', '_')
diff --git a/Lib/test/test_sysconfig.py b/Lib/test/test_sysconfig.py
--- a/Lib/test/test_sysconfig.py
+++ b/Lib/test/test_sysconfig.py
@@ -39,7 +39,7 @@
         self._config_vars = copy(sysconfig._CONFIG_VARS)
         self._added_envvars = []
         self._changed_envvars = []
-        for var in ('MACOSX_DEPLOYMENT_TARGET', 'Path'):
+        for var in ('MACOSX_DEPLOYMENT_TARGET', 'PATH'):
             if var in os.environ:
                 self._changed_envvars.append((var, os.environ[var]))
             else:
@@ -87,21 +87,19 @@
         scheme = get_paths()
         default_scheme = _get_default_scheme()
         wanted = _expand_vars(default_scheme, None)
-        wanted = list(wanted.items())
-        wanted.sort()
-        scheme = list(scheme.items())
-        scheme.sort()
+        wanted = sorted(wanted.items())
+        scheme = sorted(scheme.items())
         self.assertEqual(scheme, wanted)
 
     def test_get_path(self):
-        # xxx make real tests here
+        # XXX make real tests here
         for scheme in _SCHEMES:
             for name in _SCHEMES[scheme]:
                 res = get_path(name, scheme)
 
     def test_get_config_vars(self):
         cvars = get_config_vars()
-        self.assertTrue(isinstance(cvars, dict))
+        self.assertIsInstance(cvars, dict)
         self.assertTrue(cvars)
 
     def test_get_platform(self):
@@ -236,8 +234,8 @@
         # On Windows, the EXE needs to know where pythonXY.dll is at so we have
         # to add the directory to the path.
         if sys.platform == "win32":
-            os.environ["Path"] = "{};{}".format(
-                os.path.dirname(sys.executable), os.environ["Path"])
+            os.environ["PATH"] = "{};{}".format(
+                os.path.dirname(sys.executable), os.environ["PATH"])
 
         # Issue 7880
         def get(python):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 16:55:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 19 Sep 2011 16:55:07 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Avoid_matching_=27=27_or_?=
 =?utf8?q?=27yn=27_when_asking_for_=27y=27_or_=27n=27_in_interactive_code?=
Message-ID: <E1R5fFT-0001sH-Vf@dinsdale.python.org>

http://hg.python.org/cpython/rev/10fd0d0895a1
changeset:   72412:10fd0d0895a1
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 16:21:37 2011 +0200
summary:
  Avoid matching '' or 'yn' when asking for 'y' or 'n' in interactive code

files:
  Lib/packaging/command/register.py |  2 +-
  Lib/packaging/create.py           |  2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Lib/packaging/command/register.py b/Lib/packaging/command/register.py
--- a/Lib/packaging/command/register.py
+++ b/Lib/packaging/command/register.py
@@ -178,7 +178,7 @@
                         'will be faster.\n(the login will be stored in %s)',
                         get_pypirc_path())
                     choice = 'X'
-                    while choice.lower() not in 'yn':
+                    while choice.lower() not in ('y', 'n'):
                         choice = input('Save your login (y/N)?')
                         if not choice:
                             choice = 'n'
diff --git a/Lib/packaging/create.py b/Lib/packaging/create.py
--- a/Lib/packaging/create.py
+++ b/Lib/packaging/create.py
@@ -121,7 +121,7 @@
     question += ' (y/n)'
     while True:
         answer = ask(question, default, helptext, required=True)
-        if answer and answer[0].lower() in 'yn':
+        if answer and answer[0].lower() in ('y', 'n'):
             return answer[0].lower()
 
         print('\nERROR: You must select "Y" or "N".\n')

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 17:13:28 2011
From: python-checkins at python.org (jesus.cea)
Date: Mon, 19 Sep 2011 17:13:28 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogQ2xvc2UgIzEzMDA3?=
 =?utf8?q?=3A_whichdb_should_recognize_gdbm_1=2E9_magic_numbers?=
Message-ID: <E1R5fXE-00040p-6w@dinsdale.python.org>

http://hg.python.org/cpython/rev/14cafb8d1480
changeset:   72413:14cafb8d1480
branch:      2.7
parent:      72398:9f3595fb6c92
user:        Jesus Cea <jcea at jcea.es>
date:        Mon Sep 19 16:57:18 2011 +0200
summary:
  Close #13007: whichdb should recognize gdbm 1.9 magic numbers

files:
  Lib/whichdb.py |  2 +-
  Misc/NEWS      |  2 ++
  2 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/Lib/whichdb.py b/Lib/whichdb.py
--- a/Lib/whichdb.py
+++ b/Lib/whichdb.py
@@ -91,7 +91,7 @@
         return ""
 
     # Check for GNU dbm
-    if magic == 0x13579ace:
+    if magic in (0x13579ace, 0x13579acd, 0x13579acf):
         return "gdbm"
 
     # Check for old Berkeley db hash file format v2
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -76,6 +76,8 @@
 - Issue #12326: sys.platform is now always 'linux2' on Linux, even if Python
   is compiled on Linux 3.
 
+- Issue #13007: whichdb should recognize gdbm 1.9 magic numbers.
+
 - Issue #9173: Let shutil._make_archive work if the logger argument is None.
 
 - Issue #12650: Fix a race condition where a subprocess.Popen could leak

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 17:13:28 2011
From: python-checkins at python.org (jesus.cea)
Date: Mon, 19 Sep 2011 17:13:28 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogQ2xvc2UgIzEzMDA3?=
 =?utf8?q?=3A_whichdb_should_recognize_gdbm_1=2E9_magic_numbers?=
Message-ID: <E1R5fXE-00040t-UU@dinsdale.python.org>

http://hg.python.org/cpython/rev/7a41855b6196
changeset:   72414:7a41855b6196
branch:      3.2
parent:      72396:5deecc04b7a2
user:        Jesus Cea <jcea at jcea.es>
date:        Mon Sep 19 17:08:18 2011 +0200
summary:
  Close #13007: whichdb should recognize gdbm 1.9 magic numbers

files:
  Lib/dbm/__init__.py |  2 +-
  Misc/NEWS           |  2 ++
  2 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/Lib/dbm/__init__.py b/Lib/dbm/__init__.py
--- a/Lib/dbm/__init__.py
+++ b/Lib/dbm/__init__.py
@@ -166,7 +166,7 @@
         return ""
 
     # Check for GNU dbm
-    if magic == 0x13579ace:
+    if magic in (0x13579ace, 0x13579acd, 0x13579acf):
         return "dbm.gnu"
 
     # Later versions of Berkeley db hash file have a 12-byte pad in
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -55,6 +55,8 @@
 - Issue #11657: Fix sending file descriptors over 255 over a multiprocessing
   Pipe.
 
+- Issue #13007: whichdb should recognize gdbm 1.9 magic numbers.
+
 - Issue #12213: Fix a buffering bug with interleaved reads and writes that
   could appear on BufferedRandom streams.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 17:13:32 2011
From: python-checkins at python.org (jesus.cea)
Date: Mon, 19 Sep 2011 17:13:32 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Close_=2313007=3A_whichdb_should_recognize_gdbm_1=2E9_magic_?=
 =?utf8?q?numbers?=
Message-ID: <E1R5fXI-00040x-9X@dinsdale.python.org>

http://hg.python.org/cpython/rev/b194af345cb5
changeset:   72415:b194af345cb5
parent:      72412:10fd0d0895a1
parent:      72414:7a41855b6196
user:        Jesus Cea <jcea at jcea.es>
date:        Mon Sep 19 17:11:26 2011 +0200
summary:
  Close #13007: whichdb should recognize gdbm 1.9 magic numbers

files:
  Lib/dbm/__init__.py |  2 +-
  Misc/NEWS           |  2 ++
  2 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/Lib/dbm/__init__.py b/Lib/dbm/__init__.py
--- a/Lib/dbm/__init__.py
+++ b/Lib/dbm/__init__.py
@@ -166,7 +166,7 @@
         return ""
 
     # Check for GNU dbm
-    if magic == 0x13579ace:
+    if magic in (0x13579ace, 0x13579acd, 0x13579acf):
         return "dbm.gnu"
 
     # Later versions of Berkeley db hash file have a 12-byte pad in
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -236,6 +236,8 @@
 - Issue #10829: Refactor PyUnicode_FromFormat(), use the same function to parse
   the format string in the 3 steps, fix crashs on invalid format strings.
 
+- Issue #13007: whichdb should recognize gdbm 1.9 magic numbers.
+
 - Issue #11246: Fix PyUnicode_FromFormat("%V") to decode the byte string from
   UTF-8 (with replace error handler) instead of ISO-8859-1 (in strict mode).
   Patch written by Ray Allen.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 17:38:17 2011
From: python-checkins at python.org (mark.dickinson)
Date: Mon, 19 Sep 2011 17:38:17 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyOTcz?=
 =?utf8?q?=3A_Fix_int=2E=5F=5Fpow=5F=5F_overflow_checks_that_invoked_undef?=
 =?utf8?q?ined_behaviour=2C?=
Message-ID: <E1R5fvF-0006nQ-T4@dinsdale.python.org>

http://hg.python.org/cpython/rev/07efe83795b0
changeset:   72416:07efe83795b0
branch:      2.7
parent:      72413:14cafb8d1480
user:        Mark Dickinson <mdickinson at enthought.com>
date:        Mon Sep 19 16:38:08 2011 +0100
summary:
  Issue #12973: Fix int.__pow__ overflow checks that invoked undefined behaviour, thereby producing incorrect results on Clang.

files:
  Misc/NEWS           |   5 +++++
  Objects/intobject.c |  10 ++++++++--
  2 files changed, 13 insertions(+), 2 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -9,6 +9,11 @@
 Core and Builtins
 -----------------
 
+- Issue #12973: Fix overflow checks that invoked undefined behaviour in
+  int.__pow__.  These overflow checks were causing int.__pow__ to produce
+  incorrect results with recent versions of Clang, as a result of the
+  compiler optimizing the check away.
+
 - Issue #12266: Fix str.capitalize() to correctly uppercase/lowercase
   titlecased and cased non-letter characters.
 
diff --git a/Objects/intobject.c b/Objects/intobject.c
--- a/Objects/intobject.c
+++ b/Objects/intobject.c
@@ -751,7 +751,13 @@
     while (iw > 0) {
         prev = ix;              /* Save value for overflow check */
         if (iw & 1) {
-            ix = ix*temp;
+            /*
+             * The (unsigned long) cast below ensures that the multiplication
+             * is interpreted as an unsigned operation rather than a signed one
+             * (C99 6.3.1.8p1), thus avoiding the perils of undefined behaviour
+             * from signed arithmetic overflow (C99 6.5p5).  See issue #12973.
+             */
+            ix = (unsigned long)ix * temp;
             if (temp == 0)
                 break; /* Avoid ix / 0 */
             if (ix / temp != prev) {
@@ -764,7 +770,7 @@
         iw >>= 1;               /* Shift exponent down by 1 bit */
         if (iw==0) break;
         prev = temp;
-        temp *= temp;           /* Square the value of temp */
+        temp = (unsigned long)temp * temp;  /* Square the value of temp */
         if (prev != 0 && temp / prev != prev) {
             return PyLong_Type.tp_as_number->nb_power(
                 (PyObject *)v, (PyObject *)w, (PyObject *)z);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 20:20:34 2011
From: python-checkins at python.org (mark.dickinson)
Date: Mon, 19 Sep 2011 20:20:34 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyOTcz?=
 =?utf8?q?=3A_Fix_undefined-behaviour-inducing_overflow_check_in_list=5Fre?=
 =?utf8?q?peat=2E?=
Message-ID: <E1R5iSI-0007U7-39@dinsdale.python.org>

http://hg.python.org/cpython/rev/99b808c94834
changeset:   72417:99b808c94834
branch:      3.2
parent:      72414:7a41855b6196
user:        Mark Dickinson <mdickinson at enthought.com>
date:        Mon Sep 19 19:18:37 2011 +0100
summary:
  Issue #12973: Fix undefined-behaviour-inducing overflow check in list_repeat.

files:
  Misc/NEWS            |  4 ++++
  Objects/listobject.c |  6 +++---
  2 files changed, 7 insertions(+), 3 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,10 @@
 Core and Builtins
 -----------------
 
+- Issue #12973: Fix overflow check that relied on undefined behaviour in
+  list_repeat.  This bug caused test_list to fail with recent versions
+  of Clang.
+
 - Issue #12802: the Windows error ERROR_DIRECTORY (numbered 267) is now
   mapped to POSIX errno ENOTDIR (previously EINVAL).
 
diff --git a/Objects/listobject.c b/Objects/listobject.c
--- a/Objects/listobject.c
+++ b/Objects/listobject.c
@@ -58,7 +58,7 @@
     if (newsize == 0)
         new_allocated = 0;
     items = self->ob_item;
-    if (new_allocated <= ((~(size_t)0) / sizeof(PyObject *)))
+    if (new_allocated <= (PY_SIZE_MAX / sizeof(PyObject *)))
         PyMem_RESIZE(items, PyObject *, new_allocated);
     else
         items = NULL;
@@ -510,9 +510,9 @@
     PyObject *elem;
     if (n < 0)
         n = 0;
+    if (n > 0 && Py_SIZE(a) > PY_SSIZE_T_MAX / n)
+        return PyErr_NoMemory();
     size = Py_SIZE(a) * n;
-    if (n && size/n != Py_SIZE(a))
-        return PyErr_NoMemory();
     if (size == 0)
         return PyList_New(0);
     np = (PyListObject *) PyList_New(size);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 20:20:34 2011
From: python-checkins at python.org (mark.dickinson)
Date: Mon, 19 Sep 2011 20:20:34 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_issue_=2312973_list=5Frepeat_fix=2E?=
Message-ID: <E1R5iSI-0007UB-Sx@dinsdale.python.org>

http://hg.python.org/cpython/rev/2dbd5870de0b
changeset:   72418:2dbd5870de0b
parent:      72415:b194af345cb5
parent:      72417:99b808c94834
user:        Mark Dickinson <mdickinson at enthought.com>
date:        Mon Sep 19 19:19:50 2011 +0100
summary:
  Merge issue #12973 list_repeat fix.

files:
  Misc/NEWS            |  4 ++++
  Objects/listobject.c |  6 +++---
  2 files changed, 7 insertions(+), 3 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,10 @@
 Core and Builtins
 -----------------
 
+- Issue #12973: Fix overflow check that relied on undefined behaviour in
+  list_repeat.  This bug caused test_list to fail with recent versions
+  of Clang.
+
 - Issue #12904: os.utime, os.futimes, os.lutimes, and os.futimesat now write
   atime and mtime with nanosecond precision on modern POSIX platforms.
 
diff --git a/Objects/listobject.c b/Objects/listobject.c
--- a/Objects/listobject.c
+++ b/Objects/listobject.c
@@ -58,7 +58,7 @@
     if (newsize == 0)
         new_allocated = 0;
     items = self->ob_item;
-    if (new_allocated <= ((~(size_t)0) / sizeof(PyObject *)))
+    if (new_allocated <= (PY_SIZE_MAX / sizeof(PyObject *)))
         PyMem_RESIZE(items, PyObject *, new_allocated);
     else
         items = NULL;
@@ -510,9 +510,9 @@
     PyObject *elem;
     if (n < 0)
         n = 0;
+    if (n > 0 && Py_SIZE(a) > PY_SSIZE_T_MAX / n)
+        return PyErr_NoMemory();
     size = Py_SIZE(a) * n;
-    if (n && size/n != Py_SIZE(a))
-        return PyErr_NoMemory();
     if (size == 0)
         return PyList_New(0);
     np = (PyListObject *) PyList_New(size);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Sep 19 20:24:37 2011
From: python-checkins at python.org (mark.dickinson)
Date: Mon, 19 Sep 2011 20:24:37 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Backport_issue_?=
 =?utf8?q?=2312973_list=5Frepeat_fix_from_3=2Ex=2E?=
Message-ID: <E1R5iWD-0007qo-Vx@dinsdale.python.org>

http://hg.python.org/cpython/rev/f8280cf63d9e
changeset:   72419:f8280cf63d9e
branch:      2.7
parent:      72416:07efe83795b0
user:        Mark Dickinson <mdickinson at enthought.com>
date:        Mon Sep 19 19:23:55 2011 +0100
summary:
  Backport issue #12973 list_repeat fix from 3.x.

files:
  Misc/NEWS            |  4 +++-
  Objects/listobject.c |  6 +++---
  2 files changed, 6 insertions(+), 4 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -12,7 +12,9 @@
 - Issue #12973: Fix overflow checks that invoked undefined behaviour in
   int.__pow__.  These overflow checks were causing int.__pow__ to produce
   incorrect results with recent versions of Clang, as a result of the
-  compiler optimizing the check away.
+  compiler optimizing the check away.  Also fix similar overflow check
+  in list_repeat (which caused test_list to fail with recent versions
+  of Clang).
 
 - Issue #12266: Fix str.capitalize() to correctly uppercase/lowercase
   titlecased and cased non-letter characters.
diff --git a/Objects/listobject.c b/Objects/listobject.c
--- a/Objects/listobject.c
+++ b/Objects/listobject.c
@@ -58,7 +58,7 @@
     if (newsize == 0)
         new_allocated = 0;
     items = self->ob_item;
-    if (new_allocated <= ((~(size_t)0) / sizeof(PyObject *)))
+    if (new_allocated <= (PY_SIZE_MAX / sizeof(PyObject *)))
         PyMem_RESIZE(items, PyObject *, new_allocated);
     else
         items = NULL;
@@ -551,9 +551,9 @@
     PyObject *elem;
     if (n < 0)
         n = 0;
+    if (n > 0 && Py_SIZE(a) > PY_SSIZE_T_MAX / n)
+        return PyErr_NoMemory();
     size = Py_SIZE(a) * n;
-    if (n && size/n != Py_SIZE(a))
-        return PyErr_NoMemory();
     if (size == 0)
         return PyList_New(0);
     np = (PyListObject *) PyList_New(size);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep 20 04:04:41 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 20 Sep 2011 04:04:41 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_os=2Estatvfs=28=29_encodes_?=
 =?utf8?q?the_filename_to_the_filesystem_encoding?=
Message-ID: <E1R5phR-0006Nz-6t@dinsdale.python.org>

http://hg.python.org/cpython/rev/e14e70e1a390
changeset:   72420:e14e70e1a390
parent:      72418:2dbd5870de0b
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Sep 20 04:04:33 2011 +0200
summary:
  os.statvfs() encodes the filename to the filesystem encoding

files:
  Modules/posixmodule.c |  20 ++++++++++++--------
  1 files changed, 12 insertions(+), 8 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -8071,16 +8071,20 @@
 static PyObject *
 posix_statvfs(PyObject *self, PyObject *args)
 {
-    char *path;
+    PyObject *path;
     int res;
     struct statvfs st;
-    if (!PyArg_ParseTuple(args, "s:statvfs", &path))
-        return NULL;
-    Py_BEGIN_ALLOW_THREADS
-    res = statvfs(path, &st);
-    Py_END_ALLOW_THREADS
-    if (res != 0)
-        return posix_error_with_filename(path);
+    if (!PyArg_ParseTuple(args, "O&:statvfs", PyUnicode_FSConverter, &path))
+        return NULL;
+    Py_BEGIN_ALLOW_THREADS
+    res = statvfs(PyBytes_AS_STRING(path), &st);
+    Py_END_ALLOW_THREADS
+    if (res != 0) {
+        posix_error_with_filename(PyBytes_AS_STRING(path));
+        Py_DECREF(path);
+        return NULL;
+    }
+    Py_DECREF(path);
 
     return _pystatvfs_fromstructstatvfs(st);
 }

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Tue Sep 20 05:20:52 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Tue, 20 Sep 2011 05:20:52 +0200
Subject: [Python-checkins] Daily reference leaks (2dbd5870de0b): sum=0
Message-ID: <E1R5qtA-0005V9-QP@ap.vmr.nerim.net>

results for 2dbd5870de0b on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogMUVD6X', '-x']

From python-checkins at python.org  Tue Sep 20 19:25:52 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Tue, 20 Sep 2011 19:25:52 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312996=3A_multiproc?=
 =?utf8?q?essing=2Econnection=3A_transmit_the_header_in_network_byte?=
Message-ID: <E1R644u-0003S6-JK@dinsdale.python.org>

http://hg.python.org/cpython/rev/9c1c81d24e23
changeset:   72421:9c1c81d24e23
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Tue Sep 20 19:27:39 2011 +0200
summary:
  Issue #12996: multiprocessing.connection: transmit the header in network byte
order (endpoints machines can have different endianness).

files:
  Lib/multiprocessing/connection.py |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py
--- a/Lib/multiprocessing/connection.py
+++ b/Lib/multiprocessing/connection.py
@@ -422,7 +422,7 @@
     def _send_bytes(self, buf):
         # For wire compatibility with 3.2 and lower
         n = len(buf)
-        self._send(struct.pack("=i", len(buf)))
+        self._send(struct.pack("!i", n))
         # The condition is necessary to avoid "broken pipe" errors
         # when sending a 0-length buffer if the other end closed the pipe.
         if n > 0:
@@ -430,7 +430,7 @@
 
     def _recv_bytes(self, maxsize=None, sentinels=()):
         buf = self._recv(4, sentinels)
-        size, = struct.unpack("=i", buf.getvalue())
+        size, = struct.unpack("!i", buf.getvalue())
         if maxsize is not None and size > maxsize:
             return None
         return self._recv(size, sentinels)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep 20 20:34:55 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Tue, 20 Sep 2011 20:34:55 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312981=3A_test=5Fmu?=
 =?utf8?q?ltiprocessing=3A_catch_ImportError_when_importing?=
Message-ID: <E1R659j-0002Jj-T7@dinsdale.python.org>

http://hg.python.org/cpython/rev/1d91a3ba5c87
changeset:   72422:1d91a3ba5c87
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Tue Sep 20 20:36:51 2011 +0200
summary:
  Issue #12981: test_multiprocessing: catch ImportError when importing
multiprocessing.reduction, which may not be available (e.g. if the OS doesn't
support FD passing over Unix domain sockets).

files:
  Lib/test/test_multiprocessing.py |  14 ++++++++++++--
  1 files changed, 12 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -35,7 +35,13 @@
 import multiprocessing.heap
 import multiprocessing.pool
 
-from multiprocessing import util, reduction
+from multiprocessing import util
+
+try:
+    from multiprocessing import reduction
+    HAS_REDUCTION = True
+except ImportError:
+    HAS_REDUCTION = False
 
 try:
     from multiprocessing.sharedctypes import Value, copy
@@ -1631,6 +1637,7 @@
         os.write(fd, data)
         os.close(fd)
 
+    @unittest.skipUnless(HAS_REDUCTION, "test needs multiprocessing.reduction")
     def test_fd_transfer(self):
         if self.TYPE != 'processes':
             self.skipTest("only makes sense with processes")
@@ -1648,6 +1655,7 @@
         with open(test.support.TESTFN, "rb") as f:
             self.assertEqual(f.read(), b"foo")
 
+    @unittest.skipUnless(HAS_REDUCTION, "test needs multiprocessing.reduction")
     @unittest.skipIf(sys.platform == "win32",
                      "test semantics don't make sense on Windows")
     @unittest.skipIf(MAXFD <= 256,
@@ -1987,10 +1995,12 @@
             'multiprocessing', 'multiprocessing.connection',
             'multiprocessing.heap', 'multiprocessing.managers',
             'multiprocessing.pool', 'multiprocessing.process',
-            'multiprocessing.reduction',
             'multiprocessing.synchronize', 'multiprocessing.util'
             ]
 
+        if HAS_REDUCTION:
+            modules.append('multiprocessing.reduction')
+
         if c_int is not None:
             # This module requires _ctypes
             modules.append('multiprocessing.sharedctypes')

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep 20 20:58:40 2011
From: python-checkins at python.org (barry.warsaw)
Date: Tue, 20 Sep 2011 20:58:40 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_-_Issue_=231302?=
 =?utf8?q?1=3A_Missing_decref_on_an_error_path=2E__Thanks_to_Suman_Saha_fo?=
 =?utf8?q?r?=
Message-ID: <E1R65Wi-0004qy-Sq@dinsdale.python.org>

http://hg.python.org/cpython/rev/7a48e98915f2
changeset:   72423:7a48e98915f2
branch:      3.2
parent:      72417:99b808c94834
user:        Barry Warsaw <barry at python.org>
date:        Tue Sep 20 14:45:44 2011 -0400
summary:
  - Issue #13021: Missing decref on an error path.  Thanks to Suman Saha for
  finding the bug and providing a patch.

files:
  Misc/NEWS          |  3 +++
  Python/pythonrun.c |  4 +++-
  2 files changed, 6 insertions(+), 1 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #13021: Missing decref on an error path.  Thanks to Suman Saha for
+  finding the bug and providing a patch.
+
 - Issue #12973: Fix overflow check that relied on undefined behaviour in
   list_repeat.  This bug caused test_list to fail with recent versions
   of Clang.
diff --git a/Python/pythonrun.c b/Python/pythonrun.c
--- a/Python/pythonrun.c
+++ b/Python/pythonrun.c
@@ -1241,8 +1241,10 @@
             Py_DECREF(f);
             return -1;
         }
-        if (PyDict_SetItemString(d, "__cached__", Py_None) < 0)
+        if (PyDict_SetItemString(d, "__cached__", Py_None) < 0) {
+            Py_DECREF(f);
             return -1;
+        }
         set_file_name = 1;
         Py_DECREF(f);
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep 20 20:58:41 2011
From: python-checkins at python.org (barry.warsaw)
Date: Tue, 20 Sep 2011 20:58:41 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_-_Issue_=2313021=3A_Missing_decref_on_an_error_path=2E__Than?=
 =?utf8?q?ks_to_Suman_Saha_for?=
Message-ID: <E1R65Wj-0004r2-ON@dinsdale.python.org>

http://hg.python.org/cpython/rev/ad0804d3dbd0
changeset:   72424:ad0804d3dbd0
parent:      72421:9c1c81d24e23
parent:      72423:7a48e98915f2
user:        Barry Warsaw <barry at python.org>
date:        Tue Sep 20 14:58:01 2011 -0400
summary:
  - Issue #13021: Missing decref on an error path.  Thanks to Suman Saha for
  finding the bug and providing a patch.

files:
  Misc/NEWS          |  3 +++
  Python/pythonrun.c |  4 +++-
  2 files changed, 6 insertions(+), 1 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #13021: Missing decref on an error path.  Thanks to Suman Saha for
+  finding the bug and providing a patch.
+
 - Issue #12973: Fix overflow check that relied on undefined behaviour in
   list_repeat.  This bug caused test_list to fail with recent versions
   of Clang.
diff --git a/Python/pythonrun.c b/Python/pythonrun.c
--- a/Python/pythonrun.c
+++ b/Python/pythonrun.c
@@ -1251,8 +1251,10 @@
             Py_DECREF(f);
             return -1;
         }
-        if (PyDict_SetItemString(d, "__cached__", Py_None) < 0)
+        if (PyDict_SetItemString(d, "__cached__", Py_None) < 0) {
+            Py_DECREF(f);
             return -1;
+        }
         set_file_name = 1;
         Py_DECREF(f);
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Sep 20 20:58:42 2011
From: python-checkins at python.org (barry.warsaw)
Date: Tue, 20 Sep 2011 20:58:42 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Head_merge?=
Message-ID: <E1R65Wk-0004r6-Fu@dinsdale.python.org>

http://hg.python.org/cpython/rev/016543a9333a
changeset:   72425:016543a9333a
parent:      72424:ad0804d3dbd0
parent:      72422:1d91a3ba5c87
user:        Barry Warsaw <barry at python.org>
date:        Tue Sep 20 14:58:19 2011 -0400
summary:
  Head merge

files:
  Lib/test/test_multiprocessing.py |  14 ++++++++++++--
  1 files changed, 12 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -35,7 +35,13 @@
 import multiprocessing.heap
 import multiprocessing.pool
 
-from multiprocessing import util, reduction
+from multiprocessing import util
+
+try:
+    from multiprocessing import reduction
+    HAS_REDUCTION = True
+except ImportError:
+    HAS_REDUCTION = False
 
 try:
     from multiprocessing.sharedctypes import Value, copy
@@ -1631,6 +1637,7 @@
         os.write(fd, data)
         os.close(fd)
 
+    @unittest.skipUnless(HAS_REDUCTION, "test needs multiprocessing.reduction")
     def test_fd_transfer(self):
         if self.TYPE != 'processes':
             self.skipTest("only makes sense with processes")
@@ -1648,6 +1655,7 @@
         with open(test.support.TESTFN, "rb") as f:
             self.assertEqual(f.read(), b"foo")
 
+    @unittest.skipUnless(HAS_REDUCTION, "test needs multiprocessing.reduction")
     @unittest.skipIf(sys.platform == "win32",
                      "test semantics don't make sense on Windows")
     @unittest.skipIf(MAXFD <= 256,
@@ -1987,10 +1995,12 @@
             'multiprocessing', 'multiprocessing.connection',
             'multiprocessing.heap', 'multiprocessing.managers',
             'multiprocessing.pool', 'multiprocessing.process',
-            'multiprocessing.reduction',
             'multiprocessing.synchronize', 'multiprocessing.util'
             ]
 
+        if HAS_REDUCTION:
+            modules.append('multiprocessing.reduction')
+
         if c_int is not None:
             # This module requires _ctypes
             modules.append('multiprocessing.sharedctypes')

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 01:09:55 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 21 Sep 2011 01:09:55 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_test=5Fhttplib_?=
	=?utf8?q?removes_temporary_files?=
Message-ID: <E1R69Rr-0004Kk-Uw@dinsdale.python.org>

http://hg.python.org/cpython/rev/950515bff239
changeset:   72426:950515bff239
branch:      3.2
parent:      72423:7a48e98915f2
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 21 01:09:04 2011 +0200
summary:
  test_httplib removes temporary files

files:
  Lib/test/test_httplib.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py
--- a/Lib/test/test_httplib.py
+++ b/Lib/test/test_httplib.py
@@ -588,6 +588,7 @@
         self.assertEqual(b'body\xc1', f.read())
 
     def test_file_body(self):
+        self.addCleanup(support.unlink, support.TESTFN)
         with open(support.TESTFN, "w") as f:
             f.write("body")
         with open(support.TESTFN) as f:
@@ -599,6 +600,7 @@
             self.assertEqual(b'body', f.read())
 
     def test_binary_file_body(self):
+        self.addCleanup(support.unlink, support.TESTFN)
         with open(support.TESTFN, "wb") as f:
             f.write(b"body\xc1")
         with open(support.TESTFN, "rb") as f:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 01:09:56 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 21 Sep 2011 01:09:56 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_3=2E2=3A_test=5Fhttplib_removes_temporary_files?=
Message-ID: <E1R69Rs-0004Ku-OT@dinsdale.python.org>

http://hg.python.org/cpython/rev/1d71aa589549
changeset:   72427:1d71aa589549
parent:      72425:016543a9333a
parent:      72426:950515bff239
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 21 01:09:29 2011 +0200
summary:
  Merge 3.2: test_httplib removes temporary files

files:
  Lib/test/test_httplib.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py
--- a/Lib/test/test_httplib.py
+++ b/Lib/test/test_httplib.py
@@ -588,6 +588,7 @@
         self.assertEqual(b'body\xc1', f.read())
 
     def test_file_body(self):
+        self.addCleanup(support.unlink, support.TESTFN)
         with open(support.TESTFN, "w") as f:
             f.write("body")
         with open(support.TESTFN) as f:
@@ -599,6 +600,7 @@
             self.assertEqual(b'body', f.read())
 
     def test_binary_file_body(self):
+        self.addCleanup(support.unlink, support.TESTFN)
         with open(support.TESTFN, "wb") as f:
             f.write(b"body\xc1")
         with open(support.TESTFN, "rb") as f:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 01:13:07 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 21 Sep 2011 01:13:07 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_test=5Fmultipro?=
 =?utf8?q?cessing_removes_temporary_files?=
Message-ID: <E1R69Ux-0004eK-6u@dinsdale.python.org>

http://hg.python.org/cpython/rev/e06db421e4c4
changeset:   72428:e06db421e4c4
branch:      3.2
parent:      72426:950515bff239
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 21 01:10:29 2011 +0200
summary:
  test_multiprocessing removes temporary files

files:
  Lib/test/test_multiprocessing.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -1590,6 +1590,7 @@
         p = self.Process(target=self._writefd, args=(child_conn, b"foo"))
         p.daemon = True
         p.start()
+        self.addCleanup(test.support.unlink, test.support.TESTFN)
         with open(test.support.TESTFN, "wb") as f:
             fd = f.fileno()
             if msvcrt:
@@ -1614,6 +1615,7 @@
         p = self.Process(target=self._writefd, args=(child_conn, b"bar", True))
         p.daemon = True
         p.start()
+        self.addCleanup(test.support.unlink, test.support.TESTFN)
         with open(test.support.TESTFN, "wb") as f:
             fd = f.fileno()
             for newfd in range(256, MAXFD):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 01:13:07 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 21 Sep 2011 01:13:07 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_3=2E2=3A_test=5Fmultiprocessing_removes_temporary_file?=
 =?utf8?q?s?=
Message-ID: <E1R69Ux-0004eU-VH@dinsdale.python.org>

http://hg.python.org/cpython/rev/c545ed50eac8
changeset:   72429:c545ed50eac8
parent:      72427:1d71aa589549
parent:      72428:e06db421e4c4
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 21 01:12:59 2011 +0200
summary:
  Merge 3.2: test_multiprocessing removes temporary files

files:
  Lib/test/test_multiprocessing.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -1646,6 +1646,7 @@
         p = self.Process(target=self._writefd, args=(child_conn, b"foo"))
         p.daemon = True
         p.start()
+        self.addCleanup(test.support.unlink, test.support.TESTFN)
         with open(test.support.TESTFN, "wb") as f:
             fd = f.fileno()
             if msvcrt:
@@ -1671,6 +1672,7 @@
         p = self.Process(target=self._writefd, args=(child_conn, b"bar", True))
         p.daemon = True
         p.start()
+        self.addCleanup(test.support.unlink, test.support.TESTFN)
         with open(test.support.TESTFN, "wb") as f:
             fd = f.fileno()
             for newfd in range(256, MAXFD):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 03:04:06 2011
From: python-checkins at python.org (meador.inge)
Date: Wed, 21 Sep 2011 03:04:06 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=231172711=3A_Add_=27?=
 =?utf8?q?long_long=27_support_to_the_array_module=2E?=
Message-ID: <E1R6BEM-0004dG-4V@dinsdale.python.org>

http://hg.python.org/cpython/rev/15659e0e2b2e
changeset:   72430:15659e0e2b2e
user:        Meador Inge <meadori at gmail.com>
date:        Tue Sep 20 19:55:51 2011 -0500
summary:
  Issue #1172711: Add 'long long' support to the array module.

Initial patch by Oren Tirosh and Hirokazu Yamamoto.

files:
  Doc/library/array.rst  |  66 ++++++++++++++----------
  Lib/test/test_array.py |  21 +++++++
  Misc/NEWS              |   3 +
  Modules/arraymodule.c  |  79 +++++++++++++++++++++++++++++-
  4 files changed, 141 insertions(+), 28 deletions(-)


diff --git a/Doc/library/array.rst b/Doc/library/array.rst
--- a/Doc/library/array.rst
+++ b/Doc/library/array.rst
@@ -14,37 +14,49 @@
 :dfn:`type code`, which is a single character.  The following type codes are
 defined:
 
-+-----------+----------------+-------------------+-----------------------+
-| Type code | C Type         | Python Type       | Minimum size in bytes |
-+===========+================+===================+=======================+
-| ``'b'``   | signed char    | int               | 1                     |
-+-----------+----------------+-------------------+-----------------------+
-| ``'B'``   | unsigned char  | int               | 1                     |
-+-----------+----------------+-------------------+-----------------------+
-| ``'u'``   | Py_UNICODE     | Unicode character | 2 (see note)          |
-+-----------+----------------+-------------------+-----------------------+
-| ``'h'``   | signed short   | int               | 2                     |
-+-----------+----------------+-------------------+-----------------------+
-| ``'H'``   | unsigned short | int               | 2                     |
-+-----------+----------------+-------------------+-----------------------+
-| ``'i'``   | signed int     | int               | 2                     |
-+-----------+----------------+-------------------+-----------------------+
-| ``'I'``   | unsigned int   | int               | 2                     |
-+-----------+----------------+-------------------+-----------------------+
-| ``'l'``   | signed long    | int               | 4                     |
-+-----------+----------------+-------------------+-----------------------+
-| ``'L'``   | unsigned long  | int               | 4                     |
-+-----------+----------------+-------------------+-----------------------+
-| ``'f'``   | float          | float             | 4                     |
-+-----------+----------------+-------------------+-----------------------+
-| ``'d'``   | double         | float             | 8                     |
-+-----------+----------------+-------------------+-----------------------+
++-----------+--------------------+-------------------+-----------------------+-------+
+| Type code | C Type             | Python Type       | Minimum size in bytes | Notes |
++===========+====================+===================+=======================+=======+
+| ``'b'``   | signed char        | int               | 1                     |       |
++-----------+--------------------+-------------------+-----------------------+-------+
+| ``'B'``   | unsigned char      | int               | 1                     |       |
++-----------+--------------------+-------------------+-----------------------+-------+
+| ``'u'``   | Py_UNICODE         | Unicode character | 2                     | \(1)  |
++-----------+--------------------+-------------------+-----------------------+-------+
+| ``'h'``   | signed short       | int               | 2                     |       |
++-----------+--------------------+-------------------+-----------------------+-------+
+| ``'H'``   | unsigned short     | int               | 2                     |       |
++-----------+--------------------+-------------------+-----------------------+-------+
+| ``'i'``   | signed int         | int               | 2                     |       |
++-----------+--------------------+-------------------+-----------------------+-------+
+| ``'I'``   | unsigned int       | int               | 2                     |       |
++-----------+--------------------+-------------------+-----------------------+-------+
+| ``'l'``   | signed long        | int               | 4                     |       |
++-----------+--------------------+-------------------+-----------------------+-------+
+| ``'L'``   | unsigned long      | int               | 4                     |       |
++-----------+--------------------+-------------------+-----------------------+-------+
+| ``'q'``   | signed long long   | int               | 8                     | \(2)  |
++-----------+--------------------+-------------------+-----------------------+-------+
+| ``'Q'``   | unsigned long long | int               | 8                     | \(2)  |
++-----------+--------------------+-------------------+-----------------------+-------+
+| ``'f'``   | float              | float             | 4                     |       |
++-----------+--------------------+-------------------+-----------------------+-------+
+| ``'d'``   | double             | float             | 8                     |       |
++-----------+--------------------+-------------------+-----------------------+-------+
 
-.. note::
+Notes:
 
-   The ``'u'`` typecode corresponds to Python's unicode character.  On narrow
+(1)
+   The ``'u'`` type code corresponds to Python's unicode character.  On narrow
    Unicode builds this is 2-bytes, on wide builds this is 4-bytes.
 
+(2)
+   The ``'q'`` and ``'Q'`` type codes are available only if
+   the platform C compiler used to build Python supports C :ctype:`long long`,
+   or, on Windows, :ctype:`__int64`.
+
+   .. versionadded:: 3.3
+
 The actual representation of values is determined by the machine architecture
 (strictly speaking, by the C implementation).  The actual size can be accessed
 through the :attr:`itemsize` attribute.
diff --git a/Lib/test/test_array.py b/Lib/test/test_array.py
--- a/Lib/test/test_array.py
+++ b/Lib/test/test_array.py
@@ -16,6 +16,13 @@
 import array
 from array import _array_reconstructor as array_reconstructor
 
+try:
+    # Try to determine availability of long long independently
+    # of the array module under test
+    struct.calcsize('@q')
+    have_long_long = True
+except struct.error:
+    have_long_long = False
 
 class ArraySubclass(array.array):
     pass
@@ -26,6 +33,8 @@
 
 tests = [] # list to accumulate all tests
 typecodes = "ubBhHiIlLfd"
+if have_long_long:
+    typecodes += 'qQ'
 
 class BadConstructorTest(unittest.TestCase):
 
@@ -1205,6 +1214,18 @@
     minitemsize = 4
 tests.append(UnsignedLongTest)
 
+ at unittest.skipIf(not have_long_long, 'need long long support')
+class LongLongTest(SignedNumberTest):
+    typecode = 'q'
+    minitemsize = 8
+tests.append(LongLongTest)
+
+ at unittest.skipIf(not have_long_long, 'need long long support')
+class UnsignedLongLongTest(UnsignedNumberTest):
+    typecode = 'Q'
+    minitemsize = 8
+tests.append(UnsignedLongLongTest)
+
 class FPTest(NumberTest):
     example = [-42.0, 0, 42, 1e5, -1e10]
     smallerexample = [-42.0, 0, 42, 1e5, -2e10]
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1287,6 +1287,9 @@
 Extension Modules
 -----------------
 
+- Issue #1172711: Add 'long long' support to the array module.
+  Initial patch by Oren Tirosh and Hirokazu Yamamoto.
+
 - Issue #12483: ctypes: Fix a crash when the destruction of a callback
   object triggers the garbage collector.
 
diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c
--- a/Modules/arraymodule.c
+++ b/Modules/arraymodule.c
@@ -356,6 +356,59 @@
     return 0;
 }
 
+#ifdef HAVE_LONG_LONG
+
+static PyObject *
+q_getitem(arrayobject *ap, Py_ssize_t i)
+{
+    return PyLong_FromLongLong(((PY_LONG_LONG *)ap->ob_item)[i]);
+}
+
+static int
+q_setitem(arrayobject *ap, Py_ssize_t i, PyObject *v)
+{
+    PY_LONG_LONG x;
+    if (!PyArg_Parse(v, "L;array item must be integer", &x))
+        return -1;
+    if (i >= 0)
+        ((PY_LONG_LONG *)ap->ob_item)[i] = x;
+    return 0;
+}
+
+static PyObject *
+QQ_getitem(arrayobject *ap, Py_ssize_t i)
+{
+    return PyLong_FromUnsignedLongLong(
+        ((unsigned PY_LONG_LONG *)ap->ob_item)[i]);
+}
+
+static int
+QQ_setitem(arrayobject *ap, Py_ssize_t i, PyObject *v)
+{
+    unsigned PY_LONG_LONG x;
+    if (PyLong_Check(v)) {
+        x = PyLong_AsUnsignedLongLong(v);
+        if (x == (unsigned PY_LONG_LONG) -1 && PyErr_Occurred())
+            return -1;
+    }
+    else {
+        PY_LONG_LONG y;
+        if (!PyArg_Parse(v, "L;array item must be integer", &y))
+            return -1;
+        if (y < 0) {
+            PyErr_SetString(PyExc_OverflowError,
+                "unsigned long long is less than minimum");
+            return -1;
+        }
+        x = (unsigned PY_LONG_LONG)y;
+    }
+
+    if (i >= 0)
+        ((unsigned PY_LONG_LONG *)ap->ob_item)[i] = x;
+    return 0;
+}
+#endif
+
 static PyObject *
 f_getitem(arrayobject *ap, Py_ssize_t i)
 {
@@ -406,6 +459,10 @@
     {'I', sizeof(int), II_getitem, II_setitem, "I", 1, 0},
     {'l', sizeof(long), l_getitem, l_setitem, "l", 1, 1},
     {'L', sizeof(long), LL_getitem, LL_setitem, "L", 1, 0},
+#ifdef HAVE_LONG_LONG
+    {'q', sizeof(PY_LONG_LONG), q_getitem, q_setitem, "q", 1, 1},
+    {'Q', sizeof(PY_LONG_LONG), QQ_getitem, QQ_setitem, "Q", 1, 0},
+#endif
     {'f', sizeof(float), f_getitem, f_setitem, "f", 0, 0},
     {'d', sizeof(double), d_getitem, d_setitem, "d", 0, 0},
     {'\0', 0, 0, 0, 0, 0, 0} /* Sentinel */
@@ -1655,6 +1712,16 @@
         intsize = sizeof(long);
         is_signed = 0;
         break;
+#if HAVE_LONG_LONG
+    case 'q':
+        intsize = sizeof(PY_LONG_LONG);
+        is_signed = 1;
+        break;
+    case 'Q':
+        intsize = sizeof(PY_LONG_LONG);
+        is_signed = 0;
+        break;
+#endif 
     default:
         return UNKNOWN_FORMAT;
     }
@@ -2501,7 +2568,11 @@
         }
     }
     PyErr_SetString(PyExc_ValueError,
+#ifdef HAVE_LONG_LONG
+        "bad typecode (must be b, B, u, h, H, i, I, l, L, q, Q, f or d)");
+#else
         "bad typecode (must be b, B, u, h, H, i, I, l, L, f or d)");
+#endif
     return NULL;
 }
 
@@ -2524,12 +2595,18 @@
     'I'         unsigned integer   2 \n\
     'l'         signed integer     4 \n\
     'L'         unsigned integer   4 \n\
+    'q'         signed integer     8 (see note) \n\
+    'Q'         unsigned integer   8 (see note) \n\
     'f'         floating point     4 \n\
     'd'         floating point     8 \n\
 \n\
-NOTE: The 'u' typecode corresponds to Python's unicode character. On \n\
+NOTE: The 'u' type code corresponds to Python's unicode character. On \n\
 narrow builds this is 2-bytes on wide builds this is 4-bytes.\n\
 \n\
+NOTE: The 'q' and 'Q' type codes are only available if the platform \n\
+C compiler used to build Python supports 'long long', or, on Windows, \n\
+'__int64'.\n\
+\n\
 The constructor is:\n\
 \n\
 array(typecode [, initializer]) -- create a new array\n\

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 03:20:07 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 21 Sep 2011 03:20:07 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=231172711=3A_Update_?=
 =?utf8?q?What=27s_New_in_Python_3=2E3_document_for_the_struct_module?=
Message-ID: <E1R6BTr-0005sU-W3@dinsdale.python.org>

http://hg.python.org/cpython/rev/3c56e546dc60
changeset:   72431:3c56e546dc60
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 21 03:20:03 2011 +0200
summary:
  Issue #1172711: Update What's New in Python 3.3 document for the struct module

files:
  Doc/whatsnew/3.3.rst |  21 +++++++++++++++------
  1 files changed, 15 insertions(+), 6 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -115,6 +115,17 @@
  * :option:`-X` ``faulthandler``
 
 
+ftplib
+------
+
+The :class:`~ftplib.FTP_TLS` class now provides a new
+:func:`~ftplib.FTP_TLS.ccc` function to revert control channel back to
+plaintex.  This can be useful to take advantage of firewalls that know how to
+handle NAT with non-secure FTP without opening fixed ports.
+
+(Contributed by Giampaolo Rodol? in :issue:`12139`)
+
+
 math
 ----
 
@@ -242,15 +253,13 @@
   * :func:`~ssl.RAND_pseudo_bytes`: generate pseudo-random bytes.
 
 
-ftplib
+struct
 ------
 
-The :class:`~ftplib.FTP_TLS` class now provides a new
-:func:`~ftplib.FTP_TLS.ccc` function to revert control channel back to
-plaintex.  This can be useful to take advantage of firewalls that know how to
-handle NAT with non-secure FTP without opening fixed ports.
+The :mod:`struct` module supports the :c:type:`long long` type using ``q`` and
+``Q`` type codes.
 
-(Contributed by Giampaolo Rodol? in :issue:`12139`)
+(Contributed by Oren Tirosh and Hirokazu Yamamoto in :issue:`1172711`)
 
 
 shutil

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 03:35:58 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 21 Sep 2011 03:35:58 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_meth=2Elog2_doc=3A_add_a_li?=
 =?utf8?q?nk_to_int=2Ebit=5Flength=28=29?=
Message-ID: <E1R6BjC-0007CP-6N@dinsdale.python.org>

http://hg.python.org/cpython/rev/2d9c4635caf0
changeset:   72432:2d9c4635caf0
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 21 03:35:18 2011 +0200
summary:
  meth.log2 doc: add a link to int.bit_length()

files:
  Doc/library/math.rst |  5 +++++
  1 files changed, 5 insertions(+), 0 deletions(-)


diff --git a/Doc/library/math.rst b/Doc/library/math.rst
--- a/Doc/library/math.rst
+++ b/Doc/library/math.rst
@@ -191,6 +191,11 @@
 
    .. versionadded:: 3.3
 
+   .. seealso::
+
+      :meth:`int.bit_length` returns the number of bits necessary to represent
+      an integer in binary, excluding the sign and leading zeros.
+
 
 .. function:: log10(x)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 03:35:58 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 21 Sep 2011 03:35:58 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_What=27s_New_in_Python_3=2E?=
 =?utf8?q?3=3A_add_curses=2Eunget=5Fwch=28=29?=
Message-ID: <E1R6BjC-0007CZ-Ur@dinsdale.python.org>

http://hg.python.org/cpython/rev/c372a79c771e
changeset:   72433:c372a79c771e
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 21 03:35:44 2011 +0200
summary:
  What's New in Python 3.3: add curses.unget_wch()

files:
  Doc/whatsnew/3.3.rst |  12 ++++++++----
  1 files changed, 8 insertions(+), 4 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -94,17 +94,21 @@
 crypt
 -----
 
-Addition of salf and modular crypt format to the :mod:`crypt` module.
+Addition of salt and modular crypt format and the :func:`~crypt.mksalt`
+function to the :mod:`crypt` module.
 
 (:issue:`10924`)
 
 curses
 ------
 
-The :class:`curses.window` class has a new :meth:`~curses.window.get_wch` method
-to get a wide character.  Patch by I?igo Serna.
+ * The :class:`curses.window` class has a new :meth:`~curses.window.get_wch`
+   method to get a wide character
+ * The :mod:`curses` module has a new :meth:`~curses.unget_wch` function to
+   push a wide character so the next :meth:`~curses.window.get_wch` will return
+   it
 
-(:issue:`6755`)
+(Contributed by I?igo Serna in :issue:`6755`)
 
 faulthandler
 ------------

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 03:56:10 2011
From: python-checkins at python.org (jesus.cea)
Date: Wed, 21 Sep 2011 03:56:10 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogQ2xvc2UgIzEzMDIy?=
 =?utf8?q?=3A_=5Fmultiprocessing=2Erecvfd=28=29_doesn=27t_check_that_file_?=
 =?utf8?q?descriptor_was?=
Message-ID: <E1R6C2k-0000Gi-GY@dinsdale.python.org>

http://hg.python.org/cpython/rev/21e7a55cb943
changeset:   72434:21e7a55cb943
branch:      2.7
parent:      72419:f8280cf63d9e
user:        Jesus Cea <jcea at jcea.es>
date:        Wed Sep 21 03:47:39 2011 +0200
summary:
  Close #13022: _multiprocessing.recvfd() doesn't check that file descriptor was actually received

files:
  Lib/test/test_multiprocessing.py           |  17 ++++++++++
  Misc/NEWS                                  |   3 +
  Modules/_multiprocessing/multiprocessing.c |  11 ++++++
  3 files changed, 31 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -1574,6 +1574,23 @@
         with open(test_support.TESTFN, "rb") as f:
             self.assertEqual(f.read(), b"bar")
 
+    @classmethod
+    def _send_data_without_fd(self, conn):
+        os.write(conn.fileno(), b"\0")
+
+    @unittest.skipIf(sys.platform == "win32", "doesn't make sense on Windows")
+    def test_missing_fd_transfer(self):
+        # Check that exception is raised when received data is not
+        # accompanied by a file descriptor in ancillary data.
+        if self.TYPE != 'processes':
+            self.skipTest("only makes sense with processes")
+        conn, child_conn = self.Pipe(duplex=True)
+
+        p = self.Process(target=self._send_data_without_fd, args=(child_conn,))
+        p.daemon = True
+        p.start()
+        self.assertRaises(RuntimeError, reduction.recv_handle, conn)
+        p.join()
 
 class _TestListenerClient(BaseTestCase):
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -201,6 +201,9 @@
 Extension Modules
 -----------------
 
+- Issue #13022: Fix: _multiprocessing.recvfd() doesn't check that
+  file descriptor was actually received.
+
 - Issue #12483: ctypes: Fix a crash when the destruction of a callback
   object triggers the garbage collector.
 
diff --git a/Modules/_multiprocessing/multiprocessing.c b/Modules/_multiprocessing/multiprocessing.c
--- a/Modules/_multiprocessing/multiprocessing.c
+++ b/Modules/_multiprocessing/multiprocessing.c
@@ -177,6 +177,17 @@
     if (res < 0)
         return PyErr_SetFromErrno(PyExc_OSError);
 
+    if (msg.msg_controllen < CMSG_LEN(sizeof(int)) ||
+        (cmsg = CMSG_FIRSTHDR(&msg)) == NULL ||
+        cmsg->cmsg_level != SOL_SOCKET ||
+        cmsg->cmsg_type != SCM_RIGHTS ||
+        cmsg->cmsg_len < CMSG_LEN(sizeof(int))) {
+        /* If at least one control message is present, there should be
+           no room for any further data in the buffer. */
+        PyErr_SetString(PyExc_RuntimeError, "No file descriptor received");
+        return NULL;
+    }
+
     fd = * (int *) CMSG_DATA(cmsg);
     return Py_BuildValue("i", fd);
 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 03:56:11 2011
From: python-checkins at python.org (jesus.cea)
Date: Wed, 21 Sep 2011 03:56:11 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogQ2xvc2UgIzEzMDIy?=
 =?utf8?q?=3A_=5Fmultiprocessing=2Erecvfd=28=29_doesn=27t_check_that_file_?=
 =?utf8?q?descriptor_was?=
Message-ID: <E1R6C2l-0000Gn-CX@dinsdale.python.org>

http://hg.python.org/cpython/rev/447770470d00
changeset:   72435:447770470d00
branch:      3.2
parent:      72428:e06db421e4c4
user:        Jesus Cea <jcea at jcea.es>
date:        Wed Sep 21 03:53:25 2011 +0200
summary:
  Close #13022: _multiprocessing.recvfd() doesn't check that file descriptor was actually received

files:
  Lib/test/test_multiprocessing.py           |  17 ++++++++++
  Misc/NEWS                                  |   3 +
  Modules/_multiprocessing/multiprocessing.c |  11 ++++++
  3 files changed, 31 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -1632,6 +1632,23 @@
         with open(test.support.TESTFN, "rb") as f:
             self.assertEqual(f.read(), b"bar")
 
+    @classmethod
+    def _send_data_without_fd(self, conn):
+        os.write(conn.fileno(), b"\0")
+
+    @unittest.skipIf(sys.platform == "win32", "doesn't make sense on Windows")
+    def test_missing_fd_transfer(self):
+        # Check that exception is raised when received data is not
+        # accompanied by a file descriptor in ancillary data.
+        if self.TYPE != 'processes':
+            self.skipTest("only makes sense with processes")
+        conn, child_conn = self.Pipe(duplex=True)
+
+        p = self.Process(target=self._send_data_without_fd, args=(child_conn,))
+        p.daemon = True
+        p.start()
+        self.assertRaises(RuntimeError, reduction.recv_handle, conn)
+        p.join()
 
 class _TestListenerClient(BaseTestCase):
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -78,6 +78,9 @@
 Extension Modules
 -----------------
  
+- Issue #13022: Fix: _multiprocessing.recvfd() doesn't check that
+  file descriptor was actually received.
+
 - Issue #12483: ctypes: Fix a crash when the destruction of a callback
   object triggers the garbage collector.
 
diff --git a/Modules/_multiprocessing/multiprocessing.c b/Modules/_multiprocessing/multiprocessing.c
--- a/Modules/_multiprocessing/multiprocessing.c
+++ b/Modules/_multiprocessing/multiprocessing.c
@@ -177,6 +177,17 @@
     if (res < 0)
         return PyErr_SetFromErrno(PyExc_OSError);
 
+    if (msg.msg_controllen < CMSG_LEN(sizeof(int)) ||
+        (cmsg = CMSG_FIRSTHDR(&msg)) == NULL ||
+        cmsg->cmsg_level != SOL_SOCKET ||
+        cmsg->cmsg_type != SCM_RIGHTS ||
+        cmsg->cmsg_len < CMSG_LEN(sizeof(int))) {
+        /* If at least one control message is present, there should be
+           no room for any further data in the buffer. */
+        PyErr_SetString(PyExc_RuntimeError, "No file descriptor received");
+        return NULL;
+    }
+
     fd = * (int *) CMSG_DATA(cmsg);
     return Py_BuildValue("i", fd);
 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 03:56:12 2011
From: python-checkins at python.org (jesus.cea)
Date: Wed, 21 Sep 2011 03:56:12 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Close_=2313022=3A_=5Fmultiprocessing=2Erecvfd=28=29_doesn=27?=
 =?utf8?q?t_check_that_file_descriptor_was?=
Message-ID: <E1R6C2m-0000Gr-6d@dinsdale.python.org>

http://hg.python.org/cpython/rev/74434cff2c90
changeset:   72436:74434cff2c90
parent:      72433:c372a79c771e
parent:      72435:447770470d00
user:        Jesus Cea <jcea at jcea.es>
date:        Wed Sep 21 03:56:05 2011 +0200
summary:
  Close #13022: _multiprocessing.recvfd() doesn't check that file descriptor was actually received

files:
  Lib/test/test_multiprocessing.py           |  17 ++++++++++
  Misc/NEWS                                  |   3 +
  Modules/_multiprocessing/multiprocessing.c |  11 ++++++
  3 files changed, 31 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -1689,6 +1689,23 @@
         with open(test.support.TESTFN, "rb") as f:
             self.assertEqual(f.read(), b"bar")
 
+    @classmethod
+    def _send_data_without_fd(self, conn):
+        os.write(conn.fileno(), b"\0")
+
+    @unittest.skipIf(sys.platform == "win32", "doesn't make sense on Windows")
+    def test_missing_fd_transfer(self):
+        # Check that exception is raised when received data is not
+        # accompanied by a file descriptor in ancillary data.
+        if self.TYPE != 'processes':
+            self.skipTest("only makes sense with processes")
+        conn, child_conn = self.Pipe(duplex=True)
+
+        p = self.Process(target=self._send_data_without_fd, args=(child_conn,))
+        p.daemon = True
+        p.start()
+        self.assertRaises(RuntimeError, reduction.recv_handle, conn)
+        p.join()
 
 class _TestListenerClient(BaseTestCase):
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1287,6 +1287,9 @@
 Extension Modules
 -----------------
 
+- Issue #13022: Fix: _multiprocessing.recvfd() doesn't check that
+  file descriptor was actually received.
+
 - Issue #1172711: Add 'long long' support to the array module.
   Initial patch by Oren Tirosh and Hirokazu Yamamoto.
 
diff --git a/Modules/_multiprocessing/multiprocessing.c b/Modules/_multiprocessing/multiprocessing.c
--- a/Modules/_multiprocessing/multiprocessing.c
+++ b/Modules/_multiprocessing/multiprocessing.c
@@ -166,6 +166,17 @@
     if (res < 0)
         return PyErr_SetFromErrno(PyExc_OSError);
 
+    if (msg.msg_controllen < CMSG_LEN(sizeof(int)) ||
+        (cmsg = CMSG_FIRSTHDR(&msg)) == NULL ||
+        cmsg->cmsg_level != SOL_SOCKET ||
+        cmsg->cmsg_type != SCM_RIGHTS ||
+        cmsg->cmsg_len < CMSG_LEN(sizeof(int))) {
+        /* If at least one control message is present, there should be
+           no room for any further data in the buffer. */
+        PyErr_SetString(PyExc_RuntimeError, "No file descriptor received");
+        return NULL;
+    }
+
     fd = * (int *) CMSG_DATA(cmsg);
     return Py_BuildValue("i", fd);
 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 04:49:26 2011
From: python-checkins at python.org (meador.inge)
Date: Wed, 21 Sep 2011 04:49:26 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=231172711=3A_Update_?=
 =?utf8?q?What=27s_New_in_Python_3=2E3_document_for_the_array_module=2E?=
Message-ID: <E1R6CsI-0004Ja-Ud@dinsdale.python.org>

http://hg.python.org/cpython/rev/672b63aff0f4
changeset:   72437:672b63aff0f4
user:        Meador Inge <meadori at gmail.com>
date:        Tue Sep 20 21:48:16 2011 -0500
summary:
  Issue #1172711: Update What's New in Python 3.3 document for the array module.

The commit fixes 3c56e546dc60, which documented #1172711 as a struct module
change;  it is an array module change.

files:
  Doc/whatsnew/3.3.rst |  17 +++++++++--------
  1 files changed, 9 insertions(+), 8 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -68,6 +68,15 @@
 
 * Stub
 
+array
+-----
+
+The :mod:`array` module supports the :c:type:`long long` type using ``q`` and
+``Q`` type codes.
+
+(Contributed by Oren Tirosh and Hirokazu Yamamoto in :issue:`1172711`)
+
+
 codecs
 ------
 
@@ -257,15 +266,6 @@
   * :func:`~ssl.RAND_pseudo_bytes`: generate pseudo-random bytes.
 
 
-struct
-------
-
-The :mod:`struct` module supports the :c:type:`long long` type using ``q`` and
-``Q`` type codes.
-
-(Contributed by Oren Tirosh and Hirokazu Yamamoto in :issue:`1172711`)
-
-
 shutil
 ------
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Wed Sep 21 05:20:38 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Wed, 21 Sep 2011 05:20:38 +0200
Subject: [Python-checkins] Daily reference leaks (3c56e546dc60): sum=0
Message-ID: <E1R6DMU-00061J-Vp@ap.vmr.nerim.net>

results for 3c56e546dc60 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogYlKf07', '-x']

From ezio.melotti at gmail.com  Wed Sep 21 15:25:29 2011
From: ezio.melotti at gmail.com (Ezio Melotti)
Date: Wed, 21 Sep 2011 16:25:29 +0300
Subject: [Python-checkins] cpython: Issue #1172711: Add 'long long'
 support to the array module.
In-Reply-To: <E1R6BEM-0004dG-4V@dinsdale.python.org>
References: <E1R6BEM-0004dG-4V@dinsdale.python.org>
Message-ID: <4E79E5C9.9080204@gmail.com>

Hi,

On 21/09/2011 4.04, meador.inge wrote:
> http://hg.python.org/cpython/rev/15659e0e2b2e
> changeset:   72430:15659e0e2b2e
> user:        Meador Inge<meadori at gmail.com>
> date:        Tue Sep 20 19:55:51 2011 -0500
> summary:
>    Issue #1172711: Add 'long long' support to the array module.
>
> Initial patch by Oren Tirosh and Hirokazu Yamamoto.
>
> files:
>    Doc/library/array.rst  |  66 ++++++++++++++----------
>    Lib/test/test_array.py |  21 +++++++
>    Misc/NEWS              |   3 +
>    Modules/arraymodule.c  |  79 +++++++++++++++++++++++++++++-
>    4 files changed, 141 insertions(+), 28 deletions(-)
>
>
> diff --git a/Doc/library/array.rst b/Doc/library/array.rst
> --- a/Doc/library/array.rst
> +++ b/Doc/library/array.rst
> @@ -14,37 +14,49 @@
>   :dfn:`type code`, which is a single character.  The following type codes are
>   defined:
>
> -+-----------+----------------+-------------------+-----------------------+
> -| Type code | C Type         | Python Type       | Minimum size in bytes |
> -+===========+================+===================+=======================+
> -| ``'b'``   | signed char    | int               | 1                     |
> -+-----------+----------------+-------------------+-----------------------+
> -| ``'B'``   | unsigned char  | int               | 1                     |
> -+-----------+----------------+-------------------+-----------------------+
> -| ``'u'``   | Py_UNICODE     | Unicode character | 2 (see note)          |
> -+-----------+----------------+-------------------+-----------------------+
> -| ``'h'``   | signed short   | int               | 2                     |
> -+-----------+----------------+-------------------+-----------------------+
> -| ``'H'``   | unsigned short | int               | 2                     |
> -+-----------+----------------+-------------------+-----------------------+
> -| ``'i'``   | signed int     | int               | 2                     |
> -+-----------+----------------+-------------------+-----------------------+
> -| ``'I'``   | unsigned int   | int               | 2                     |
> -+-----------+----------------+-------------------+-----------------------+
> -| ``'l'``   | signed long    | int               | 4                     |
> -+-----------+----------------+-------------------+-----------------------+
> -| ``'L'``   | unsigned long  | int               | 4                     |
> -+-----------+----------------+-------------------+-----------------------+
> -| ``'f'``   | float          | float             | 4                     |
> -+-----------+----------------+-------------------+-----------------------+
> -| ``'d'``   | double         | float             | 8                     |
> -+-----------+----------------+-------------------+-----------------------+
> ++-----------+--------------------+-------------------+-----------------------+-------+
> +| Type code | C Type             | Python Type       | Minimum size in bytes | Notes |
> ++===========+====================+===================+=======================+=======+
> +| ``'b'``   | signed char        | int               | 1                     |       |
> ++-----------+--------------------+-------------------+-----------------------+-------+
> +| ``'B'``   | unsigned char      | int               | 1                     |       |
> ++-----------+--------------------+-------------------+-----------------------+-------+
> +| ``'u'``   | Py_UNICODE         | Unicode character | 2                     | \(1)  |
> ++-----------+--------------------+-------------------+-----------------------+-------+
> +| ``'h'``   | signed short       | int               | 2                     |       |
> ++-----------+--------------------+-------------------+-----------------------+-------+
> +| ``'H'``   | unsigned short     | int               | 2                     |       |
> ++-----------+--------------------+-------------------+-----------------------+-------+
> +| ``'i'``   | signed int         | int               | 2                     |       |
> ++-----------+--------------------+-------------------+-----------------------+-------+
> +| ``'I'``   | unsigned int       | int               | 2                     |       |
> ++-----------+--------------------+-------------------+-----------------------+-------+
> +| ``'l'``   | signed long        | int               | 4                     |       |
> ++-----------+--------------------+-------------------+-----------------------+-------+
> +| ``'L'``   | unsigned long      | int               | 4                     |       |
> ++-----------+--------------------+-------------------+-----------------------+-------+
> +| ``'q'``   | signed long long   | int               | 8                     | \(2)  |
> ++-----------+--------------------+-------------------+-----------------------+-------+
> +| ``'Q'``   | unsigned long long | int               | 8                     | \(2)  |
> ++-----------+--------------------+-------------------+-----------------------+-------+
> +| ``'f'``   | float              | float             | 4                     |       |
> ++-----------+--------------------+-------------------+-----------------------+-------+
> +| ``'d'``   | double             | float             | 8                     |       |
> ++-----------+--------------------+-------------------+-----------------------+-------+
>
> -.. note::
> +Notes:
>
> -   The ``'u'`` typecode corresponds to Python's unicode character.  On narrow
> +(1)
> +   The ``'u'`` type code corresponds to Python's unicode character.  On narrow
>      Unicode builds this is 2-bytes, on wide builds this is 4-bytes.
>
> +(2)
> +   The ``'q'`` and ``'Q'`` type codes are available only if
> +   the platform C compiler used to build Python supports C :ctype:`long long`,
> +   or, on Windows, :ctype:`__int64`.
> +
> +   .. versionadded:: 3.3
> +
>   The actual representation of values is determined by the machine architecture
>   (strictly speaking, by the C implementation).  The actual size can be accessed
>   through the :attr:`itemsize` attribute.
> diff --git a/Lib/test/test_array.py b/Lib/test/test_array.py
> --- a/Lib/test/test_array.py
> +++ b/Lib/test/test_array.py
> @@ -16,6 +16,13 @@
>   import array
>   from array import _array_reconstructor as array_reconstructor
>
> +try:
> +    # Try to determine availability of long long independently
> +    # of the array module under test
> +    struct.calcsize('@q')
> +    have_long_long = True
> +except struct.error:
> +    have_long_long = False
>
>   class ArraySubclass(array.array):
>       pass
> @@ -26,6 +33,8 @@
>
>   tests = [] # list to accumulate all tests
>   typecodes = "ubBhHiIlLfd"
> +if have_long_long:
> +    typecodes += 'qQ'
>
>   class BadConstructorTest(unittest.TestCase):
>
> @@ -1205,6 +1214,18 @@
>       minitemsize = 4
>   tests.append(UnsignedLongTest)
>
> + at unittest.skipIf(not have_long_long, 'need long long support')

I think this would read better with skipUnless and s/have/has/:

@unittest.skipUnless(HAS_LONG_LONG, 'need long long support')


> +class LongLongTest(SignedNumberTest):
> +    typecode = 'q'
> +    minitemsize = 8
> +tests.append(LongLongTest)
> +
> + at unittest.skipIf(not have_long_long, 'need long long support')
> +class UnsignedLongLongTest(UnsignedNumberTest):
> +    typecode = 'Q'
> +    minitemsize = 8
> +tests.append(UnsignedLongLongTest)
> +
>   class FPTest(NumberTest):
>       example = [-42.0, 0, 42, 1e5, -1e10]
>       smallerexample = [-42.0, 0, 42, 1e5, -2e10]
>
> [...]

Best Regards,
Ezio Melotti

From python-checkins at python.org  Wed Sep 21 16:35:44 2011
From: python-checkins at python.org (eric.araujo)
Date: Wed, 21 Sep 2011 16:35:44 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Squash_last_remaining_te?=
	=?utf8?q?mpdir_leak_in_tests?=
Message-ID: <E1R6Nto-0004hd-W1@dinsdale.python.org>

http://hg.python.org/distutils2/rev/2f7040100a12
changeset:   1182:2f7040100a12
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 20:59:26 2011 +0200
summary:
  Squash last remaining tempdir leak in tests

files:
  distutils2/tests/test_pypi_dist.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/distutils2/tests/test_pypi_dist.py b/distutils2/tests/test_pypi_dist.py
--- a/distutils2/tests/test_pypi_dist.py
+++ b/distutils2/tests/test_pypi_dist.py
@@ -1,6 +1,7 @@
 """Tests for the distutils2.pypi.dist module."""
 
 import os
+import shutil
 from distutils2.version import VersionPredicate
 from distutils2.pypi.dist import (ReleaseInfo, ReleasesList, DistInfo,
                                  split_archive_name, get_infos_from_url)
@@ -185,6 +186,7 @@
         dist2 = Dist(url=url)
         # doing an unpack
         dist2_there = dist2.unpack()
+        self.addCleanup(shutil.rmtree, dist2_there)
         dist2_result = os.listdir(dist2_there)
         self.assertIn('paf', dist2_result)
         os.remove(os.path.join(dist2_there, 'paf'))

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Wed Sep 21 16:35:45 2011
From: python-checkins at python.org (eric.araujo)
Date: Wed, 21 Sep 2011 16:35:45 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Remove_two_unneeded_attr?=
	=?utf8?q?ibutes?=
Message-ID: <E1R6Ntp-0004hm-1v@dinsdale.python.org>

http://hg.python.org/distutils2/rev/586dd5bb494c
changeset:   1183:586dd5bb494c
user:        ?ric Araujo <merwok at netwok.org>
date:        Wed Sep 21 16:31:25 2011 +0200
summary:
  Remove two unneeded attributes

files:
  distutils2/command/install_dist.py |  1 -
  distutils2/pypi/simple.py          |  8 +++-----
  2 files changed, 3 insertions(+), 6 deletions(-)


diff --git a/distutils2/command/install_dist.py b/distutils2/command/install_dist.py
--- a/distutils2/command/install_dist.py
+++ b/distutils2/command/install_dist.py
@@ -186,7 +186,6 @@
         self.installer = None
         self.requested = None
         self.no_record = None
-        self.no_resources = None
 
     # -- Option finalizing methods -------------------------------------
     # (This is rather more involved than for most commands,
diff --git a/distutils2/pypi/simple.py b/distutils2/pypi/simple.py
--- a/distutils2/pypi/simple.py
+++ b/distutils2/pypi/simple.py
@@ -118,10 +118,9 @@
     def __init__(self, index_url=DEFAULT_SIMPLE_INDEX_URL, prefer_final=False,
                  prefer_source=True, hosts=DEFAULT_HOSTS,
                  follow_externals=False, mirrors_url=None, mirrors=None,
-                 timeout=SOCKET_TIMEOUT, mirrors_max_tries=0, verbose=False):
+                 timeout=SOCKET_TIMEOUT, mirrors_max_tries=0):
         super(Crawler, self).__init__(prefer_final, prefer_source)
         self.follow_externals = follow_externals
-        self.verbose = verbose
 
         # mirroring attributes.
         parsed = urlparse.urlparse(index_url)
@@ -327,9 +326,8 @@
                                 infos = get_infos_from_url(link, project_name,
                                             is_external=self.index_url not in url)
                             except CantParseArchiveName, e:
-                                if self.verbose:
-                                    logger.warning(
-                                        "version has not been parsed: %s", e)
+                                logger.warning(
+                                    "version has not been parsed: %s", e)
                             else:
                                 self._register_release(release_info=infos)
                         else:

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Wed Sep 21 16:35:45 2011
From: python-checkins at python.org (eric.araujo)
Date: Wed, 21 Sep 2011 16:35:45 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_pep8_compat=2Epy?=
Message-ID: <E1R6Ntp-0004hq-35@dinsdale.python.org>

http://hg.python.org/distutils2/rev/5434147ae874
changeset:   1184:5434147ae874
user:        ?ric Araujo <merwok at netwok.org>
date:        Wed Sep 21 16:31:52 2011 +0200
summary:
  pep8 compat.py

files:
  distutils2/compat.py |  6 +++++-
  1 files changed, 5 insertions(+), 1 deletions(-)


diff --git a/distutils2/compat.py b/distutils2/compat.py
--- a/distutils2/compat.py
+++ b/distutils2/compat.py
@@ -74,6 +74,7 @@
 
 _cookie_re = re.compile("coding[:=]\s*([-\w.]+)")
 
+
 def _get_normal_name(orig_enc):
     """Imitates get_normal_name in tokenizer.c."""
     # Only care about the first 12 characters.
@@ -85,6 +86,7 @@
         return "iso-8859-1"
     return orig_enc
 
+
 def detect_encoding(readline):
     """
     The detect_encoding() function is used to detect the encoding that should
@@ -105,6 +107,7 @@
     bom_found = False
     encoding = None
     default = 'utf-8'
+
     def read_or_stop():
         try:
             return readline()
@@ -168,7 +171,8 @@
     elif isinstance(filename, unicode):
         return filename.encode(sys.getfilesystemencoding())
     else:
-        raise TypeError("expect bytes or str, not %s" % type(filename).__name__)
+        raise TypeError("expect bytes or str, not %s" %
+                        type(filename).__name__)
 
 
 try:

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Wed Sep 21 16:35:45 2011
From: python-checkins at python.org (eric.araujo)
Date: Wed, 21 Sep 2011 16:35:45 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_name_and_attribute_e?=
	=?utf8?q?rrors?=
Message-ID: <E1R6Ntp-0004hu-5Y@dinsdale.python.org>

http://hg.python.org/distutils2/rev/248b102b7d69
changeset:   1185:248b102b7d69
user:        ?ric Araujo <merwok at netwok.org>
date:        Wed Sep 21 16:33:53 2011 +0200
summary:
  Fix name and attribute errors

files:
  distutils2/_backport/tests/test_sysconfig.py |  2 +-
  distutils2/pypi/wrapper.py                   |  4 ++--
  distutils2/tests/test_dist.py                |  2 +-
  3 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/distutils2/_backport/tests/test_sysconfig.py b/distutils2/_backport/tests/test_sysconfig.py
--- a/distutils2/_backport/tests/test_sysconfig.py
+++ b/distutils2/_backport/tests/test_sysconfig.py
@@ -301,7 +301,7 @@
                 stderr=devnull_fp,
                 env=env)
         finally:
-            fp.close()
+            devnull_fp.close()
         test_platform = p.communicate()[0].strip()
         test_platform = test_platform.decode('utf-8')
         status = p.wait()
diff --git a/distutils2/pypi/wrapper.py b/distutils2/pypi/wrapper.py
--- a/distutils2/pypi/wrapper.py
+++ b/distutils2/pypi/wrapper.py
@@ -25,8 +25,8 @@
         exception = None
         methods = [func]
         for f in wrapper._indexes.values():
-            if f != func.__self__ and hasattr(f, func.__name__):
-                methods.append(getattr(f, func.__name__))
+            if f != func.im_self and hasattr(f, func.f_name):
+                methods.append(getattr(f, func.f_name))
         for method in methods:
             try:
                 response = method(*args, **kwargs)
diff --git a/distutils2/tests/test_dist.py b/distutils2/tests/test_dist.py
--- a/distutils2/tests/test_dist.py
+++ b/distutils2/tests/test_dist.py
@@ -70,7 +70,7 @@
             __, stdout = captured_stdout(create_distribution, files)
             self.assertEqual(stdout, '')
         finally:
-            packaging.dist.DEBUG = False
+            distutils2.dist.DEBUG = False
 
     def test_bad_attr(self):
         Distribution(attrs={'author': 'xxx',

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Wed Sep 21 16:35:45 2011
From: python-checkins at python.org (eric.araujo)
Date: Wed, 21 Sep 2011 16:35:45 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Improve_tests=2Esh=2E?=
Message-ID: <E1R6Ntp-0004hy-7Q@dinsdale.python.org>

http://hg.python.org/distutils2/rev/fce523641deb
changeset:   1186:fce523641deb
user:        ?ric Araujo <merwok at netwok.org>
date:        Wed Sep 21 16:35:33 2011 +0200
summary:
  Improve tests.sh.

- When a run fails, re-run without quiet mode (wow automation it?s
  incredible)

- 3.x versions are not tested, as this branch of d2 is only for 2.x
  (I?ll push a branch for 3.1-3.2 later)

files:
  tests.sh |  68 +++++++++++---------------------------------
  1 files changed, 17 insertions(+), 51 deletions(-)


diff --git a/tests.sh b/tests.sh
--- a/tests.sh
+++ b/tests.sh
@@ -1,82 +1,48 @@
 #!/bin/sh
-echo -n "Running tests for Python 2.4... "
-rm -f distutils2/_backport/_hashlib.so
+echo -n "Running tests with Python 2.4... "
 python2.4 setup.py build_ext -f -q 2> /dev/null > /dev/null
-python2.4 -Wd runtests.py -q 2> /dev/null
+python2.4 -Wd runtests.py -q
 if [ $? -ne 0 ];then
-    echo Failed
-    rm -f distutils2/_backport/_hashlib.so
+    echo Failed, re-running
+    python2.4 -Wd runtests.py
     exit 1
 else
     echo Success
 fi
 
-echo -n "Running tests for Python 2.5... "
-python2.5 -Wd runtests.py -q 2> /dev/null
+echo -n "Running tests with Python 2.5... "
+python2.5 -Wd runtests.py -q
 if [ $? -ne 0 ];then
-    echo Failed
+    echo Failed, re-running
+    python2.5 -Wd runtests.py
     exit 1
 else
     echo Success
 fi
 
-echo -n "Running tests for Python 2.6... "
-python2.6 -Wd runtests.py -q 2> /dev/null
+echo -n "Running tests with Python 2.6... "
+python2.6 -Wd runtests.py -q
 if [ $? -ne 0 ];then
-    echo Failed
+    echo Failed, re-running
+    python2.6 -Wd runtests.py
     exit 1
 else
     echo Success
 fi
 
-echo -n "Running tests for Python 2.7... "
-python2.7 -Wd -bb -3 runtests.py -q 2> /dev/null
+echo -n "Running tests with Python 2.7... "
+python2.7 -Wd -bb -3 runtests.py -q
 if [ $? -ne 0 ];then
-    echo Failed
+    echo Failed, re-running
+    python2.7 -Wd -bb -3 runtests.py
     exit 1
 else
     echo Success
 fi
 
-LIB=$( python3.1 -c "from distutils.util import get_platform; import sys; print('lib.%s-%s' % (get_platform(), sys.version[0:3]))" )
-
-
-echo -n "Running tests for Python 3.1... "
-python3.1 setup.py build -q 2> /dev/null > /dev/null
-cp runtests.py build/
-cd build
-PYTHONPATH=${LIB} python3.1 runtests.py -q 2> /dev/null 
-
 if [ $? -ne 0 ];then
     echo Failed
     exit 1
 else
-    echo Success
+    echo "Good job, commit now! (or add tests)"
 fi
-
-echo -n "Running tests for Python 3.2... "
-python3.2 setup.py build -q 2> /dev/null > /dev/null
-cp runtests.py build/
-cd build
-PYTHONPATH=${LIB} python3.2 runtests.py -q 2> /dev/null
-
-if [ $? -ne 0 ];then
-    echo Failed
-    exit 1
-else
-    echo Success
-fi
-
-echo -n "Running tests for Python 3.3... "
-python3.2 setup.py build -q 2> /dev/null > /dev/null
-cp runtests.py build/
-cd build
-PYTHONPATH=${LIB} python3.3 runtests.py -q 2> /dev/null
-
-if [ $? -ne 0 ];then
-    echo Failed
-    exit 1
-else
-    echo Success
-fi
-echo "Good job, commit now! (or add tests)"

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Wed Sep 21 16:36:42 2011
From: python-checkins at python.org (eric.araujo)
Date: Wed, 21 Sep 2011 16:36:42 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Squash_last_tempdir_leak_in?=
	=?utf8?q?_packaging_tests=2E?=
Message-ID: <E1R6Nuk-0004le-26@dinsdale.python.org>

http://hg.python.org/cpython/rev/69915be386b2
changeset:   72438:69915be386b2
parent:      72415:b194af345cb5
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Sep 19 20:57:52 2011 +0200
summary:
  Squash last tempdir leak in packaging tests.

Sweet taste of victory!  Alexis, you can fix the threads leaks :)

files:
  Lib/packaging/tests/test_pypi_dist.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/packaging/tests/test_pypi_dist.py b/Lib/packaging/tests/test_pypi_dist.py
--- a/Lib/packaging/tests/test_pypi_dist.py
+++ b/Lib/packaging/tests/test_pypi_dist.py
@@ -1,6 +1,7 @@
 """Tests for the packaging.pypi.dist module."""
 
 import os
+import shutil
 from packaging.version import VersionPredicate
 from packaging.pypi.dist import (ReleaseInfo, ReleasesList, DistInfo,
                                  split_archive_name, get_infos_from_url)
@@ -185,6 +186,7 @@
         dist2 = Dist(url=url)
         # doing an unpack
         dist2_there = dist2.unpack()
+        self.addCleanup(shutil.rmtree, dist2_there)
         dist2_result = os.listdir(dist2_there)
         self.assertIn('paf', dist2_result)
         os.remove(os.path.join(dist2_there, 'paf'))

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 16:36:43 2011
From: python-checkins at python.org (eric.araujo)
Date: Wed, 21 Sep 2011 16:36:43 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_two_unneeded_attribu?=
	=?utf8?q?tes_in_packaging?=
Message-ID: <E1R6Nul-0004li-0c@dinsdale.python.org>

http://hg.python.org/cpython/rev/04a0af3ce5dc
changeset:   72439:04a0af3ce5dc
user:        ?ric Araujo <merwok at netwok.org>
date:        Wed Sep 21 16:28:03 2011 +0200
summary:
  Remove two unneeded attributes in packaging

files:
  Doc/library/packaging.pypi.simple.rst |  2 +-
  Lib/packaging/command/install_dist.py |  1 -
  Lib/packaging/pypi/simple.py          |  8 +++-----
  3 files changed, 4 insertions(+), 7 deletions(-)


diff --git a/Doc/library/packaging.pypi.simple.rst b/Doc/library/packaging.pypi.simple.rst
--- a/Doc/library/packaging.pypi.simple.rst
+++ b/Doc/library/packaging.pypi.simple.rst
@@ -24,7 +24,7 @@
                    prefer_final=False, prefer_source=True, \
                    hosts=('*',), follow_externals=False, \
                    mirrors_url=None, mirrors=None, timeout=15, \
-                   mirrors_max_tries=0, verbose=False)
+                   mirrors_max_tries=0)
 
       *index_url* is the address of the index to use for requests.
 
diff --git a/Lib/packaging/command/install_dist.py b/Lib/packaging/command/install_dist.py
--- a/Lib/packaging/command/install_dist.py
+++ b/Lib/packaging/command/install_dist.py
@@ -176,7 +176,6 @@
         self.installer = None
         self.requested = None
         self.no_record = None
-        self.no_resources = None
 
     # -- Option finalizing methods -------------------------------------
     # (This is rather more involved than for most commands,
diff --git a/Lib/packaging/pypi/simple.py b/Lib/packaging/pypi/simple.py
--- a/Lib/packaging/pypi/simple.py
+++ b/Lib/packaging/pypi/simple.py
@@ -119,10 +119,9 @@
     def __init__(self, index_url=DEFAULT_SIMPLE_INDEX_URL, prefer_final=False,
                  prefer_source=True, hosts=DEFAULT_HOSTS,
                  follow_externals=False, mirrors_url=None, mirrors=None,
-                 timeout=SOCKET_TIMEOUT, mirrors_max_tries=0, verbose=False):
+                 timeout=SOCKET_TIMEOUT, mirrors_max_tries=0):
         super(Crawler, self).__init__(prefer_final, prefer_source)
         self.follow_externals = follow_externals
-        self.verbose = verbose
 
         # mirroring attributes.
         parsed = urllib.parse.urlparse(index_url)
@@ -322,9 +321,8 @@
                                 infos = get_infos_from_url(link, project_name,
                                             is_external=self.index_url not in url)
                             except CantParseArchiveName as e:
-                                if self.verbose:
-                                    logger.warning(
-                                        "version has not been parsed: %s", e)
+                                logger.warning(
+                                    "version has not been parsed: %s", e)
                             else:
                                 self._register_release(release_info=infos)
                         else:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 16:36:44 2011
From: python-checkins at python.org (eric.araujo)
Date: Wed, 21 Sep 2011 16:36:44 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Branch_merge?=
Message-ID: <E1R6Num-0004mE-9T@dinsdale.python.org>

http://hg.python.org/cpython/rev/210b2773c87a
changeset:   72440:210b2773c87a
parent:      72437:672b63aff0f4
parent:      72439:04a0af3ce5dc
user:        ?ric Araujo <merwok at netwok.org>
date:        Wed Sep 21 16:36:08 2011 +0200
summary:
  Branch merge

files:
  Doc/library/packaging.pypi.simple.rst |  2 +-
  Lib/packaging/command/install_dist.py |  1 -
  Lib/packaging/pypi/simple.py          |  8 +++-----
  Lib/packaging/tests/test_pypi_dist.py |  2 ++
  4 files changed, 6 insertions(+), 7 deletions(-)


diff --git a/Doc/library/packaging.pypi.simple.rst b/Doc/library/packaging.pypi.simple.rst
--- a/Doc/library/packaging.pypi.simple.rst
+++ b/Doc/library/packaging.pypi.simple.rst
@@ -24,7 +24,7 @@
                    prefer_final=False, prefer_source=True, \
                    hosts=('*',), follow_externals=False, \
                    mirrors_url=None, mirrors=None, timeout=15, \
-                   mirrors_max_tries=0, verbose=False)
+                   mirrors_max_tries=0)
 
       *index_url* is the address of the index to use for requests.
 
diff --git a/Lib/packaging/command/install_dist.py b/Lib/packaging/command/install_dist.py
--- a/Lib/packaging/command/install_dist.py
+++ b/Lib/packaging/command/install_dist.py
@@ -176,7 +176,6 @@
         self.installer = None
         self.requested = None
         self.no_record = None
-        self.no_resources = None
 
     # -- Option finalizing methods -------------------------------------
     # (This is rather more involved than for most commands,
diff --git a/Lib/packaging/pypi/simple.py b/Lib/packaging/pypi/simple.py
--- a/Lib/packaging/pypi/simple.py
+++ b/Lib/packaging/pypi/simple.py
@@ -119,10 +119,9 @@
     def __init__(self, index_url=DEFAULT_SIMPLE_INDEX_URL, prefer_final=False,
                  prefer_source=True, hosts=DEFAULT_HOSTS,
                  follow_externals=False, mirrors_url=None, mirrors=None,
-                 timeout=SOCKET_TIMEOUT, mirrors_max_tries=0, verbose=False):
+                 timeout=SOCKET_TIMEOUT, mirrors_max_tries=0):
         super(Crawler, self).__init__(prefer_final, prefer_source)
         self.follow_externals = follow_externals
-        self.verbose = verbose
 
         # mirroring attributes.
         parsed = urllib.parse.urlparse(index_url)
@@ -322,9 +321,8 @@
                                 infos = get_infos_from_url(link, project_name,
                                             is_external=self.index_url not in url)
                             except CantParseArchiveName as e:
-                                if self.verbose:
-                                    logger.warning(
-                                        "version has not been parsed: %s", e)
+                                logger.warning(
+                                    "version has not been parsed: %s", e)
                             else:
                                 self._register_release(release_info=infos)
                         else:
diff --git a/Lib/packaging/tests/test_pypi_dist.py b/Lib/packaging/tests/test_pypi_dist.py
--- a/Lib/packaging/tests/test_pypi_dist.py
+++ b/Lib/packaging/tests/test_pypi_dist.py
@@ -1,6 +1,7 @@
 """Tests for the packaging.pypi.dist module."""
 
 import os
+import shutil
 from packaging.version import VersionPredicate
 from packaging.pypi.dist import (ReleaseInfo, ReleasesList, DistInfo,
                                  split_archive_name, get_infos_from_url)
@@ -185,6 +186,7 @@
         dist2 = Dist(url=url)
         # doing an unpack
         dist2_there = dist2.unpack()
+        self.addCleanup(shutil.rmtree, dist2_there)
         dist2_result = os.listdir(dist2_there)
         self.assertIn('paf', dist2_result)
         os.remove(os.path.join(dist2_there, 'paf'))

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 18:47:44 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Wed, 21 Sep 2011 18:47:44 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyOTgx?=
 =?utf8?q?=3A_test=5Fmultiprocessing=3A_catch_ImportError_when_importing?=
Message-ID: <E1R6PxY-0002iY-IV@dinsdale.python.org>

http://hg.python.org/cpython/rev/c158eac8e951
changeset:   72441:c158eac8e951
branch:      2.7
parent:      72434:21e7a55cb943
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Wed Sep 21 18:44:49 2011 +0200
summary:
  Issue #12981: test_multiprocessing: catch ImportError when importing
multiprocessing.reduction, which may not be available (e.g. if the OS doesn't
support FD passing over Unix domain sockets).

files:
  Lib/test/test_multiprocessing.py |  15 +++++++++++++--
  1 files changed, 13 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -33,7 +33,13 @@
 import multiprocessing.heap
 import multiprocessing.pool
 
-from multiprocessing import util, reduction
+from multiprocessing import util
+
+try:
+    from multiprocessing import reduction
+    HAS_REDUCTION = True
+except ImportError:
+    HAS_REDUCTION = False
 
 try:
     from multiprocessing.sharedctypes import Value, copy
@@ -1526,6 +1532,7 @@
         os.write(fd, data)
         os.close(fd)
 
+    @unittest.skipUnless(HAS_REDUCTION, "test needs multiprocessing.reduction")
     def test_fd_transfer(self):
         if self.TYPE != 'processes':
             self.skipTest("only makes sense with processes")
@@ -1543,6 +1550,7 @@
         with open(test_support.TESTFN, "rb") as f:
             self.assertEqual(f.read(), b"foo")
 
+    @unittest.skipUnless(HAS_REDUCTION, "test needs multiprocessing.reduction")
     @unittest.skipIf(sys.platform == "win32",
                      "test semantics don't make sense on Windows")
     @unittest.skipIf(MAXFD <= 256,
@@ -1578,6 +1586,7 @@
     def _send_data_without_fd(self, conn):
         os.write(conn.fileno(), b"\0")
 
+    @unittest.skipUnless(HAS_REDUCTION, "test needs multiprocessing.reduction")
     @unittest.skipIf(sys.platform == "win32", "doesn't make sense on Windows")
     def test_missing_fd_transfer(self):
         # Check that exception is raised when received data is not
@@ -1899,10 +1908,12 @@
             'multiprocessing', 'multiprocessing.connection',
             'multiprocessing.heap', 'multiprocessing.managers',
             'multiprocessing.pool', 'multiprocessing.process',
-            'multiprocessing.reduction',
             'multiprocessing.synchronize', 'multiprocessing.util'
             ]
 
+        if HAS_REDUCTION:
+            modules.append('multiprocessing.reduction')
+
         if c_int is not None:
             # This module requires _ctypes
             modules.append('multiprocessing.sharedctypes')

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 18:47:45 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Wed, 21 Sep 2011 18:47:45 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyOTgx?=
 =?utf8?q?=3A_test=5Fmultiprocessing=3A_catch_ImportError_when_importing?=
Message-ID: <E1R6PxZ-0002ic-Ak@dinsdale.python.org>

http://hg.python.org/cpython/rev/6e04d406bb86
changeset:   72442:6e04d406bb86
branch:      3.2
parent:      72435:447770470d00
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Wed Sep 21 18:48:21 2011 +0200
summary:
  Issue #12981: test_multiprocessing: catch ImportError when importing
multiprocessing.reduction, which may not be available (e.g. if the OS doesn't
support FD passing over Unix domain sockets).

files:
  Lib/test/test_multiprocessing.py |  15 +++++++++++++--
  1 files changed, 13 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -35,7 +35,13 @@
 import multiprocessing.heap
 import multiprocessing.pool
 
-from multiprocessing import util, reduction
+from multiprocessing import util
+
+try:
+    from multiprocessing import reduction
+    HAS_REDUCTION = True
+except ImportError:
+    HAS_REDUCTION = False
 
 try:
     from multiprocessing.sharedctypes import Value, copy
@@ -1582,6 +1588,7 @@
         os.write(fd, data)
         os.close(fd)
 
+    @unittest.skipUnless(HAS_REDUCTION, "test needs multiprocessing.reduction")
     def test_fd_transfer(self):
         if self.TYPE != 'processes':
             self.skipTest("only makes sense with processes")
@@ -1600,6 +1607,7 @@
         with open(test.support.TESTFN, "rb") as f:
             self.assertEqual(f.read(), b"foo")
 
+    @unittest.skipUnless(HAS_REDUCTION, "test needs multiprocessing.reduction")
     @unittest.skipIf(sys.platform == "win32",
                      "test semantics don't make sense on Windows")
     @unittest.skipIf(MAXFD <= 256,
@@ -1636,6 +1644,7 @@
     def _send_data_without_fd(self, conn):
         os.write(conn.fileno(), b"\0")
 
+    @unittest.skipUnless(HAS_REDUCTION, "test needs multiprocessing.reduction")
     @unittest.skipIf(sys.platform == "win32", "doesn't make sense on Windows")
     def test_missing_fd_transfer(self):
         # Check that exception is raised when received data is not
@@ -1957,10 +1966,12 @@
             'multiprocessing', 'multiprocessing.connection',
             'multiprocessing.heap', 'multiprocessing.managers',
             'multiprocessing.pool', 'multiprocessing.process',
-            'multiprocessing.reduction',
             'multiprocessing.synchronize', 'multiprocessing.util'
             ]
 
+        if HAS_REDUCTION:
+            modules.append('multiprocessing.reduction')
+
         if c_int is not None:
             # This module requires _ctypes
             modules.append('multiprocessing.sharedctypes')

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 18:47:46 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Wed, 21 Sep 2011 18:47:46 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2312981=3A_test=5Fmultiprocessing=3A_catch_ImportErro?=
 =?utf8?q?r_when_importing?=
Message-ID: <E1R6Pxa-0002ig-2Z@dinsdale.python.org>

http://hg.python.org/cpython/rev/21b837aa07b9
changeset:   72443:21b837aa07b9
parent:      72440:210b2773c87a
parent:      72442:6e04d406bb86
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Wed Sep 21 18:49:18 2011 +0200
summary:
  Issue #12981: test_multiprocessing: catch ImportError when importing
multiprocessing.reduction, which may not be available (e.g. if the OS doesn't
support FD passing over Unix domain sockets).

files:
  Lib/test/test_multiprocessing.py |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -1693,6 +1693,7 @@
     def _send_data_without_fd(self, conn):
         os.write(conn.fileno(), b"\0")
 
+    @unittest.skipUnless(HAS_REDUCTION, "test needs multiprocessing.reduction")
     @unittest.skipIf(sys.platform == "win32", "doesn't make sense on Windows")
     def test_missing_fd_transfer(self):
         # Check that exception is raised when received data is not

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 19:10:10 2011
From: python-checkins at python.org (stefan.krah)
Date: Wed, 21 Sep 2011 19:10:10 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313002=3A_Fix_Visua?=
 =?utf8?q?l_Studio_warning_=28not_enough_actual_parameters=29=2E?=
Message-ID: <E1R6QJG-0005L7-RE@dinsdale.python.org>

http://hg.python.org/cpython/rev/573d73e62bbc
changeset:   72444:573d73e62bbc
user:        Stefan Krah <skrah at bytereef.org>
date:        Wed Sep 21 19:08:39 2011 +0200
summary:
  Issue #13002: Fix Visual Studio warning (not enough actual parameters).

files:
  Python/peephole.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Python/peephole.c b/Python/peephole.c
--- a/Python/peephole.c
+++ b/Python/peephole.c
@@ -66,7 +66,7 @@
     const_stack_top = -1; \
     } while(0)
 
-#define CONST_STACK_TOP(x) \
+#define CONST_STACK_TOP() \
     const_stack[const_stack_top]
 
 #define CONST_STACK_LASTN(i) \

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 20:10:24 2011
From: python-checkins at python.org (senthil.kumaran)
Date: Wed, 21 Sep 2011 20:10:24 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_closes_issu?=
 =?utf8?q?e13005_-_Remove_the_mention_of_=27repeat=27_method_in_the_operat?=
 =?utf8?q?or?=
Message-ID: <E1R6RFY-00033p-0J@dinsdale.python.org>

http://hg.python.org/cpython/rev/0d0bfbaaf95c
changeset:   72445:0d0bfbaaf95c
branch:      3.2
parent:      72442:6e04d406bb86
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Thu Sep 22 02:09:17 2011 +0800
summary:
  Fix closes issue13005 - Remove the mention of 'repeat' method in the operator module documentation.

files:
  Doc/library/operator.rst |  2 --
  1 files changed, 0 insertions(+), 2 deletions(-)


diff --git a/Doc/library/operator.rst b/Doc/library/operator.rst
--- a/Doc/library/operator.rst
+++ b/Doc/library/operator.rst
@@ -378,8 +378,6 @@
 +-----------------------+-------------------------+---------------------------------------+
 | Right Shift           | ``a >> b``              | ``rshift(a, b)``                      |
 +-----------------------+-------------------------+---------------------------------------+
-| Sequence Repetition   | ``seq * i``             | ``repeat(seq, i)``                    |
-+-----------------------+-------------------------+---------------------------------------+
 | Slice Assignment      | ``seq[i:j] = values``   | ``setitem(seq, slice(i, j), values)`` |
 +-----------------------+-------------------------+---------------------------------------+
 | Slice Deletion        | ``del seq[i:j]``        | ``delitem(seq, slice(i, j))``         |

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 20:10:24 2011
From: python-checkins at python.org (senthil.kumaran)
Date: Wed, 21 Sep 2011 20:10:24 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_merge_3=2E2=2E__Fix_closes_issue13005_-_Remove_the_mention_o?=
 =?utf8?q?f_=27repeat=27_method_in?=
Message-ID: <E1R6RFY-00033t-PR@dinsdale.python.org>

http://hg.python.org/cpython/rev/6c60f2aacc83
changeset:   72446:6c60f2aacc83
parent:      72444:573d73e62bbc
parent:      72445:0d0bfbaaf95c
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Thu Sep 22 02:10:09 2011 +0800
summary:
  merge 3.2.  Fix closes issue13005 - Remove the mention of 'repeat' method in the operator module documentation.

files:
  Doc/library/operator.rst |  2 --
  1 files changed, 0 insertions(+), 2 deletions(-)


diff --git a/Doc/library/operator.rst b/Doc/library/operator.rst
--- a/Doc/library/operator.rst
+++ b/Doc/library/operator.rst
@@ -378,8 +378,6 @@
 +-----------------------+-------------------------+---------------------------------------+
 | Right Shift           | ``a >> b``              | ``rshift(a, b)``                      |
 +-----------------------+-------------------------+---------------------------------------+
-| Sequence Repetition   | ``seq * i``             | ``repeat(seq, i)``                    |
-+-----------------------+-------------------------+---------------------------------------+
 | Slice Assignment      | ``seq[i:j] = values``   | ``setitem(seq, slice(i, j), values)`` |
 +-----------------------+-------------------------+---------------------------------------+
 | Slice Deletion        | ``del seq[i:j]``        | ``delitem(seq, slice(i, j))``         |

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 22:03:24 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Wed, 21 Sep 2011 22:03:24 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_a_race_cond?=
 =?utf8?q?ition_in_test=5Fsocket=2EThreadableTest=3A_the_client_is_reporte?=
 =?utf8?q?d?=
Message-ID: <E1R6T0u-0005Yf-0t@dinsdale.python.org>

http://hg.python.org/cpython/rev/93a5388a3721
changeset:   72447:93a5388a3721
branch:      2.7
parent:      72441:c158eac8e951
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Wed Sep 21 22:02:27 2011 +0200
summary:
  Fix a race condition in test_socket.ThreadableTest: the client is reported
ready before having been set up.

files:
  Lib/test/test_socket.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -145,8 +145,8 @@
 
     def clientRun(self, test_func):
         self.server_ready.wait()
+        self.clientSetUp()
         self.client_ready.set()
-        self.clientSetUp()
         with test_support.check_py3k_warnings():
             if not callable(test_func):
                 raise TypeError("test_func must be a callable function.")

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 22:03:24 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Wed, 21 Sep 2011 22:03:24 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_a_race_cond?=
 =?utf8?q?ition_in_test=5Fsocket=2EThreadableTest=3A_the_client_is_reporte?=
 =?utf8?q?d_as?=
Message-ID: <E1R6T0u-0005Yp-Q1@dinsdale.python.org>

http://hg.python.org/cpython/rev/aa4b9ed71538
changeset:   72448:aa4b9ed71538
branch:      3.2
parent:      72445:0d0bfbaaf95c
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Wed Sep 21 22:03:58 2011 +0200
summary:
  Fix a race condition in test_socket.ThreadableTest: the client is reported as
ready before having been set up.

files:
  Lib/test/test_socket.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -156,8 +156,8 @@
 
     def clientRun(self, test_func):
         self.server_ready.wait()
+        self.clientSetUp()
         self.client_ready.set()
-        self.clientSetUp()
         if not hasattr(test_func, '__call__'):
             raise TypeError("test_func must be a callable function")
         try:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 22:03:25 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Wed, 21 Sep 2011 22:03:25 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Fix_a_race_condition_in_test=5Fsocket=2EThreadableTest=3A_th?=
 =?utf8?q?e_client_is_reported?=
Message-ID: <E1R6T0v-0005Yt-J5@dinsdale.python.org>

http://hg.python.org/cpython/rev/a06ef7ab7321
changeset:   72449:a06ef7ab7321
parent:      72446:6c60f2aacc83
parent:      72448:aa4b9ed71538
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Wed Sep 21 22:05:01 2011 +0200
summary:
  Fix a race condition in test_socket.ThreadableTest: the client is reported
ready before having been set up.

files:
  Lib/test/test_socket.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -161,8 +161,8 @@
 
     def clientRun(self, test_func):
         self.server_ready.wait()
+        self.clientSetUp()
         self.client_ready.set()
-        self.clientSetUp()
         if not hasattr(test_func, '__call__'):
             raise TypeError("test_func must be a callable function")
         try:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 21 23:07:12 2011
From: python-checkins at python.org (ezio.melotti)
Date: Wed, 21 Sep 2011 23:07:12 +0200
Subject: [Python-checkins] =?utf8?q?devguide=3A_=2313010=3A_explain_why_th?=
 =?utf8?q?e_python_executable_has_an_=2Eexe_extension_on_OS_X=2E?=
Message-ID: <E1R6U0e-0003Ou-Uz@dinsdale.python.org>

http://hg.python.org/devguide/rev/9363db42626a
changeset:   453:9363db42626a
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Sep 22 00:07:04 2011 +0300
summary:
  #13010: explain why the python executable has an .exe extension on OS X.

files:
  setup.rst |  5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)


diff --git a/setup.rst b/setup.rst
--- a/setup.rst
+++ b/setup.rst
@@ -124,8 +124,9 @@
 
 Once CPython is done building you will then have a working build
 that can be run in-place; ``./python`` on most machines (and what is used in
-all examples), ``./python.exe`` on OS X (when on a case-insensitive filesystem,
-which is the default). There is normally no need to install your built copy
+all examples), ``./python.exe`` wherever a case-insensitive filesystem is used
+(e.g. on OS X by default), in order to avoid conflicts with the ``Python``
+directory. There is normally no need to install your built copy
 of Python! The interpreter will realize where it is being run from
 and thus use the files found in the working copy. If you are worried
 you might accidentally install your working copy build, you can add

-- 
Repository URL: http://hg.python.org/devguide

From solipsis at pitrou.net  Thu Sep 22 05:20:37 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Thu, 22 Sep 2011 05:20:37 +0200
Subject: [Python-checkins] Daily reference leaks (a06ef7ab7321): sum=0
Message-ID: <E1R6Zq1-0005ul-Qz@ap.vmr.nerim.net>

results for a06ef7ab7321 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogdo_8PY', '-x']

From python-checkins at python.org  Fri Sep 23 01:31:09 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 23 Sep 2011 01:31:09 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyOTMx?=
 =?utf8?q?=3A_xmlrpclib_now_encodes_Unicode_URI_to_ISO-8859-1=2C_instead_o?=
 =?utf8?q?f?=
Message-ID: <E1R6sjV-0007BN-JZ@dinsdale.python.org>

http://hg.python.org/cpython/rev/c02e790c4535
changeset:   72450:c02e790c4535
branch:      2.7
parent:      72447:93a5388a3721
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 23 01:15:32 2011 +0200
summary:
  Issue #12931: xmlrpclib now encodes Unicode URI to ISO-8859-1, instead of
failing with a UnicodeDecodeError.

files:
  Lib/test/test_xmlrpc.py |  3 +++
  Lib/xmlrpclib.py        |  3 +++
  Misc/NEWS               |  3 +++
  3 files changed, 9 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py
--- a/Lib/test/test_xmlrpc.py
+++ b/Lib/test/test_xmlrpc.py
@@ -472,6 +472,9 @@
                 # protocol error; provide additional information in test output
                 self.fail("%s\n%s" % (e, getattr(e, "headers", "")))
 
+    def test_unicode_host(self):
+        server = xmlrpclib.ServerProxy(u"http://%s:%d/RPC2"%(ADDR, PORT))
+        self.assertEqual(server.add("a", u"\xe9"), u"a\xe9")
 
     # [ch] The test 404 is causing lots of false alarms.
     def XXXtest_404(self):
diff --git a/Lib/xmlrpclib.py b/Lib/xmlrpclib.py
--- a/Lib/xmlrpclib.py
+++ b/Lib/xmlrpclib.py
@@ -1539,6 +1539,9 @@
                  allow_none=0, use_datetime=0):
         # establish a "logical" server connection
 
+        if isinstance(uri, unicode):
+            uri = uri.encode('ISO-8859-1')
+
         # get the url
         import urllib
         type, uri = urllib.splittype(uri)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -47,6 +47,9 @@
 Library
 -------
 
+- Issue #12931: xmlrpclib now encodes Unicode URI to ISO-8859-1, instead of
+  failing with a UnicodeDecodeError.
+
 - Issue #8933: distutils' PKG-INFO files will now correctly report
   Metadata-Version: 1.1 instead of 1.0 if a Classifier or Download-URL field is
   present.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep 23 01:31:10 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 23 Sep 2011 01:31:10 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyOTMx?=
 =?utf8?q?=3A_Add_a_test_with_Unicode_URI_to_test=5Fxmlrpc?=
Message-ID: <E1R6sjW-0007BR-Bb@dinsdale.python.org>

http://hg.python.org/cpython/rev/5ceab07bcd02
changeset:   72451:5ceab07bcd02
branch:      3.2
parent:      72448:aa4b9ed71538
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 23 01:29:44 2011 +0200
summary:
  Issue #12931: Add a test with Unicode URI to test_xmlrpc

files:
  Lib/test/test_xmlrpc.py |  4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py
--- a/Lib/test/test_xmlrpc.py
+++ b/Lib/test/test_xmlrpc.py
@@ -584,6 +584,10 @@
         # This avoids waiting for the socket timeout.
         self.test_simple1()
 
+    def test_unicode_host(self):
+        server = xmlrpclib.ServerProxy("http://%s:%d/RPC2" % (ADDR, PORT))
+        self.assertEqual(server.add("a", "\xe9"), "a\xe9")
+
 class MultiPathServerTestCase(BaseServerTestCase):
     threadFunc = staticmethod(http_multi_server)
     request_count = 2

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep 23 01:31:11 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 23 Sep 2011 01:31:11 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_3=2E2=3A_Issue_=2312931=3A_Add_a_test_with_Unicode_URI?=
 =?utf8?q?_to_test=5Fxmlrpc?=
Message-ID: <E1R6sjX-0007BW-3M@dinsdale.python.org>

http://hg.python.org/cpython/rev/3b46f2e2d280
changeset:   72452:3b46f2e2d280
parent:      72449:a06ef7ab7321
parent:      72451:5ceab07bcd02
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 23 01:31:04 2011 +0200
summary:
  Merge 3.2: Issue #12931: Add a test with Unicode URI to test_xmlrpc

files:
  Lib/test/test_xmlrpc.py |  4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py
--- a/Lib/test/test_xmlrpc.py
+++ b/Lib/test/test_xmlrpc.py
@@ -584,6 +584,10 @@
         # This avoids waiting for the socket timeout.
         self.test_simple1()
 
+    def test_unicode_host(self):
+        server = xmlrpclib.ServerProxy("http://%s:%d/RPC2" % (ADDR, PORT))
+        self.assertEqual(server.add("a", "\xe9"), "a\xe9")
+
 class MultiPathServerTestCase(BaseServerTestCase):
     threadFunc = staticmethod(http_multi_server)
     request_count = 2

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Fri Sep 23 05:21:01 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Fri, 23 Sep 2011 05:21:01 +0200
Subject: [Python-checkins] Daily reference leaks (3b46f2e2d280): sum=0
Message-ID: <E1R6wJx-0006Gg-5k@ap.vmr.nerim.net>

results for 3b46f2e2d280 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogdBxCiw', '-x']

From python-checkins at python.org  Fri Sep 23 18:59:13 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 23 Sep 2011 18:59:13 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzc3MzI6?=
 =?utf8?q?_Don=27t_open_a_directory_as_a_file_anymore_while_importing_a?=
Message-ID: <E1R795l-0004o0-38@dinsdale.python.org>

http://hg.python.org/cpython/rev/125887a41a6f
changeset:   72453:125887a41a6f
branch:      3.2
parent:      72451:5ceab07bcd02
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 23 18:54:40 2011 +0200
summary:
  Issue #7732: Don't open a directory as a file anymore while importing a
module. Ignore the direcotry if its name matchs the module name (e.g.
"__init__.py") and raise a ImportError instead.

files:
  Lib/test/test_import.py |  9 +++++++++
  Misc/NEWS               |  6 +++++-
  Python/import.c         |  9 ++++++++-
  3 files changed, 22 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_import.py b/Lib/test/test_import.py
--- a/Lib/test/test_import.py
+++ b/Lib/test/test_import.py
@@ -139,6 +139,15 @@
             self.assertIs(orig_path, new_os.path)
             self.assertIsNot(orig_getenv, new_os.getenv)
 
+    def test_bug7732(self):
+        source = TESTFN + '.py'
+        os.mkdir(source)
+        try:
+            self.assertRaisesRegex(ImportError, '^No module',
+                imp.find_module, TESTFN, ["."])
+        finally:
+            os.rmdir(source)
+
     def test_module_with_large_stack(self, module='longlist'):
         # Regression test for http://bugs.python.org/issue561858.
         filename = module + '.py'
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,10 @@
 Core and Builtins
 -----------------
 
+- Issue #7732: Don't open a directory as a file anymore while importing a
+  module. Ignore the direcotry if its name matchs the module name (e.g.
+  "__init__.py") and raise a ImportError instead.
+
 - Issue #13021: Missing decref on an error path.  Thanks to Suman Saha for
   finding the bug and providing a patch.
 
@@ -77,7 +81,7 @@
 
 Extension Modules
 -----------------
- 
+
 - Issue #13022: Fix: _multiprocessing.recvfd() doesn't check that
   file descriptor was actually received.
 
diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -1763,6 +1763,7 @@
         saved_namelen = namelen;
 #endif /* PYOS_OS2 */
         for (fdp = _PyImport_Filetab; fdp->suffix != NULL; fdp++) {
+            struct stat statbuf;
 #if defined(PYOS_OS2) && defined(HAVE_DYNAMIC_LOADING)
             /* OS/2 limits DLLs to 8 character names (w/o
                extension)
@@ -1791,10 +1792,16 @@
             strcpy(buf+len, fdp->suffix);
             if (Py_VerboseFlag > 1)
                 PySys_WriteStderr("# trying %s\n", buf);
+
             filemode = fdp->mode;
             if (filemode[0] == 'U')
                 filemode = "r" PY_STDIOTEXTMODE;
-            fp = fopen(buf, filemode);
+
+            if (stat(buf, &statbuf) == 0 && S_ISDIR(statbuf.st_mode))
+                /* it's a directory */
+                fp = NULL;
+            else
+                fp = fopen(buf, filemode);
             if (fp != NULL) {
                 if (case_ok(buf, len, namelen, name))
                     break;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep 23 18:59:13 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 23 Sep 2011 18:59:13 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_3=2E2=3A_Issue_=237732=3A_Don=27t_open_a_directory_as_?=
 =?utf8?q?a_file_anymore_while?=
Message-ID: <E1R795l-0004o5-Vw@dinsdale.python.org>

http://hg.python.org/cpython/rev/8c6fea5794b2
changeset:   72454:8c6fea5794b2
parent:      72452:3b46f2e2d280
parent:      72453:125887a41a6f
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 23 18:59:08 2011 +0200
summary:
  Merge 3.2: Issue #7732: Don't open a directory as a file anymore while
importing a module. Ignore the direcotry if its name matchs the module name
(e.g.  "__init__.py") and raise a ImportError instead.

files:
  Lib/test/test_import.py |  9 +++++++++
  Misc/NEWS               |  6 +++++-
  Python/import.c         |  9 +++++++++
  3 files changed, 23 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_import.py b/Lib/test/test_import.py
--- a/Lib/test/test_import.py
+++ b/Lib/test/test_import.py
@@ -139,6 +139,15 @@
             self.assertIs(orig_path, new_os.path)
             self.assertIsNot(orig_getenv, new_os.getenv)
 
+    def test_bug7732(self):
+        source = TESTFN + '.py'
+        os.mkdir(source)
+        try:
+            self.assertRaisesRegex(ImportError, '^No module',
+                imp.find_module, TESTFN, ["."])
+        finally:
+            os.rmdir(source)
+
     def test_module_with_large_stack(self, module='longlist'):
         # Regression test for http://bugs.python.org/issue561858.
         filename = module + '.py'
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,10 @@
 Core and Builtins
 -----------------
 
+- Issue #7732: Don't open a directory as a file anymore while importing a
+  module. Ignore the direcotry if its name matchs the module name (e.g.
+  "__init__.py") and raise a ImportError instead.
+
 - Issue #13021: Missing decref on an error path.  Thanks to Suman Saha for
   finding the bug and providing a patch.
 
@@ -291,7 +295,7 @@
   ZLIB_RUNTIME_VERSION, in the zlib module. Patch by Torsten Landschoff.
 
 - Issue #12959: Add collections.ChainMap to collections.__all__.
- 
+
 - Issue #8933: distutils' PKG-INFO files and packaging's METADATA files will
   now correctly report Metadata-Version: 1.1 instead of 1.0 if a Classifier or
   Download-URL field is present.
diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -1892,6 +1892,8 @@
         }
 
         for (fdp = _PyImport_Filetab; fdp->suffix != NULL; fdp++) {
+            struct stat statbuf;
+
             filemode = fdp->mode;
             if (filemode[0] == 'U')
                 filemode = "r" PY_STDIOTEXTMODE;
@@ -1905,6 +1907,13 @@
             if (Py_VerboseFlag > 1)
                 PySys_FormatStderr("# trying %R\n", filename);
 
+            if (_Py_stat(filename, &statbuf) == 0 &&         /* it exists */
+                S_ISDIR(statbuf.st_mode))           /* it's a directory */
+            {
+                Py_DECREF(filename);
+                continue;
+            }
+
             fp = _Py_fopen(filename, filemode);
             if (fp == NULL) {
                 Py_DECREF(filename);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep 23 19:23:31 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Fri, 23 Sep 2011 19:23:31 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_disable_unused_result_warni?=
	=?utf8?q?ngs_when_possible?=
Message-ID: <E1R79TH-00089F-V2@dinsdale.python.org>

http://hg.python.org/cpython/rev/e5dade9bf96f
changeset:   72455:e5dade9bf96f
user:        Benjamin Peterson <benjamin at python.org>
date:        Fri Sep 23 13:23:22 2011 -0400
summary:
  disable unused result warnings when possible

files:
  configure    |  43 ++++++++++++++++++++++++++++++++++++++++
  configure.in |  22 ++++++++++++++++++++
  2 files changed, 65 insertions(+), 0 deletions(-)


diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -5561,6 +5561,49 @@
       BASECFLAGS="$BASECFLAGS -fno-strict-aliasing"
     fi
 
+    { $as_echo "$as_me:${as_lineno-$LINENO}: checking if we can turn off $CC unused result warning" >&5
+$as_echo_n "checking if we can turn off $CC unused result warning... " >&6; }
+     ac_save_cc="$CC"
+     CC="$CC -Wunused-result -Werror"
+     save_CFLAGS="$CFLAGS"
+     if ${ac_cv_disable_unused_result_warning+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+
+
+int
+main ()
+{
+
+  ;
+  return 0;
+}
+
+_ACEOF
+if ac_fn_c_try_compile "$LINENO"; then :
+
+           ac_cv_disable_unused_result_warning=yes
+
+else
+
+           ac_cv_disable_unused_result_warning=no
+
+fi
+rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
+fi
+
+     CFLAGS="$save_CFLAGS"
+     CC="$ac_save_cc"
+    { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_disable_unused_result_warning" >&5
+$as_echo "$ac_cv_disable_unused_result_warning" >&6; }
+
+    if test $ac_cv_disable_unused_result_warning = yes
+    then
+      BASECFLAGS="$BASECFLAGS -Wno-unused-result"
+    fi
+
     # if using gcc on alpha, use -mieee to get (near) full IEEE 754
     # support.  Without this, treatment of subnormals doesn't follow
     # the standard.
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -981,6 +981,28 @@
       BASECFLAGS="$BASECFLAGS -fno-strict-aliasing"
     fi
 
+    AC_MSG_CHECKING(if we can turn off $CC unused result warning)
+     ac_save_cc="$CC"
+     CC="$CC -Wunused-result -Werror"
+     save_CFLAGS="$CFLAGS"
+     AC_CACHE_VAL(ac_cv_disable_unused_result_warning,
+       AC_COMPILE_IFELSE(
+         [
+	   AC_LANG_PROGRAM([[]], [[]])
+	 ],[
+           ac_cv_disable_unused_result_warning=yes
+	 ],[
+           ac_cv_disable_unused_result_warning=no
+	 ]))
+     CFLAGS="$save_CFLAGS"
+     CC="$ac_save_cc"
+    AC_MSG_RESULT($ac_cv_disable_unused_result_warning)
+
+    if test $ac_cv_disable_unused_result_warning = yes
+    then
+      BASECFLAGS="$BASECFLAGS -Wno-unused-result"
+    fi
+
     # if using gcc on alpha, use -mieee to get (near) full IEEE 754
     # support.  Without this, treatment of subnormals doesn't follow
     # the standard.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep 23 19:37:21 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 23 Sep 2011 19:37:21 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzc3MzI6?=
 =?utf8?q?_Fix_a_crash_on_importing_a_module_if_a_directory_has_the_same_n?=
 =?utf8?q?ame?=
Message-ID: <E1R79gf-0001YV-Uh@dinsdale.python.org>

http://hg.python.org/cpython/rev/0f5b64630fda
changeset:   72456:0f5b64630fda
branch:      2.7
parent:      72450:c02e790c4535
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 23 19:37:03 2011 +0200
summary:
  Issue #7732: Fix a crash on importing a module if a directory has the same name
than a Python module (e.g. "__init__.py"): don't close the file twice.

PyFile_FromFile() does also close the file if PyString_FromString() failed. It
did already close the file on fill_file_fields() error (e.g. if the file is a
directory).

files:
  Doc/c-api/file.rst      |   3 +-
  Lib/test/test_import.py |   8 +++++++
  Misc/NEWS               |   3 ++
  Objects/fileobject.c    |  32 +++++++++++++++++-----------
  Python/import.c         |   4 +--
  5 files changed, 33 insertions(+), 17 deletions(-)


diff --git a/Doc/c-api/file.rst b/Doc/c-api/file.rst
--- a/Doc/c-api/file.rst
+++ b/Doc/c-api/file.rst
@@ -55,7 +55,8 @@
 
    Create a new :ctype:`PyFileObject` from the already-open standard C file
    pointer, *fp*.  The function *close* will be called when the file should be
-   closed.  Return *NULL* on failure.
+   closed.  Return *NULL* and close the file using *close* on failure.
+   *close* is optional and can be set to *NULL*.
 
 
 .. cfunction:: FILE* PyFile_AsFile(PyObject \*p)
diff --git a/Lib/test/test_import.py b/Lib/test/test_import.py
--- a/Lib/test/test_import.py
+++ b/Lib/test/test_import.py
@@ -265,6 +265,14 @@
             """))
         script_helper.assert_python_ok(testfn)
 
+    def test_bug7732(self):
+        source = TESTFN + '.py'
+        os.mkdir(source)
+        try:
+            self.assertRaises(IOError, imp.find_module, TESTFN, ["."])
+        finally:
+            os.rmdir(source)
+
 
 class PycRewritingTests(unittest.TestCase):
     # Test that the `co_filename` attribute on code objects always points
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -9,6 +9,9 @@
 Core and Builtins
 -----------------
 
+- Issue #7732: Fix a crash on importing a module if a directory has the same
+  name than a Python module (e.g. "__init__.py"): don't close the file twice.
+
 - Issue #12973: Fix overflow checks that invoked undefined behaviour in
   int.__pow__.  These overflow checks were causing int.__pow__ to produce
   incorrect results with recent versions of Clang, as a result of the
diff --git a/Objects/fileobject.c b/Objects/fileobject.c
--- a/Objects/fileobject.c
+++ b/Objects/fileobject.c
@@ -468,28 +468,34 @@
 PyObject *
 PyFile_FromFile(FILE *fp, char *name, char *mode, int (*close)(FILE *))
 {
-    PyFileObject *f = (PyFileObject *)PyFile_Type.tp_new(&PyFile_Type,
-                                                         NULL, NULL);
-    if (f != NULL) {
-        PyObject *o_name = PyString_FromString(name);
-        if (o_name == NULL)
-            return NULL;
-        if (fill_file_fields(f, fp, o_name, mode, close) == NULL) {
-            Py_DECREF(f);
-            f = NULL;
-        }
+    PyFileObject *f;
+    PyObject *o_name;
+
+    f = (PyFileObject *)PyFile_Type.tp_new(&PyFile_Type, NULL, NULL);
+    if (f == NULL)
+        return NULL;
+    o_name = PyString_FromString(name);
+    if (o_name == NULL) {
+        if (close != NULL && fp != NULL)
+            close(fp);
+        Py_DECREF(f);
+        return NULL;
+    }
+    if (fill_file_fields(f, fp, o_name, mode, close) == NULL) {
+        Py_DECREF(f);
         Py_DECREF(o_name);
+        return NULL;
     }
-    return (PyObject *) f;
+    Py_DECREF(o_name);
+    return (PyObject *)f;
 }
 
 PyObject *
 PyFile_FromString(char *name, char *mode)
 {
-    extern int fclose(FILE *);
     PyFileObject *f;
 
-    f = (PyFileObject *)PyFile_FromFile((FILE *)NULL, name, mode, fclose);
+    f = (PyFileObject *)PyFile_FromFile((FILE *)NULL, name, mode, NULL);
     if (f != NULL) {
         if (open_the_file(f, name, mode) == NULL) {
             Py_DECREF(f);
diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -2845,10 +2845,8 @@
         return NULL;
     if (fp != NULL) {
         fob = PyFile_FromFile(fp, pathname, fdp->mode, fclose);
-        if (fob == NULL) {
-            fclose(fp);
+        if (fob == NULL)
             return NULL;
-        }
     }
     else {
         fob = Py_None;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep 23 19:53:10 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Fri, 23 Sep 2011 19:53:10 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_fix_compiler_co?=
 =?utf8?q?mpliant_about_=5C0_not_being_an_opcode?=
Message-ID: <E1R79vy-0003Ke-Sk@dinsdale.python.org>

http://hg.python.org/cpython/rev/db8027df8458
changeset:   72457:db8027df8458
branch:      3.2
parent:      72453:125887a41a6f
user:        Benjamin Peterson <benjamin at python.org>
date:        Fri Sep 23 13:41:41 2011 -0400
summary:
  fix compiler compliant about \0 not being an opcode

files:
  Modules/_pickle.c |  11 +++++------
  1 files changed, 5 insertions(+), 6 deletions(-)


diff --git a/Modules/_pickle.c b/Modules/_pickle.c
--- a/Modules/_pickle.c
+++ b/Modules/_pickle.c
@@ -5298,13 +5298,12 @@
         case STOP:
             break;
 
-        case '\0':
-            PyErr_SetNone(PyExc_EOFError);
-            return NULL;
-
         default:
-            PyErr_Format(UnpicklingError,
-                         "invalid load key, '%c'.", s[0]);
+            if (s[0] == '\0')
+                PyErr_SetNone(PyExc_EOFError);
+            else
+                PyErr_Format(UnpicklingError,
+                             "invalid load key, '%c'.", s[0]);
             return NULL;
         }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep 23 19:53:11 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Fri, 23 Sep 2011 19:53:11 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_merge_3=2E2?=
Message-ID: <E1R79vz-0003Ko-Lj@dinsdale.python.org>

http://hg.python.org/cpython/rev/493886a640b6
changeset:   72458:493886a640b6
parent:      72455:e5dade9bf96f
parent:      72457:db8027df8458
user:        Benjamin Peterson <benjamin at python.org>
date:        Fri Sep 23 13:53:06 2011 -0400
summary:
  merge 3.2

files:
  Modules/_pickle.c |  11 +++++------
  1 files changed, 5 insertions(+), 6 deletions(-)


diff --git a/Modules/_pickle.c b/Modules/_pickle.c
--- a/Modules/_pickle.c
+++ b/Modules/_pickle.c
@@ -5318,13 +5318,12 @@
         case STOP:
             break;
 
-        case '\0':
-            PyErr_SetNone(PyExc_EOFError);
-            return NULL;
-
         default:
-            PyErr_Format(UnpicklingError,
-                         "invalid load key, '%c'.", s[0]);
+            if (s[0] == '\0')
+                PyErr_SetNone(PyExc_EOFError);
+            else
+                PyErr_Format(UnpicklingError,
+                             "invalid load key, '%c'.", s[0]);
             return NULL;
         }
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Sat Sep 24 05:19:26 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sat, 24 Sep 2011 05:19:26 +0200
Subject: [Python-checkins] Daily reference leaks (493886a640b6): sum=0
Message-ID: <E1R7Ily-0000as-7I@ap.vmr.nerim.net>

results for 493886a640b6 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogYN8_Mx', '-x']

From python-checkins at python.org  Sat Sep 24 10:00:35 2011
From: python-checkins at python.org (mark.dickinson)
Date: Sat, 24 Sep 2011 10:00:35 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyOTcz?=
 =?utf8?q?=3A_Fix_itertools_bug_caused_by_signed_integer_overflow=2E__Than?=
 =?utf8?q?ks?=
Message-ID: <E1R7NA3-0007s6-3i@dinsdale.python.org>

http://hg.python.org/cpython/rev/b378864d8ff3
changeset:   72459:b378864d8ff3
branch:      3.2
parent:      72457:db8027df8458
user:        Mark Dickinson <mdickinson at enthought.com>
date:        Sat Sep 24 08:56:09 2011 +0100
summary:
  Issue #12973: Fix itertools bug caused by signed integer overflow.  Thanks Stefan Krah.

files:
  Misc/NEWS                 |  6 +++---
  Modules/itertoolsmodule.c |  4 +++-
  2 files changed, 6 insertions(+), 4 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -17,9 +17,9 @@
 - Issue #13021: Missing decref on an error path.  Thanks to Suman Saha for
   finding the bug and providing a patch.
 
-- Issue #12973: Fix overflow check that relied on undefined behaviour in
-  list_repeat.  This bug caused test_list to fail with recent versions
-  of Clang.
+- Issue #12973: Fix overflow checks that relied on undefined behaviour in
+  list_repeat (listobject.c) and islice_next (itertoolsmodule.c).  These bugs
+  caused test failures with recent versions of Clang.
 
 - Issue #12802: the Windows error ERROR_DIRECTORY (numbered 267) is now
   mapped to POSIX errno ENOTDIR (previously EINVAL).
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -1234,7 +1234,9 @@
         return NULL;
     lz->cnt++;
     oldnext = lz->next;
-    lz->next += lz->step;
+    /* The (size_t) cast below avoids the danger of undefined
+       behaviour from signed integer overflow. */
+    lz->next += (size_t)lz->step;
     if (lz->next < oldnext || (stop != -1 && lz->next > stop))
         lz->next = stop;
     return item;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 24 10:00:35 2011
From: python-checkins at python.org (mark.dickinson)
Date: Sat, 24 Sep 2011 10:00:35 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_=2312973_itertools_fix=2E?=
Message-ID: <E1R7NA3-0007sA-T3@dinsdale.python.org>

http://hg.python.org/cpython/rev/18eec56bcf29
changeset:   72460:18eec56bcf29
parent:      72458:493886a640b6
parent:      72459:b378864d8ff3
user:        Mark Dickinson <mdickinson at enthought.com>
date:        Sat Sep 24 08:57:00 2011 +0100
summary:
  Merge #12973 itertools fix.

files:
  Misc/NEWS                 |  6 +++---
  Modules/itertoolsmodule.c |  4 +++-
  2 files changed, 6 insertions(+), 4 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -17,9 +17,9 @@
 - Issue #13021: Missing decref on an error path.  Thanks to Suman Saha for
   finding the bug and providing a patch.
 
-- Issue #12973: Fix overflow check that relied on undefined behaviour in
-  list_repeat.  This bug caused test_list to fail with recent versions
-  of Clang.
+- Issue #12973: Fix overflow checks that relied on undefined behaviour in
+  list_repeat (listobject.c) and islice_next (itertoolsmodule.c).  These bugs
+  caused test failures with recent versions of Clang.
 
 - Issue #12904: os.utime, os.futimes, os.lutimes, and os.futimesat now write
   atime and mtime with nanosecond precision on modern POSIX platforms.
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -1234,7 +1234,9 @@
         return NULL;
     lz->cnt++;
     oldnext = lz->next;
-    lz->next += lz->step;
+    /* The (size_t) cast below avoids the danger of undefined
+       behaviour from signed integer overflow. */
+    lz->next += (size_t)lz->step;
     if (lz->next < oldnext || (stop != -1 && lz->next > stop))
         lz->next = stop;
     return item;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 24 10:02:35 2011
From: python-checkins at python.org (mark.dickinson)
Date: Sat, 24 Sep 2011 10:02:35 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Backport_issue_?=
 =?utf8?q?=2312973_itertools_fix_from_3=2Ex=2E?=
Message-ID: <E1R7NBz-00081Q-3n@dinsdale.python.org>

http://hg.python.org/cpython/rev/5a1cb8506cea
changeset:   72461:5a1cb8506cea
branch:      2.7
parent:      72456:0f5b64630fda
user:        Mark Dickinson <mdickinson at enthought.com>
date:        Sat Sep 24 09:01:16 2011 +0100
summary:
  Backport issue #12973 itertools fix from 3.x.

files:
  Misc/NEWS                 |  6 +++---
  Modules/itertoolsmodule.c |  4 +++-
  2 files changed, 6 insertions(+), 4 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -15,9 +15,9 @@
 - Issue #12973: Fix overflow checks that invoked undefined behaviour in
   int.__pow__.  These overflow checks were causing int.__pow__ to produce
   incorrect results with recent versions of Clang, as a result of the
-  compiler optimizing the check away.  Also fix similar overflow check
-  in list_repeat (which caused test_list to fail with recent versions
-  of Clang).
+  compiler optimizing the check away.  Also fix similar overflow checks
+  in list_repeat (listobject.c) and islice_next (itertoolsmodule.c).  These
+  bugs caused test failures with recent versions of Clang.
 
 - Issue #12266: Fix str.capitalize() to correctly uppercase/lowercase
   titlecased and cased non-letter characters.
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -1234,7 +1234,9 @@
         return NULL;
     lz->cnt++;
     oldnext = lz->next;
-    lz->next += lz->step;
+    /* The (size_t) cast below avoids the danger of undefined
+       behaviour from signed integer overflow. */
+    lz->next += (size_t)lz->step;
     if (lz->next < oldnext || (stop != -1 && lz->next > stop))
         lz->next = stop;
     return item;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 24 10:15:44 2011
From: python-checkins at python.org (mark.dickinson)
Date: Sat, 24 Sep 2011 10:15:44 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313012=3A_Allow_=27?=
 =?utf8?q?keepends=27_to_be_passed_as_a_keyword_argument_in?=
Message-ID: <E1R7NOi-0000ch-5p@dinsdale.python.org>

http://hg.python.org/cpython/rev/0b2660384d4e
changeset:   72462:0b2660384d4e
parent:      72460:18eec56bcf29
user:        Mark Dickinson <mdickinson at enthought.com>
date:        Sat Sep 24 09:14:39 2011 +0100
summary:
  Issue #13012: Allow 'keepends' to be passed as a keyword argument in str.splitlines, bytes.splitlines and bytearray.splitlines.

files:
  Lib/test/buffer_tests.py    |   8 +++++++-
  Lib/test/string_tests.py    |  14 +++++++++++---
  Lib/test/test_userstring.py |   4 ++--
  Misc/NEWS                   |   4 ++++
  Objects/bytearrayobject.c   |  10 ++++++----
  Objects/bytesobject.c       |   8 +++++---
  Objects/unicodeobject.c     |  10 ++++++----
  7 files changed, 41 insertions(+), 17 deletions(-)


diff --git a/Lib/test/buffer_tests.py b/Lib/test/buffer_tests.py
--- a/Lib/test/buffer_tests.py
+++ b/Lib/test/buffer_tests.py
@@ -200,7 +200,13 @@
                          self.marshal(b'abc\ndef\r\nghi\n\r').splitlines())
         self.assertEqual([b'', b'abc', b'def', b'ghi', b''],
                          self.marshal(b'\nabc\ndef\r\nghi\n\r').splitlines())
+        self.assertEqual([b'', b'abc', b'def', b'ghi', b''],
+                         self.marshal(b'\nabc\ndef\r\nghi\n\r').splitlines(False))
         self.assertEqual([b'\n', b'abc\n', b'def\r\n', b'ghi\n', b'\r'],
-                         self.marshal(b'\nabc\ndef\r\nghi\n\r').splitlines(1))
+                         self.marshal(b'\nabc\ndef\r\nghi\n\r').splitlines(True))
+        self.assertEqual([b'', b'abc', b'def', b'ghi', b''],
+                         self.marshal(b'\nabc\ndef\r\nghi\n\r').splitlines(keepends=False))
+        self.assertEqual([b'\n', b'abc\n', b'def\r\n', b'ghi\n', b'\r'],
+                         self.marshal(b'\nabc\ndef\r\nghi\n\r').splitlines(keepends=True))
 
         self.assertRaises(TypeError, self.marshal(b'abc').splitlines, 42, 42)
diff --git a/Lib/test/string_tests.py b/Lib/test/string_tests.py
--- a/Lib/test/string_tests.py
+++ b/Lib/test/string_tests.py
@@ -47,11 +47,12 @@
             return obj
 
     # check that obj.method(*args) returns result
-    def checkequal(self, result, obj, methodname, *args):
+    def checkequal(self, result, obj, methodname, *args, **kwargs):
         result = self.fixtype(result)
         obj = self.fixtype(obj)
         args = self.fixtype(args)
-        realresult = getattr(obj, methodname)(*args)
+        kwargs = self.fixtype(kwargs)
+        realresult = getattr(obj, methodname)(*args, **kwargs)
         self.assertEqual(
             result,
             realresult
@@ -908,7 +909,14 @@
         self.checkequal(['abc', 'def', 'ghi'], "abc\ndef\r\nghi\n", 'splitlines')
         self.checkequal(['abc', 'def', 'ghi', ''], "abc\ndef\r\nghi\n\r", 'splitlines')
         self.checkequal(['', 'abc', 'def', 'ghi', ''], "\nabc\ndef\r\nghi\n\r", 'splitlines')
-        self.checkequal(['\n', 'abc\n', 'def\r\n', 'ghi\n', '\r'], "\nabc\ndef\r\nghi\n\r", 'splitlines', 1)
+        self.checkequal(['', 'abc', 'def', 'ghi', ''],
+                        "\nabc\ndef\r\nghi\n\r", 'splitlines', False)
+        self.checkequal(['\n', 'abc\n', 'def\r\n', 'ghi\n', '\r'],
+                        "\nabc\ndef\r\nghi\n\r", 'splitlines', True)
+        self.checkequal(['', 'abc', 'def', 'ghi', ''], "\nabc\ndef\r\nghi\n\r",
+                        'splitlines', keepends=False)
+        self.checkequal(['\n', 'abc\n', 'def\r\n', 'ghi\n', '\r'],
+                        "\nabc\ndef\r\nghi\n\r", 'splitlines', keepends=True)
 
         self.checkraises(TypeError, 'abc', 'splitlines', 42, 42)
 
diff --git a/Lib/test/test_userstring.py b/Lib/test/test_userstring.py
--- a/Lib/test/test_userstring.py
+++ b/Lib/test/test_userstring.py
@@ -17,11 +17,11 @@
     # Overwrite the three testing methods, because UserString
     # can't cope with arguments propagated to UserString
     # (and we don't test with subclasses)
-    def checkequal(self, result, object, methodname, *args):
+    def checkequal(self, result, object, methodname, *args, **kwargs):
         result = self.fixtype(result)
         object = self.fixtype(object)
         # we don't fix the arguments, because UserString can't cope with it
-        realresult = getattr(object, methodname)(*args)
+        realresult = getattr(object, methodname)(*args, **kwargs)
         self.assertEqual(
             result,
             realresult
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,10 @@
 Core and Builtins
 -----------------
 
+- Issue #13012: The 'keepends' parameter to str.splitlines may now be passed
+  as a keyword argument:  "my_string.splitlines(keepends=True)".  The same
+  change also applies to bytes.splitlines and bytearray.splitlines.
+
 - Issue #7732: Don't open a directory as a file anymore while importing a
   module. Ignore the direcotry if its name matchs the module name (e.g.
   "__init__.py") and raise a ImportError instead.
diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c
--- a/Objects/bytearrayobject.c
+++ b/Objects/bytearrayobject.c
@@ -2608,11 +2608,13 @@
 is given and true.");
 
 static PyObject*
-bytearray_splitlines(PyObject *self, PyObject *args)
+bytearray_splitlines(PyObject *self, PyObject *args, PyObject *kwds)
 {
+    static char *kwlist[] = {"keepends", 0};
     int keepends = 0;
 
-    if (!PyArg_ParseTuple(args, "|i:splitlines", &keepends))
+    if (!PyArg_ParseTupleAndKeywords(args, kwds, "|i:splitlines",
+                                     kwlist, &keepends))
         return NULL;
 
     return stringlib_splitlines(
@@ -2801,8 +2803,8 @@
     {"rsplit", (PyCFunction)bytearray_rsplit, METH_VARARGS, rsplit__doc__},
     {"rstrip", (PyCFunction)bytearray_rstrip, METH_VARARGS, rstrip__doc__},
     {"split", (PyCFunction)bytearray_split, METH_VARARGS, split__doc__},
-    {"splitlines", (PyCFunction)bytearray_splitlines, METH_VARARGS,
-     splitlines__doc__},
+    {"splitlines", (PyCFunction)bytearray_splitlines,
+     METH_VARARGS | METH_KEYWORDS, splitlines__doc__},
     {"startswith", (PyCFunction)bytearray_startswith, METH_VARARGS ,
      startswith__doc__},
     {"strip", (PyCFunction)bytearray_strip, METH_VARARGS, strip__doc__},
diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -2312,11 +2312,13 @@
 is given and true.");
 
 static PyObject*
-bytes_splitlines(PyObject *self, PyObject *args)
+bytes_splitlines(PyObject *self, PyObject *args, PyObject *kwds)
 {
+    static char *kwlist[] = {"keepends", 0};
     int keepends = 0;
 
-    if (!PyArg_ParseTuple(args, "|i:splitlines", &keepends))
+    if (!PyArg_ParseTupleAndKeywords(args, kwds, "|i:splitlines",
+                                     kwlist, &keepends))
         return NULL;
 
     return stringlib_splitlines(
@@ -2458,7 +2460,7 @@
     {"rsplit", (PyCFunction)bytes_rsplit, METH_VARARGS, rsplit__doc__},
     {"rstrip", (PyCFunction)bytes_rstrip, METH_VARARGS, rstrip__doc__},
     {"split", (PyCFunction)bytes_split, METH_VARARGS, split__doc__},
-    {"splitlines", (PyCFunction)bytes_splitlines, METH_VARARGS,
+    {"splitlines", (PyCFunction)bytes_splitlines, METH_VARARGS | METH_KEYWORDS,
      splitlines__doc__},
     {"startswith", (PyCFunction)bytes_startswith, METH_VARARGS,
      startswith__doc__},
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -8881,11 +8881,13 @@
 is given and true.");
 
 static PyObject*
-unicode_splitlines(PyUnicodeObject *self, PyObject *args)
-{
+unicode_splitlines(PyUnicodeObject *self, PyObject *args, PyObject *kwds)
+{
+    static char *kwlist[] = {"keepends", 0};
     int keepends = 0;
 
-    if (!PyArg_ParseTuple(args, "|i:splitlines", &keepends))
+    if (!PyArg_ParseTupleAndKeywords(args, kwds, "|i:splitlines",
+                                     kwlist, &keepends))
         return NULL;
 
     return PyUnicode_Splitlines((PyObject *)self, keepends);
@@ -9273,7 +9275,7 @@
     {"rjust", (PyCFunction) unicode_rjust, METH_VARARGS, rjust__doc__},
     {"rstrip", (PyCFunction) unicode_rstrip, METH_VARARGS, rstrip__doc__},
     {"rpartition", (PyCFunction) unicode_rpartition, METH_O, rpartition__doc__},
-    {"splitlines", (PyCFunction) unicode_splitlines, METH_VARARGS, splitlines__doc__},
+    {"splitlines", (PyCFunction) unicode_splitlines, METH_VARARGS | METH_KEYWORDS, splitlines__doc__},
     {"strip", (PyCFunction) unicode_strip, METH_VARARGS, strip__doc__},
     {"swapcase", (PyCFunction) unicode_swapcase, METH_NOARGS, swapcase__doc__},
     {"translate", (PyCFunction) unicode_translate, METH_O, translate__doc__},

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 24 13:40:01 2011
From: python-checkins at python.org (local)
Date: Sat, 24 Sep 2011 13:40:01 +0200
Subject: [Python-checkins] =?utf8?q?hooks=3A_Allow_configuring_a_different?=
 =?utf8?q?_sender_=28needed_for_stackless_repo=2C_where_the?=
Message-ID: <E1R7QaP-0005ym-6f@dinsdale.python.org>

http://hg.python.org/hooks/rev/2e98611111f1
changeset:   77:2e98611111f1
user:        Georg Brandl <georg at python.org>
date:        Sat Sep 24 13:40:00 2011 +0200
summary:
  Allow configuring a different sender (needed for stackless repo, where the notification goes to an address at a different domain).

files:
  mail.py |  5 ++++-
  1 files changed, 4 insertions(+), 1 deletions(-)


diff --git a/mail.py b/mail.py
--- a/mail.py
+++ b/mail.py
@@ -76,7 +76,10 @@
     if to is None:
         print 'no email address configured'
         return False
-    sender = '%s <%s>' % (user, to)
+    from_ = ui.config('mail', 'sender', None)
+    if from_ is None:
+        from_ = to
+    sender = '%s <%s>' % (user, from_)
 
     prefixes = [path]
 

-- 
Repository URL: http://hg.python.org/hooks

From python-checkins at python.org  Sat Sep 24 17:25:17 2011
From: python-checkins at python.org (mark.dickinson)
Date: Sat, 24 Sep 2011 17:25:17 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_typo_in_comment=3A_=5FP?=
 =?utf8?q?yHash=5FDouble_-=3E_=5FPy=5FHashDouble=2E?=
Message-ID: <E1R7U6P-0007es-E2@dinsdale.python.org>

http://hg.python.org/cpython/rev/ac79caff7d6d
changeset:   72463:ac79caff7d6d
user:        Mark Dickinson <mdickinson at enthought.com>
date:        Sat Sep 24 16:24:56 2011 +0100
summary:
  Fix typo in comment: _PyHash_Double -> _Py_HashDouble.

files:
  Include/pyport.h |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Include/pyport.h b/Include/pyport.h
--- a/Include/pyport.h
+++ b/Include/pyport.h
@@ -132,7 +132,7 @@
 #endif
 
 /* Parameters used for the numeric hash implementation.  See notes for
-   _PyHash_Double in Objects/object.c.  Numeric hashes are based on
+   _Py_HashDouble in Objects/object.c.  Numeric hashes are based on
    reduction modulo the prime 2**_PyHASH_BITS - 1. */
 
 #if SIZEOF_VOID_P >= 8

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 24 19:19:42 2011
From: python-checkins at python.org (mark.dickinson)
Date: Sat, 24 Sep 2011 19:19:42 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=231621=3A_Fix_undefi?=
 =?utf8?b?bmVkIGJlaGF2aW91ciBpbiBieXRlcy5fX2hhc2hfXywgc3RyLl9faGFzaF9fLA==?=
Message-ID: <E1R7Vt8-0008H4-Bb@dinsdale.python.org>

http://hg.python.org/cpython/rev/698fa089ce70
changeset:   72464:698fa089ce70
user:        Mark Dickinson <mdickinson at enthought.com>
date:        Sat Sep 24 18:18:40 2011 +0100
summary:
  Issue #1621: Fix undefined behaviour in bytes.__hash__, str.__hash__, tuple.__hash__, frozenset.__hash__ and set indexing operations.

files:
  Objects/bytesobject.c   |   8 ++++----
  Objects/dictobject.c    |   4 ++--
  Objects/setobject.c     |  20 ++++++++++----------
  Objects/tupleobject.c   |   9 +++++----
  Objects/unicodeobject.c |  10 +++++-----
  5 files changed, 26 insertions(+), 25 deletions(-)


diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -869,16 +869,16 @@
 {
     register Py_ssize_t len;
     register unsigned char *p;
-    register Py_hash_t x;
+    register Py_uhash_t x;
 
     if (a->ob_shash != -1)
         return a->ob_shash;
     len = Py_SIZE(a);
     p = (unsigned char *) a->ob_sval;
-    x = *p << 7;
+    x = (Py_uhash_t)*p << 7;
     while (--len >= 0)
-        x = (1000003*x) ^ *p++;
-    x ^= Py_SIZE(a);
+        x = (1000003U*x) ^ (Py_uhash_t)*p++;
+    x ^= (Py_uhash_t)Py_SIZE(a);
     if (x == -1)
         x = -2;
     a->ob_shash = x;
diff --git a/Objects/dictobject.c b/Objects/dictobject.c
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -418,7 +418,7 @@
         mp->ma_lookup = lookdict;
         return lookdict(mp, key, hash);
     }
-    i = hash & mask;
+    i = (size_t)hash & mask;
     ep = &ep0[i];
     if (ep->me_key == NULL || ep->me_key == key)
         return ep;
@@ -572,7 +572,7 @@
     register PyDictEntry *ep;
 
     MAINTAIN_TRACKING(mp, key, value);
-    i = hash & mask;
+    i = (size_t)hash & mask;
     ep = &ep0[i];
     for (perturb = hash; ep->me_key != NULL; perturb >>= PERTURB_SHIFT) {
         i = (i << 2) + i + perturb + 1;
diff --git a/Objects/setobject.c b/Objects/setobject.c
--- a/Objects/setobject.c
+++ b/Objects/setobject.c
@@ -77,7 +77,7 @@
 static setentry *
 set_lookkey(PySetObject *so, PyObject *key, register Py_hash_t hash)
 {
-    register Py_ssize_t i;
+    register size_t i;
     register size_t perturb;
     register setentry *freeslot;
     register size_t mask = so->mask;
@@ -86,7 +86,7 @@
     register int cmp;
     PyObject *startkey;
 
-    i = hash & mask;
+    i = (size_t)hash & mask;
     entry = &table[i];
     if (entry->key == NULL || entry->key == key)
         return entry;
@@ -159,7 +159,7 @@
 static setentry *
 set_lookkey_unicode(PySetObject *so, PyObject *key, register Py_hash_t hash)
 {
-    register Py_ssize_t i;
+    register size_t i;
     register size_t perturb;
     register setentry *freeslot;
     register size_t mask = so->mask;
@@ -174,7 +174,7 @@
         so->lookup = set_lookkey;
         return set_lookkey(so, key, hash);
     }
-    i = hash & mask;
+    i = (size_t)hash & mask;
     entry = &table[i];
     if (entry->key == NULL || entry->key == key)
         return entry;
@@ -256,7 +256,7 @@
     setentry *table = so->table;
     register setentry *entry;
 
-    i = hash & mask;
+    i = (size_t)hash & mask;
     entry = &table[i];
     for (perturb = hash; entry->key != NULL; perturb >>= PERTURB_SHIFT) {
         i = (i << 2) + i + perturb + 1;
@@ -770,14 +770,14 @@
 frozenset_hash(PyObject *self)
 {
     PySetObject *so = (PySetObject *)self;
-    Py_hash_t h, hash = 1927868237L;
+    Py_uhash_t h, hash = 1927868237U;
     setentry *entry;
     Py_ssize_t pos = 0;
 
     if (so->hash != -1)
         return so->hash;
 
-    hash *= PySet_GET_SIZE(self) + 1;
+    hash *= (Py_uhash_t)PySet_GET_SIZE(self) + 1;
     while (set_next(so, &pos, &entry)) {
         /* Work to increase the bit dispersion for closely spaced hash
            values.  The is important because some use cases have many
@@ -785,11 +785,11 @@
            hashes so that many distinct combinations collapse to only
            a handful of distinct hash values. */
         h = entry->hash;
-        hash ^= (h ^ (h << 16) ^ 89869747L)  * 3644798167u;
+        hash ^= (h ^ (h << 16) ^ 89869747U)  * 3644798167U;
     }
-    hash = hash * 69069L + 907133923L;
+    hash = hash * 69069U + 907133923U;
     if (hash == -1)
-        hash = 590923713L;
+        hash = 590923713U;
     so->hash = hash;
     return hash;
 }
diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c
--- a/Objects/tupleobject.c
+++ b/Objects/tupleobject.c
@@ -315,11 +315,12 @@
 static Py_hash_t
 tuplehash(PyTupleObject *v)
 {
-    register Py_hash_t x, y;
+    register Py_uhash_t x;
+    register Py_hash_t y;
     register Py_ssize_t len = Py_SIZE(v);
     register PyObject **p;
-    Py_hash_t mult = 1000003L;
-    x = 0x345678L;
+    Py_uhash_t mult = 1000003;
+    x = 0x345678;
     p = v->ob_item;
     while (--len >= 0) {
         y = PyObject_Hash(*p++);
@@ -330,7 +331,7 @@
         mult += (Py_hash_t)(82520L + len + len);
     }
     x += 97531L;
-    if (x == -1)
+    if (x == (Py_uhash_t)-1)
         x = -2;
     return x;
 }
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -7721,22 +7721,22 @@
 }
 
 /* Believe it or not, this produces the same value for ASCII strings
-   as string_hash(). */
+   as bytes_hash(). */
 static Py_hash_t
 unicode_hash(PyUnicodeObject *self)
 {
     Py_ssize_t len;
     Py_UNICODE *p;
-    Py_hash_t x;
+    Py_uhash_t x;
 
     if (self->hash != -1)
         return self->hash;
     len = Py_SIZE(self);
     p = self->str;
-    x = *p << 7;
+    x = (Py_uhash_t)*p << 7;
     while (--len >= 0)
-        x = (1000003*x) ^ *p++;
-    x ^= Py_SIZE(self);
+        x = (1000003U*x) ^ (Py_uhash_t)*p++;
+    x ^= (Py_uhash_t)Py_SIZE(self);
     if (x == -1)
         x = -2;
     self->hash = x;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 24 20:04:01 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Sat, 24 Sep 2011 20:04:01 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312981=3A_rewrite_m?=
 =?utf8?q?ultiprocessing=5F=7Bsendfd=2Crecvfd=7D_in_Python=2E?=
Message-ID: <E1R7Wa1-00037I-Lc@dinsdale.python.org>

http://hg.python.org/cpython/rev/95ee0df1e746
changeset:   72465:95ee0df1e746
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Sat Sep 24 20:04:29 2011 +0200
summary:
  Issue #12981: rewrite multiprocessing_{sendfd,recvfd} in Python.

files:
  Lib/multiprocessing/reduction.py           |   21 +-
  Modules/_multiprocessing/multiprocessing.c |  128 +---------
  Modules/_multiprocessing/multiprocessing.h |   10 -
  3 files changed, 19 insertions(+), 140 deletions(-)


diff --git a/Lib/multiprocessing/reduction.py b/Lib/multiprocessing/reduction.py
--- a/Lib/multiprocessing/reduction.py
+++ b/Lib/multiprocessing/reduction.py
@@ -39,6 +39,7 @@
 import sys
 import socket
 import threading
+import struct
 
 import _multiprocessing
 from multiprocessing import current_process
@@ -51,7 +52,8 @@
 #
 #
 
-if not(sys.platform == 'win32' or hasattr(_multiprocessing, 'recvfd')):
+if not(sys.platform == 'win32' or (hasattr(socket, 'CMSG_LEN') and
+                                   hasattr(socket, 'SCM_RIGHTS'))):
     raise ImportError('pickling of connections not supported')
 
 #
@@ -77,10 +79,23 @@
 
 else:
     def send_handle(conn, handle, destination_pid):
-        _multiprocessing.sendfd(conn.fileno(), handle)
+        with socket.fromfd(conn.fileno(), socket.AF_UNIX, socket.SOCK_STREAM) as s:
+            s.sendmsg([b'x'], [(socket.SOL_SOCKET, socket.SCM_RIGHTS,
+                                struct.pack("@i", handle))])
 
     def recv_handle(conn):
-        return _multiprocessing.recvfd(conn.fileno())
+        size = struct.calcsize("@i")
+        with socket.fromfd(conn.fileno(), socket.AF_UNIX, socket.SOCK_STREAM) as s:
+            msg, ancdata, flags, addr = s.recvmsg(1, socket.CMSG_LEN(size))
+            try:
+                cmsg_level, cmsg_type, cmsg_data = ancdata[0]
+                if (cmsg_level == socket.SOL_SOCKET and
+                    cmsg_type == socket.SCM_RIGHTS):
+                    return struct.unpack("@i", cmsg_data[:size])[0]
+            except (ValueError, IndexError, struct.error):
+                pass
+            raise RuntimeError('Invalid data received')
+
 
 #
 # Support for a per-process server thread which caches pickled handles
diff --git a/Modules/_multiprocessing/multiprocessing.c b/Modules/_multiprocessing/multiprocessing.c
--- a/Modules/_multiprocessing/multiprocessing.c
+++ b/Modules/_multiprocessing/multiprocessing.c
@@ -8,11 +8,6 @@
 
 #include "multiprocessing.h"
 
-#ifdef SCM_RIGHTS
-    #define HAVE_FD_TRANSFER 1
-#else
-    #define HAVE_FD_TRANSFER 0
-#endif
 
 PyObject *create_win32_namespace(void);
 
@@ -75,115 +70,7 @@
     return FALSE;
 }
 
-/*
- * Unix only
- */
-
-#else /* !MS_WINDOWS */
-
-#if HAVE_FD_TRANSFER
-
-/* Functions for transferring file descriptors between processes.
-   Reimplements some of the functionality of the fdcred
-   module at http://www.mca-ltd.com/resources/fdcred_1.tgz. */
-/* Based in http://resin.csoft.net/cgi-bin/man.cgi?section=3&topic=CMSG_DATA */
-
-static PyObject *
-multiprocessing_sendfd(PyObject *self, PyObject *args)
-{
-    int conn, fd, res;
-    struct iovec dummy_iov;
-    char dummy_char;
-    struct msghdr msg;
-    struct cmsghdr *cmsg;
-    union {
-        struct cmsghdr hdr;
-        unsigned char buf[CMSG_SPACE(sizeof(int))];
-    } cmsgbuf;
-
-    if (!PyArg_ParseTuple(args, "ii", &conn, &fd))
-        return NULL;
-
-    dummy_iov.iov_base = &dummy_char;
-    dummy_iov.iov_len = 1;
-
-    memset(&msg, 0, sizeof(msg));
-    msg.msg_control = &cmsgbuf.buf;
-    msg.msg_controllen = sizeof(cmsgbuf.buf);
-    msg.msg_iov = &dummy_iov;
-    msg.msg_iovlen = 1;
-
-    cmsg = CMSG_FIRSTHDR(&msg);
-    cmsg->cmsg_len = CMSG_LEN(sizeof(int));
-    cmsg->cmsg_level = SOL_SOCKET;
-    cmsg->cmsg_type = SCM_RIGHTS;
-    * (int *) CMSG_DATA(cmsg) = fd;
-
-    Py_BEGIN_ALLOW_THREADS
-    res = sendmsg(conn, &msg, 0);
-    Py_END_ALLOW_THREADS
-
-    if (res < 0)
-        return PyErr_SetFromErrno(PyExc_OSError);
-    Py_RETURN_NONE;
-}
-
-static PyObject *
-multiprocessing_recvfd(PyObject *self, PyObject *args)
-{
-    int conn, fd, res;
-    char dummy_char;
-    struct iovec dummy_iov;
-    struct msghdr msg = {0};
-    struct cmsghdr *cmsg;
-    union {
-        struct cmsghdr hdr;
-        unsigned char buf[CMSG_SPACE(sizeof(int))];
-    } cmsgbuf;
-
-    if (!PyArg_ParseTuple(args, "i", &conn))
-        return NULL;
-
-    dummy_iov.iov_base = &dummy_char;
-    dummy_iov.iov_len = 1;
-
-    memset(&msg, 0, sizeof(msg));
-    msg.msg_control = &cmsgbuf.buf;
-    msg.msg_controllen = sizeof(cmsgbuf.buf);
-    msg.msg_iov = &dummy_iov;
-    msg.msg_iovlen = 1;
-
-    cmsg = CMSG_FIRSTHDR(&msg);
-    cmsg->cmsg_level = SOL_SOCKET;
-    cmsg->cmsg_type = SCM_RIGHTS;
-    cmsg->cmsg_len = CMSG_LEN(sizeof(int));
-    msg.msg_controllen = cmsg->cmsg_len;
-
-    Py_BEGIN_ALLOW_THREADS
-    res = recvmsg(conn, &msg, 0);
-    Py_END_ALLOW_THREADS
-
-    if (res < 0)
-        return PyErr_SetFromErrno(PyExc_OSError);
-
-    if (msg.msg_controllen < CMSG_LEN(sizeof(int)) ||
-        (cmsg = CMSG_FIRSTHDR(&msg)) == NULL ||
-        cmsg->cmsg_level != SOL_SOCKET ||
-        cmsg->cmsg_type != SCM_RIGHTS ||
-        cmsg->cmsg_len < CMSG_LEN(sizeof(int))) {
-        /* If at least one control message is present, there should be
-           no room for any further data in the buffer. */
-        PyErr_SetString(PyExc_RuntimeError, "No file descriptor received");
-        return NULL;
-    }
-
-    fd = * (int *) CMSG_DATA(cmsg);
-    return Py_BuildValue("i", fd);
-}
-
-#endif /* HAVE_FD_TRANSFER */
-
-#endif /* !MS_WINDOWS */
+#endif /* MS_WINDOWS */
 
 
 /*
@@ -212,16 +99,6 @@
     {"address_of_buffer", multiprocessing_address_of_buffer, METH_O,
      "address_of_buffer(obj) -> int\n"
      "Return address of obj assuming obj supports buffer inteface"},
-#if HAVE_FD_TRANSFER
-    {"sendfd", multiprocessing_sendfd, METH_VARARGS,
-     "sendfd(sockfd, fd) -> None\n"
-     "Send file descriptor given by fd over the unix domain socket\n"
-     "whose file decriptor is sockfd"},
-    {"recvfd", multiprocessing_recvfd, METH_VARARGS,
-     "recvfd(sockfd) -> fd\n"
-     "Receive a file descriptor over a unix domain socket\n"
-     "whose file decriptor is sockfd"},
-#endif
     {NULL}
 };
 
@@ -319,9 +196,6 @@
 #ifdef HAVE_SEM_TIMEDWAIT
     ADD_FLAG(HAVE_SEM_TIMEDWAIT);
 #endif
-#ifdef HAVE_FD_TRANSFER
-    ADD_FLAG(HAVE_FD_TRANSFER);
-#endif
 #ifdef HAVE_BROKEN_SEM_GETVALUE
     ADD_FLAG(HAVE_BROKEN_SEM_GETVALUE);
 #endif
diff --git a/Modules/_multiprocessing/multiprocessing.h b/Modules/_multiprocessing/multiprocessing.h
--- a/Modules/_multiprocessing/multiprocessing.h
+++ b/Modules/_multiprocessing/multiprocessing.h
@@ -3,12 +3,6 @@
 
 #define PY_SSIZE_T_CLEAN
 
-#ifdef __sun
-/* The control message API is only available on Solaris
-   if XPG 4.2 or later is requested. */
-#define _XOPEN_SOURCE 500
-#endif
-
 #include "Python.h"
 #include "structmember.h"
 #include "pythread.h"
@@ -29,10 +23,6 @@
 #  define SEM_VALUE_MAX LONG_MAX
 #else
 #  include <fcntl.h>                 /* O_CREAT and O_EXCL */
-#  include <netinet/in.h>
-#  include <sys/socket.h>
-#  include <sys/uio.h>
-#  include <arpa/inet.h>             /* htonl() and ntohl() */
 #  if defined(HAVE_SEM_OPEN) && !defined(POSIX_SEMAPHORES_NOT_ENABLED)
 #    include <semaphore.h>
      typedef sem_t *SEM_HANDLE;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Sep 24 20:11:59 2011
From: python-checkins at python.org (mark.dickinson)
Date: Sat, 24 Sep 2011 20:11:59 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=231621=3A_Fix_undefi?=
 =?utf8?q?ned_behaviour_from_signed_overflow_in_get=5Finteger?=
Message-ID: <E1R7Whj-0003m9-KQ@dinsdale.python.org>

http://hg.python.org/cpython/rev/5e456e1a9e8c
changeset:   72466:5e456e1a9e8c
user:        Mark Dickinson <mdickinson at enthought.com>
date:        Sat Sep 24 19:11:53 2011 +0100
summary:
  Issue #1621: Fix undefined behaviour from signed overflow in get_integer (stringlib/formatter.h)

files:
  Objects/stringlib/formatter.h |  16 +++++++---------
  1 files changed, 7 insertions(+), 9 deletions(-)


diff --git a/Objects/stringlib/formatter.h b/Objects/stringlib/formatter.h
--- a/Objects/stringlib/formatter.h
+++ b/Objects/stringlib/formatter.h
@@ -73,7 +73,7 @@
 get_integer(STRINGLIB_CHAR **ptr, STRINGLIB_CHAR *end,
                   Py_ssize_t *result)
 {
-    Py_ssize_t accumulator, digitval, oldaccumulator;
+    Py_ssize_t accumulator, digitval;
     int numdigits;
     accumulator = numdigits = 0;
     for (;;(*ptr)++, numdigits++) {
@@ -83,19 +83,17 @@
         if (digitval < 0)
             break;
         /*
-           This trick was copied from old Unicode format code.  It's cute,
-           but would really suck on an old machine with a slow divide
-           implementation.  Fortunately, in the normal case we do not
-           expect too many digits.
+           Detect possible overflow before it happens:
+
+              accumulator * 10 + digitval > PY_SSIZE_T_MAX if and only if
+              accumulator > (PY_SSIZE_T_MAX - digitval) / 10.
         */
-        oldaccumulator = accumulator;
-        accumulator *= 10;
-        if ((accumulator+10)/10 != oldaccumulator+1) {
+        if (accumulator > (PY_SSIZE_T_MAX - digitval) / 10) {
             PyErr_Format(PyExc_ValueError,
                          "Too many decimal digits in format string");
             return -1;
         }
-        accumulator += digitval;
+        accumulator = accumulator * 10 + digitval;
     }
     *result = accumulator;
     return numdigits;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep 25 01:42:29 2011
From: python-checkins at python.org (eric.araujo)
Date: Sun, 25 Sep 2011 01:42:29 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_HAS=5FUSER=5FSITE_in?=
	=?utf8?q?_install_command_too?=
Message-ID: <E1R7brZ-0001lx-Um@dinsdale.python.org>

http://hg.python.org/distutils2/rev/cbbbee12d192
changeset:   1187:cbbbee12d192
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Sep 22 20:27:10 2011 +0200
summary:
  Fix HAS_USER_SITE in install command too

files:
  distutils2/command/install_dist.py |  7 +++++--
  1 files changed, 5 insertions(+), 2 deletions(-)


diff --git a/distutils2/command/install_dist.py b/distutils2/command/install_dist.py
--- a/distutils2/command/install_dist.py
+++ b/distutils2/command/install_dist.py
@@ -13,8 +13,11 @@
 from distutils2.util import convert_path, change_root, get_platform
 from distutils2.errors import PackagingOptionError
 
-
-HAS_USER_SITE = True
+import site
+if sys.version_info[:2] >= (2, 6):
+    HAS_USER_SITE = True
+else:
+    HAS_USER_SITE = False
 
 
 class install_dist(Command):

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Sun Sep 25 01:42:29 2011
From: python-checkins at python.org (eric.araujo)
Date: Sun, 25 Sep 2011 01:42:29 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Remove_docstring_with_fa?=
	=?utf8?q?ctual_errors?=
Message-ID: <E1R7brZ-0001m6-VI@dinsdale.python.org>

http://hg.python.org/distutils2/rev/5ba7a25539a4
changeset:   1188:5ba7a25539a4
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Sep 23 22:18:32 2011 +0200
summary:
  Remove docstring with factual errors

files:
  distutils2/compat.py |  5 -----
  1 files changed, 0 insertions(+), 5 deletions(-)


diff --git a/distutils2/compat.py b/distutils2/compat.py
--- a/distutils2/compat.py
+++ b/distutils2/compat.py
@@ -161,11 +161,6 @@
 
 
 def fsencode(filename):
-    """
-    Encode filename to the filesystem encoding with 'surrogateescape' error
-    handler, return bytes unchanged. On Windows, use 'strict' error handler if
-    the file system encoding is 'mbcs' (which is the default encoding).
-    """
     if isinstance(filename, str):
         return filename
     elif isinstance(filename, unicode):

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Sun Sep 25 01:42:29 2011
From: python-checkins at python.org (eric.araujo)
Date: Sun, 25 Sep 2011 01:42:29 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Remove_unused_name?=
Message-ID: <E1R7brZ-0001mA-W3@dinsdale.python.org>

http://hg.python.org/distutils2/rev/f3b2b4a15fef
changeset:   1189:f3b2b4a15fef
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Sep 23 23:56:32 2011 +0200
summary:
  Remove unused name

files:
  distutils2/pypi/xmlrpc.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/distutils2/pypi/xmlrpc.py b/distutils2/pypi/xmlrpc.py
--- a/distutils2/pypi/xmlrpc.py
+++ b/distutils2/pypi/xmlrpc.py
@@ -6,7 +6,7 @@
 implementation at http://wiki.python.org/moin/PyPiXmlRpc).
 """
 
-import xmlrpclib, sys
+import xmlrpclib
 
 from distutils2 import logger
 from distutils2.errors import IrrationalVersionError

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Sun Sep 25 01:42:30 2011
From: python-checkins at python.org (eric.araujo)
Date: Sun, 25 Sep 2011 01:42:30 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Always_use_backported_ta?=
	=?utf8?q?rfile?=
Message-ID: <E1R7bra-0001mF-0I@dinsdale.python.org>

http://hg.python.org/distutils2/rev/cc976672950c
changeset:   1190:cc976672950c
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 24 00:16:13 2011 +0200
summary:
  Always use backported tarfile

files:
  distutils2/_backport/tests/test_shutil.py |  3 +--
  distutils2/tests/test_command_sdist.py    |  2 +-
  2 files changed, 2 insertions(+), 3 deletions(-)


diff --git a/distutils2/_backport/tests/test_shutil.py b/distutils2/_backport/tests/test_shutil.py
--- a/distutils2/_backport/tests/test_shutil.py
+++ b/distutils2/_backport/tests/test_shutil.py
@@ -1,14 +1,13 @@
 import os
 import sys
 import stat
-import tarfile
 import tempfile
 from os.path import splitdrive
 from StringIO import StringIO
 
 from distutils.spawn import find_executable, spawn
 from distutils2.compat import wraps
-from distutils2._backport import shutil
+from distutils2._backport import shutil, tarfile
 from distutils2._backport.shutil import (
     _make_tarball, _make_zipfile, make_archive, unpack_archive,
     register_archive_format, unregister_archive_format, get_archive_formats,
diff --git a/distutils2/tests/test_command_sdist.py b/distutils2/tests/test_command_sdist.py
--- a/distutils2/tests/test_command_sdist.py
+++ b/distutils2/tests/test_command_sdist.py
@@ -1,7 +1,6 @@
 """Tests for distutils2.command.sdist."""
 import os
 import zipfile
-import tarfile
 import logging
 
 from distutils2.tests.support import requires_zlib
@@ -22,6 +21,7 @@
 from distutils2.errors import PackagingOptionError
 from distutils2.util import find_executable
 from distutils2.tests import support
+from distutils2._backport import tarfile
 from distutils2._backport.shutil import get_archive_formats
 
 

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Sun Sep 25 01:42:30 2011
From: python-checkins at python.org (eric.araujo)
Date: Sun, 25 Sep 2011 01:42:30 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Remove_inline_comment_th?=
 =?utf8?q?at_is_not_supported_by_configparser_in_3=2E3=2E?=
Message-ID: <E1R7bra-0001mI-0g@dinsdale.python.org>

http://hg.python.org/distutils2/rev/17df7b9f9b26
changeset:   1191:17df7b9f9b26
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 24 00:30:37 2011 +0200
summary:
  Remove inline comment that is not supported by configparser in 3.3.

(Deleted rather than moved because multilib implementations vary.)

files:
  distutils2/_backport/sysconfig.cfg |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/distutils2/_backport/sysconfig.cfg b/distutils2/_backport/sysconfig.cfg
--- a/distutils2/_backport/sysconfig.cfg
+++ b/distutils2/_backport/sysconfig.cfg
@@ -31,7 +31,7 @@
 # be used directly in [resource_locations].
 confdir = /etc
 datadir = /usr/share
-libdir = /usr/lib  ; or /usr/lib64 on a multilib system
+libdir = /usr/lib
 statedir = /var
 # User resource directory
 local = ~/.local/{distribution.name}

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Sun Sep 25 01:42:30 2011
From: python-checkins at python.org (eric.araujo)
Date: Sun, 25 Sep 2011 01:42:30 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Adapt_skip_message?=
Message-ID: <E1R7bra-0001mM-16@dinsdale.python.org>

http://hg.python.org/distutils2/rev/21f1ff308dfe
changeset:   1192:21f1ff308dfe
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 24 00:33:33 2011 +0200
summary:
  Adapt skip message

files:
  distutils2/tests/support.py |  3 +--
  1 files changed, 1 insertions(+), 2 deletions(-)


diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -313,8 +313,7 @@
     """
     filename = _get_xxmodule_path()
     if filename is None:
-        raise unittest.SkipTest('cannot find xxmodule.c (test must run in '
-                                'the python build dir)')
+        raise unittest.SkipTest('cannot find xxmodule.c')
     shutil.copy(filename, directory)
 
 

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Sun Sep 25 01:42:30 2011
From: python-checkins at python.org (eric.araujo)
Date: Sun, 25 Sep 2011 01:42:30 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Remove_fake=5Fdists_dupl?=
 =?utf8?q?icates_in_=5Fbackport/tests=2E?=
Message-ID: <E1R7bra-0001mR-3Q@dinsdale.python.org>

http://hg.python.org/distutils2/rev/e2ee5ae75837
changeset:   1193:e2ee5ae75837
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 24 00:42:37 2011 +0200
summary:
  Remove fake_dists duplicates in _backport/tests.

The PEP 376 implementation used to live in d2._backport.pkgutil, which
was actually a mixed back/forward-port like sysconfig, so its tests were
in d2._backport.tests.test_pkgutil and the needed file in the same
directory.  Now the code lives in d2.database and the files in
d2/tests.)

files:
  distutils2/_backport/tests/fake_dists/babar-0.1.dist-info/INSTALLER                |  Bin 
  distutils2/_backport/tests/fake_dists/babar-0.1.dist-info/METADATA                 |    4 --
  distutils2/_backport/tests/fake_dists/babar-0.1.dist-info/RECORD                   |  Bin 
  distutils2/_backport/tests/fake_dists/babar-0.1.dist-info/REQUESTED                |  Bin 
  distutils2/_backport/tests/fake_dists/babar-0.1.dist-info/RESOURCES                |    2 -
  distutils2/_backport/tests/fake_dists/babar.cfg                                    |    1 -
  distutils2/_backport/tests/fake_dists/babar.png                                    |  Bin 
  distutils2/_backport/tests/fake_dists/bacon-0.1.egg-info/PKG-INFO                  |    6 ---
  distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/PKG-INFO             |   18 ----------
  distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/SOURCES.txt          |  Bin 
  distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/dependency_links.txt |    1 -
  distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/entry_points.txt     |    3 -
  distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/not-zip-safe         |    1 -
  distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/requires.txt         |    6 ---
  distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/top_level.txt        |  Bin 
  distutils2/_backport/tests/fake_dists/cheese-2.0.2.egg-info                        |    5 --
  distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info/INSTALLER           |  Bin 
  distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info/METADATA            |    9 -----
  distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info/RECORD              |  Bin 
  distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info/REQUESTED           |  Bin 
  distutils2/_backport/tests/fake_dists/choxie-2.0.0.9/choxie/__init__.py            |    2 -
  distutils2/_backport/tests/fake_dists/choxie-2.0.0.9/choxie/chocolate.py           |   10 -----
  distutils2/_backport/tests/fake_dists/choxie-2.0.0.9/truffles.py                   |    5 --
  distutils2/_backport/tests/fake_dists/coconuts-aster-10.3.egg-info/PKG-INFO        |    5 --
  distutils2/_backport/tests/fake_dists/grammar-1.0a4.dist-info/INSTALLER            |  Bin 
  distutils2/_backport/tests/fake_dists/grammar-1.0a4.dist-info/METADATA             |    5 --
  distutils2/_backport/tests/fake_dists/grammar-1.0a4.dist-info/RECORD               |  Bin 
  distutils2/_backport/tests/fake_dists/grammar-1.0a4.dist-info/REQUESTED            |  Bin 
  distutils2/_backport/tests/fake_dists/grammar-1.0a4/grammar/__init__.py            |    2 -
  distutils2/_backport/tests/fake_dists/grammar-1.0a4/grammar/utils.py               |    8 ----
  distutils2/_backport/tests/fake_dists/nut-funkyversion.egg-info                    |    3 -
  distutils2/_backport/tests/fake_dists/strawberry-0.6.egg                           |  Bin 
  distutils2/_backport/tests/fake_dists/towel_stuff-0.1.dist-info/INSTALLER          |  Bin 
  distutils2/_backport/tests/fake_dists/towel_stuff-0.1.dist-info/METADATA           |    7 ---
  distutils2/_backport/tests/fake_dists/towel_stuff-0.1.dist-info/RECORD             |  Bin 
  distutils2/_backport/tests/fake_dists/towel_stuff-0.1.dist-info/REQUESTED          |  Bin 
  distutils2/_backport/tests/fake_dists/towel_stuff-0.1/towel_stuff/__init__.py      |   18 ----------
  distutils2/_backport/tests/fake_dists/truffles-5.0.egg-info                        |    3 -
  38 files changed, 0 insertions(+), 124 deletions(-)


diff --git a/distutils2/_backport/tests/fake_dists/babar-0.1.dist-info/INSTALLER b/distutils2/_backport/tests/fake_dists/babar-0.1.dist-info/INSTALLER
deleted file mode 100644
diff --git a/distutils2/_backport/tests/fake_dists/babar-0.1.dist-info/METADATA b/distutils2/_backport/tests/fake_dists/babar-0.1.dist-info/METADATA
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/babar-0.1.dist-info/METADATA
+++ /dev/null
@@ -1,4 +0,0 @@
-Metadata-version: 1.2
-Name: babar
-Version: 0.1
-Author: FELD Boris
\ No newline at end of file
diff --git a/distutils2/_backport/tests/fake_dists/babar-0.1.dist-info/RECORD b/distutils2/_backport/tests/fake_dists/babar-0.1.dist-info/RECORD
deleted file mode 100644
diff --git a/distutils2/_backport/tests/fake_dists/babar-0.1.dist-info/REQUESTED b/distutils2/_backport/tests/fake_dists/babar-0.1.dist-info/REQUESTED
deleted file mode 100644
diff --git a/distutils2/_backport/tests/fake_dists/babar-0.1.dist-info/RESOURCES b/distutils2/_backport/tests/fake_dists/babar-0.1.dist-info/RESOURCES
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/babar-0.1.dist-info/RESOURCES
+++ /dev/null
@@ -1,2 +0,0 @@
-babar.png,babar.png
-babar.cfg,babar.cfg
\ No newline at end of file
diff --git a/distutils2/_backport/tests/fake_dists/babar.cfg b/distutils2/_backport/tests/fake_dists/babar.cfg
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/babar.cfg
+++ /dev/null
@@ -1,1 +0,0 @@
-Config
\ No newline at end of file
diff --git a/distutils2/_backport/tests/fake_dists/babar.png b/distutils2/_backport/tests/fake_dists/babar.png
deleted file mode 100644
diff --git a/distutils2/_backport/tests/fake_dists/bacon-0.1.egg-info/PKG-INFO b/distutils2/_backport/tests/fake_dists/bacon-0.1.egg-info/PKG-INFO
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/bacon-0.1.egg-info/PKG-INFO
+++ /dev/null
@@ -1,6 +0,0 @@
-Metadata-Version: 1.2
-Name: bacon
-Version: 0.1
-Provides-Dist: truffles (2.0)
-Provides-Dist: bacon (0.1)
-Obsoletes-Dist: truffles (>=0.9,<=1.5)
diff --git a/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/PKG-INFO b/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/PKG-INFO
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/PKG-INFO
+++ /dev/null
@@ -1,18 +0,0 @@
-Metadata-Version: 1.0
-Name: banana
-Version: 0.4
-Summary: A yellow fruit
-Home-page: http://en.wikipedia.org/wiki/Banana
-Author: Josip Djolonga
-Author-email: foo at nbar.com
-License: BSD
-Description: A fruit
-Keywords: foo bar
-Platform: UNKNOWN
-Classifier: Development Status :: 4 - Beta
-Classifier: Intended Audience :: Developers
-Classifier: Intended Audience :: Science/Research
-Classifier: License :: OSI Approved :: BSD License
-Classifier: Operating System :: OS Independent
-Classifier: Programming Language :: Python
-Classifier: Topic :: Scientific/Engineering :: GIS
diff --git a/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/SOURCES.txt b/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/SOURCES.txt
deleted file mode 100644
diff --git a/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/dependency_links.txt b/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/dependency_links.txt
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/dependency_links.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-
diff --git a/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/entry_points.txt b/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/entry_points.txt
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/entry_points.txt
+++ /dev/null
@@ -1,3 +0,0 @@
-
-      # -*- Entry points: -*-
-      
\ No newline at end of file
diff --git a/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/not-zip-safe b/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/not-zip-safe
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/not-zip-safe
+++ /dev/null
@@ -1,1 +0,0 @@
-
diff --git a/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/requires.txt b/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/requires.txt
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/requires.txt
+++ /dev/null
@@ -1,6 +0,0 @@
-# this should be ignored
-
-strawberry >=0.5
-
-[section ignored]
-foo ==0.5
diff --git a/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/top_level.txt b/distutils2/_backport/tests/fake_dists/banana-0.4.egg/EGG-INFO/top_level.txt
deleted file mode 100644
diff --git a/distutils2/_backport/tests/fake_dists/cheese-2.0.2.egg-info b/distutils2/_backport/tests/fake_dists/cheese-2.0.2.egg-info
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/cheese-2.0.2.egg-info
+++ /dev/null
@@ -1,5 +0,0 @@
-Metadata-Version: 1.2
-Name: cheese
-Version: 2.0.2
-Provides-Dist: truffles (1.0.2)
-Obsoletes-Dist: truffles (!=1.2,<=2.0)
diff --git a/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info/INSTALLER b/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info/INSTALLER
deleted file mode 100644
diff --git a/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info/METADATA b/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info/METADATA
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info/METADATA
+++ /dev/null
@@ -1,9 +0,0 @@
-Metadata-Version: 1.2
-Name: choxie
-Version: 2.0.0.9
-Summary: Chocolate with a kick!
-Requires-Dist: towel-stuff (0.1)
-Requires-Dist: nut
-Provides-Dist: truffles (1.0)
-Obsoletes-Dist: truffles (<=0.8,>=0.5)
-Obsoletes-Dist: truffles (<=0.9,>=0.6)
diff --git a/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info/RECORD b/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info/RECORD
deleted file mode 100644
diff --git a/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info/REQUESTED b/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info/REQUESTED
deleted file mode 100644
diff --git a/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9/choxie/__init__.py b/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9/choxie/__init__.py
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9/choxie/__init__.py
+++ /dev/null
@@ -1,2 +0,0 @@
-# -*- coding: utf-8 -*-
-
diff --git a/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9/choxie/chocolate.py b/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9/choxie/chocolate.py
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9/choxie/chocolate.py
+++ /dev/null
@@ -1,10 +0,0 @@
-# -*- coding: utf-8 -*-
-from towel_stuff import Towel
-
-class Chocolate(object):
-    """A piece of chocolate."""
-
-    def wrap_with_towel(self):
-        towel = Towel()
-        towel.wrap(self)
-        return towel
diff --git a/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9/truffles.py b/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9/truffles.py
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9/truffles.py
+++ /dev/null
@@ -1,5 +0,0 @@
-# -*- coding: utf-8 -*-
-from choxie.chocolate import Chocolate
-
-class Truffle(Chocolate):
-    """A truffle."""
diff --git a/distutils2/_backport/tests/fake_dists/coconuts-aster-10.3.egg-info/PKG-INFO b/distutils2/_backport/tests/fake_dists/coconuts-aster-10.3.egg-info/PKG-INFO
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/coconuts-aster-10.3.egg-info/PKG-INFO
+++ /dev/null
@@ -1,5 +0,0 @@
-Metadata-Version: 1.2
-Name: coconuts-aster
-Version: 10.3
-Provides-Dist: strawberry (0.6)
-Provides-Dist: banana (0.4)
diff --git a/distutils2/_backport/tests/fake_dists/grammar-1.0a4.dist-info/INSTALLER b/distutils2/_backport/tests/fake_dists/grammar-1.0a4.dist-info/INSTALLER
deleted file mode 100644
diff --git a/distutils2/_backport/tests/fake_dists/grammar-1.0a4.dist-info/METADATA b/distutils2/_backport/tests/fake_dists/grammar-1.0a4.dist-info/METADATA
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/grammar-1.0a4.dist-info/METADATA
+++ /dev/null
@@ -1,5 +0,0 @@
-Metadata-Version: 1.2
-Name: grammar
-Version: 1.0a4
-Requires-Dist: truffles (>=1.2)
-Author: Sherlock Holmes
diff --git a/distutils2/_backport/tests/fake_dists/grammar-1.0a4.dist-info/RECORD b/distutils2/_backport/tests/fake_dists/grammar-1.0a4.dist-info/RECORD
deleted file mode 100644
diff --git a/distutils2/_backport/tests/fake_dists/grammar-1.0a4.dist-info/REQUESTED b/distutils2/_backport/tests/fake_dists/grammar-1.0a4.dist-info/REQUESTED
deleted file mode 100644
diff --git a/distutils2/_backport/tests/fake_dists/grammar-1.0a4/grammar/__init__.py b/distutils2/_backport/tests/fake_dists/grammar-1.0a4/grammar/__init__.py
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/grammar-1.0a4/grammar/__init__.py
+++ /dev/null
@@ -1,2 +0,0 @@
-# -*- coding: utf-8 -*-
-
diff --git a/distutils2/_backport/tests/fake_dists/grammar-1.0a4/grammar/utils.py b/distutils2/_backport/tests/fake_dists/grammar-1.0a4/grammar/utils.py
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/grammar-1.0a4/grammar/utils.py
+++ /dev/null
@@ -1,8 +0,0 @@
-# -*- coding: utf-8 -*-
-from random import randint
-
-def is_valid_grammar(sentence):
-    if randint(0, 10) < 2:
-        return False
-    else:
-        return True
diff --git a/distutils2/_backport/tests/fake_dists/nut-funkyversion.egg-info b/distutils2/_backport/tests/fake_dists/nut-funkyversion.egg-info
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/nut-funkyversion.egg-info
+++ /dev/null
@@ -1,3 +0,0 @@
-Metadata-Version: 1.2
-Name: nut
-Version: funkyversion
diff --git a/distutils2/_backport/tests/fake_dists/strawberry-0.6.egg b/distutils2/_backport/tests/fake_dists/strawberry-0.6.egg
deleted file mode 100644
Binary file distutils2/_backport/tests/fake_dists/strawberry-0.6.egg has changed
diff --git a/distutils2/_backport/tests/fake_dists/towel_stuff-0.1.dist-info/INSTALLER b/distutils2/_backport/tests/fake_dists/towel_stuff-0.1.dist-info/INSTALLER
deleted file mode 100644
diff --git a/distutils2/_backport/tests/fake_dists/towel_stuff-0.1.dist-info/METADATA b/distutils2/_backport/tests/fake_dists/towel_stuff-0.1.dist-info/METADATA
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/towel_stuff-0.1.dist-info/METADATA
+++ /dev/null
@@ -1,7 +0,0 @@
-Metadata-Version: 1.2
-Name: towel-stuff
-Version: 0.1
-Provides-Dist: truffles (1.1.2)
-Provides-Dist: towel-stuff (0.1)
-Obsoletes-Dist: truffles (!=0.8,<1.0)
-Requires-Dist: bacon (<=0.2)
diff --git a/distutils2/_backport/tests/fake_dists/towel_stuff-0.1.dist-info/RECORD b/distutils2/_backport/tests/fake_dists/towel_stuff-0.1.dist-info/RECORD
deleted file mode 100644
diff --git a/distutils2/_backport/tests/fake_dists/towel_stuff-0.1.dist-info/REQUESTED b/distutils2/_backport/tests/fake_dists/towel_stuff-0.1.dist-info/REQUESTED
deleted file mode 100644
diff --git a/distutils2/_backport/tests/fake_dists/towel_stuff-0.1/towel_stuff/__init__.py b/distutils2/_backport/tests/fake_dists/towel_stuff-0.1/towel_stuff/__init__.py
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/towel_stuff-0.1/towel_stuff/__init__.py
+++ /dev/null
@@ -1,18 +0,0 @@
-# -*- coding: utf-8 -*-
-
-class Towel(object):
-    """A towel, that one should never be without."""
-
-    def __init__(self, color='tie-dye'):
-        self.color = color
-        self.wrapped_obj = None
-
-    def wrap(self, obj):
-        """Wrap an object up in our towel."""
-        self.wrapped_obj = obj
-
-    def unwrap(self):
-        """Unwrap whatever is in our towel and return whatever it is."""
-        obj = self.wrapped_obj
-        self.wrapped_obj = None
-        return obj
diff --git a/distutils2/_backport/tests/fake_dists/truffles-5.0.egg-info b/distutils2/_backport/tests/fake_dists/truffles-5.0.egg-info
deleted file mode 100644
--- a/distutils2/_backport/tests/fake_dists/truffles-5.0.egg-info
+++ /dev/null
@@ -1,3 +0,0 @@
-Metadata-Version: 1.2
-Name: truffles
-Version: 5.0

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Sun Sep 25 01:42:30 2011
From: python-checkins at python.org (eric.araujo)
Date: Sun, 25 Sep 2011 01:42:30 +0200
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Update_developers?=
	=?utf8?b?4oCZIG5vdGVz?=
Message-ID: <E1R7bra-0001mY-D0@dinsdale.python.org>

http://hg.python.org/distutils2/rev/9e4d083c4ad6
changeset:   1195:9e4d083c4ad6
parent:      1193:e2ee5ae75837
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 24 01:14:34 2011 +0200
summary:
  Update developers? notes

files:
  DEVNOTES.txt |  14 ++++++++------
  1 files changed, 8 insertions(+), 6 deletions(-)


diff --git a/DEVNOTES.txt b/DEVNOTES.txt
--- a/DEVNOTES.txt
+++ b/DEVNOTES.txt
@@ -2,11 +2,13 @@
 ====================
 
 - Distutils2 runs on Python from 2.4 to 2.7 so make sure you don't use code
-  that doesn't work under one of these Python versions.
+  that doesn't work under one of these Python versions.  There is also a
+  version compatible with 3.1-3.3 in the "python3" branch.
 
-- For 2.4, you need to run "python2.4 setup.py build" before you can run tests
-  or pysetup.
+- For 2.4, you need to run "python2.4 setup.py build" before you can try out
+  pysetup or run tests (unless you use the runtests.py script which will call
+  "setup.py build" automatically).
 
-- Always run tests.sh before you push a change. This implies
-  that you have all Python versions installed from 2.4 to 2.7. Be sure to have 
-  docutils installed on all python versions no avoid skipping tests as well.
+- Always run tests.sh before you commit a change.  This implies that you have
+  all Python versions installed from 2.4 to 2.7.  Be sure to also have docutils
+  installed on all Python versions to avoid skipping tests.

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Sun Sep 25 01:42:30 2011
From: python-checkins at python.org (eric.araujo)
Date: Sun, 25 Sep 2011 01:42:30 +0200
Subject: [Python-checkins] =?utf8?q?distutils2_=28python3=29=3A_Start_a_br?=
 =?utf8?q?anch_to_provide_Distutils2_for_Python_3=2E?=
Message-ID: <E1R7bra-0001mV-BD@dinsdale.python.org>

http://hg.python.org/distutils2/rev/d9ce66d22bb5
changeset:   1194:d9ce66d22bb5
branch:      python3
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 24 01:06:28 2011 +0200
summary:
  Start a branch to provide Distutils2 for Python 3.

This codebase is compatible with 3.1, 3.2 and 3.3.  It was converted with 2to3
and a semi-automated diff/merge with packaging in 3.3 to fix some idioms.
We?ve now come full circle from 2.x to 3.x to 2.x to 3.x again :)

Starting from now, contributors can make patches for packaging (preferred, as
the stdlib?s regrtest is very useful), distutils2 or distutils-python3, and
we?ll make patches flow between versions.

files:
  PC/build_ssl.py                                   |  282 ---
  distutils2/_backport/_hashopenssl.c               |  524 ------
  distutils2/_backport/hashlib.py                   |  143 -
  distutils2/_backport/md5.c                        |  381 ----
  distutils2/_backport/md5.h                        |   91 -
  distutils2/_backport/md5module.c                  |  312 ----
  distutils2/_backport/sha256module.c               |  701 ---------
  distutils2/_backport/sha512module.c               |  769 ----------
  distutils2/_backport/shamodule.c                  |  593 -------
  distutils2/_backport/shutil.py                    |   39 +-
  distutils2/_backport/sysconfig.cfg                |    4 +-
  distutils2/_backport/sysconfig.py                 |   46 +-
  distutils2/_backport/tarfile.py                   |  304 +--
  distutils2/_backport/tests/test_shutil.py         |   17 +-
  distutils2/_backport/tests/test_sysconfig.py      |   54 +-
  distutils2/command/bdist_msi.py                   |   10 +-
  distutils2/command/bdist_wininst.py               |   36 +-
  distutils2/command/build_clib.py                  |    4 +-
  distutils2/command/build_ext.py                   |   35 +-
  distutils2/command/build_py.py                    |    4 +-
  distutils2/command/build_scripts.py               |   18 +-
  distutils2/command/cmd.py                         |   12 +-
  distutils2/command/config.py                      |   23 +-
  distutils2/command/install_data.py                |    2 +-
  distutils2/command/install_dist.py                |   56 +-
  distutils2/command/install_distinfo.py            |   26 +-
  distutils2/command/install_lib.py                 |    2 +-
  distutils2/command/install_scripts.py             |    2 +-
  distutils2/command/register.py                    |   37 +-
  distutils2/command/sdist.py                       |    8 +-
  distutils2/command/upload.py                      |   29 +-
  distutils2/command/upload_docs.py                 |   18 +-
  distutils2/compat.py                              |  126 +-
  distutils2/compiler/__init__.py                   |    4 +-
  distutils2/compiler/bcppcompiler.py               |   12 +-
  distutils2/compiler/ccompiler.py                  |   21 +-
  distutils2/compiler/cygwinccompiler.py            |   11 +-
  distutils2/compiler/extension.py                  |    6 +-
  distutils2/compiler/msvc9compiler.py              |   16 +-
  distutils2/compiler/msvccompiler.py               |   12 +-
  distutils2/compiler/unixccompiler.py              |    8 +-
  distutils2/config.py                              |   31 +-
  distutils2/create.py                              |  120 +-
  distutils2/database.py                            |   38 +-
  distutils2/depgraph.py                            |   49 +-
  distutils2/dist.py                                |   36 +-
  distutils2/fancy_getopt.py                        |   10 +-
  distutils2/install.py                             |   21 +-
  distutils2/manifest.py                            |   18 +-
  distutils2/markers.py                             |   18 +-
  distutils2/metadata.py                            |   25 +-
  distutils2/pypi/base.py                           |    2 +-
  distutils2/pypi/dist.py                           |   24 +-
  distutils2/pypi/simple.py                         |   73 +-
  distutils2/pypi/wrapper.py                        |    8 +-
  distutils2/pypi/xmlrpc.py                         |    6 +-
  distutils2/run.py                                 |   67 +-
  distutils2/tests/__init__.py                      |   10 +-
  distutils2/tests/__main__.py                      |    2 +-
  distutils2/tests/pypi_server.py                   |   36 +-
  distutils2/tests/pypi_test_server.py              |    2 +-
  distutils2/tests/support.py                       |   31 +-
  distutils2/tests/test_command_build_clib.py       |    2 +-
  distutils2/tests/test_command_build_ext.py        |   19 +-
  distutils2/tests/test_command_build_py.py         |    4 +-
  distutils2/tests/test_command_build_scripts.py    |    5 +-
  distutils2/tests/test_command_config.py           |    5 +-
  distutils2/tests/test_command_install_dist.py     |   26 +-
  distutils2/tests/test_command_install_distinfo.py |   45 +-
  distutils2/tests/test_command_install_lib.py      |    5 +-
  distutils2/tests/test_command_install_scripts.py  |    5 +-
  distutils2/tests/test_command_register.py         |   39 +-
  distutils2/tests/test_command_sdist.py            |   30 +-
  distutils2/tests/test_command_test.py             |    6 +-
  distutils2/tests/test_command_upload.py           |   14 +-
  distutils2/tests/test_command_upload_docs.py      |   32 +-
  distutils2/tests/test_compiler.py                 |    4 +-
  distutils2/tests/test_config.py                   |   14 +-
  distutils2/tests/test_create.py                   |   29 +-
  distutils2/tests/test_database.py                 |   63 +-
  distutils2/tests/test_depgraph.py                 |    2 +-
  distutils2/tests/test_dist.py                     |   16 +-
  distutils2/tests/test_install.py                  |    6 +-
  distutils2/tests/test_manifest.py                 |   12 +-
  distutils2/tests/test_markers.py                  |    3 +-
  distutils2/tests/test_metadata.py                 |   64 +-
  distutils2/tests/test_mixin2to3.py                |   33 +-
  distutils2/tests/test_msvc9compiler.py            |   10 +-
  distutils2/tests/test_pypi_server.py              |   33 +-
  distutils2/tests/test_pypi_simple.py              |   59 +-
  distutils2/tests/test_pypi_xmlrpc.py              |    5 +-
  distutils2/tests/test_run.py                      |   11 +-
  distutils2/tests/test_uninstall.py                |    2 +-
  distutils2/tests/test_util.py                     |   72 +-
  distutils2/tests/xxmodule.c                       |  507 +++---
  distutils2/util.py                                |  132 +-
  distutils2/version.py                             |    8 +-
  runtests.py                                       |    6 -
  setup.py                                          |  143 +-
  tests.sh                                          |   29 +-
  100 files changed, 1215 insertions(+), 5710 deletions(-)


diff --git a/PC/build_ssl.py b/PC/build_ssl.py
deleted file mode 100644
--- a/PC/build_ssl.py
+++ /dev/null
@@ -1,282 +0,0 @@
-# Script for building the _ssl and _hashlib modules for Windows.
-# Uses Perl to setup the OpenSSL environment correctly
-# and build OpenSSL, then invokes a simple nmake session
-# for the actual _ssl.pyd and _hashlib.pyd DLLs.
-
-# THEORETICALLY, you can:
-# * Unpack the latest SSL release one level above your main Python source
-#   directory.  It is likely you will already find the zlib library and
-#   any other external packages there.
-# * Install ActivePerl and ensure it is somewhere on your path.
-# * Run this script from the PCBuild directory.
-#
-# it should configure and build SSL, then build the _ssl and _hashlib
-# Python extensions without intervention.
-
-# Modified by Christian Heimes
-# Now this script supports pre-generated makefiles and assembly files.
-# Developers don't need an installation of Perl anymore to build Python. A svn
-# checkout from our svn repository is enough.
-#
-# In Order to create the files in the case of an update you still need Perl.
-# Run build_ssl in this order:
-# python.exe build_ssl.py Release x64
-# python.exe build_ssl.py Release Win32
-
-import os, sys, re, shutil
-
-# Find all "foo.exe" files on the PATH.
-def find_all_on_path(filename, extras = None):
-    entries = os.environ["PATH"].split(os.pathsep)
-    ret = []
-    for p in entries:
-        fname = os.path.abspath(os.path.join(p, filename))
-        if os.path.isfile(fname) and fname not in ret:
-            ret.append(fname)
-    if extras:
-        for p in extras:
-            fname = os.path.abspath(os.path.join(p, filename))
-            if os.path.isfile(fname) and fname not in ret:
-                ret.append(fname)
-    return ret
-
-# Find a suitable Perl installation for OpenSSL.
-# cygwin perl does *not* work.  ActivePerl does.
-# Being a Perl dummy, the simplest way I can check is if the "Win32" package
-# is available.
-def find_working_perl(perls):
-    for perl in perls:
-        fh = os.popen('"%s" -e "use Win32;"' % perl)
-        fh.read()
-        rc = fh.close()
-        if rc:
-            continue
-        return perl
-    print("Can not find a suitable PERL:")
-    if perls:
-        print(" the following perl interpreters were found:")
-        for p in perls:
-            print(" ", p)
-        print(" None of these versions appear suitable for building OpenSSL")
-    else:
-        print(" NO perl interpreters were found on this machine at all!")
-    print(" Please install ActivePerl and ensure it appears on your path")
-    return None
-
-# Locate the best SSL directory given a few roots to look into.
-def find_best_ssl_dir(sources):
-    candidates = []
-    for s in sources:
-        try:
-            # note: do not abspath s; the build will fail if any
-            # higher up directory name has spaces in it.
-            fnames = os.listdir(s)
-        except os.error:
-            fnames = []
-        for fname in fnames:
-            fqn = os.path.join(s, fname)
-            if os.path.isdir(fqn) and fname.startswith("openssl-"):
-                candidates.append(fqn)
-    # Now we have all the candidates, locate the best.
-    best_parts = []
-    best_name = None
-    for c in candidates:
-        parts = re.split("[.-]", os.path.basename(c))[1:]
-        # eg - openssl-0.9.7-beta1 - ignore all "beta" or any other qualifiers
-        if len(parts) >= 4:
-            continue
-        if parts > best_parts:
-            best_parts = parts
-            best_name = c
-    if best_name is not None:
-        print("Found an SSL directory at '%s'" % (best_name,))
-    else:
-        print("Could not find an SSL directory in '%s'" % (sources,))
-    sys.stdout.flush()
-    return best_name
-
-def create_makefile64(makefile, m32):
-    """Create and fix makefile for 64bit
-
-    Replace 32 with 64bit directories
-    """
-    if not os.path.isfile(m32):
-        return
-    with open(m32) as fin:
-        with open(makefile, 'w') as fout:
-            for line in fin:
-                line = line.replace("=tmp32", "=tmp64")
-                line = line.replace("=out32", "=out64")
-                line = line.replace("=inc32", "=inc64")
-                # force 64 bit machine
-                line = line.replace("MKLIB=lib", "MKLIB=lib /MACHINE:X64")
-                line = line.replace("LFLAGS=", "LFLAGS=/MACHINE:X64 ")
-                # don't link against the lib on 64bit systems
-                line = line.replace("bufferoverflowu.lib", "")
-                fout.write(line)
-    os.unlink(m32)
-
-def fix_makefile(makefile):
-    """Fix some stuff in all makefiles
-    """
-    if not os.path.isfile(makefile):
-        return
-    with open(makefile) as fin:
-        lines = fin.readlines()
-    with open(makefile, 'w') as fout:
-        for line in lines:
-            if line.startswith("PERL="):
-                continue
-            if line.startswith("CP="):
-                line = "CP=copy\n"
-            if line.startswith("MKDIR="):
-                line = "MKDIR=mkdir\n"
-            if line.startswith("CFLAG="):
-                line = line.strip()
-                for algo in ("RC5", "MDC2", "IDEA"):
-                    noalgo = " -DOPENSSL_NO_%s" % algo
-                    if noalgo not in line:
-                        line = line + noalgo
-                line = line + '\n'
-            fout.write(line)
-
-def run_configure(configure, do_script):
-    print("perl Configure "+configure+" no-idea no-mdc2")
-    os.system("perl Configure "+configure+" no-idea no-mdc2")
-    print(do_script)
-    os.system(do_script)
-
-def cmp(f1, f2):
-    bufsize = 1024 * 8
-    with open(f1, 'rb') as fp1, open(f2, 'rb') as fp2:
-        while True:
-            b1 = fp1.read(bufsize)
-            b2 = fp2.read(bufsize)
-            if b1 != b2:
-                return False
-            if not b1:
-                return True
-
-def copy(src, dst):
-    if os.path.isfile(dst) and cmp(src, dst):
-        return
-    shutil.copy(src, dst)
-
-def main():
-    build_all = "-a" in sys.argv
-    # Default to 'Release' configuration on for the 'Win32' platform
-    try:
-        configuration, platform = sys.argv[1:3]
-    except ValueError:
-        configuration, platform = 'Release', 'Win32'
-    if configuration == "Release":
-        debug = False
-    elif configuration == "Debug":
-        debug = True
-    else:
-        raise ValueError(str(sys.argv))
-
-    if platform == "Win32":
-        arch = "x86"
-        configure = "VC-WIN32"
-        do_script = "ms\\do_nasm"
-        makefile="ms\\nt.mak"
-        m32 = makefile
-        dirsuffix = "32"
-    elif platform == "x64":
-        arch="amd64"
-        configure = "VC-WIN64A"
-        do_script = "ms\\do_win64a"
-        makefile = "ms\\nt64.mak"
-        m32 = makefile.replace('64', '')
-        dirsuffix = "64"
-        #os.environ["VSEXTCOMP_USECL"] = "MS_OPTERON"
-    else:
-        raise ValueError(str(sys.argv))
-
-    make_flags = ""
-    if build_all:
-        make_flags = "-a"
-    # perl should be on the path, but we also look in "\perl" and "c:\\perl"
-    # as "well known" locations
-    perls = find_all_on_path("perl.exe", ["\\perl\\bin", "C:\\perl\\bin"])
-    perl = find_working_perl(perls)
-    if perl:
-        print("Found a working perl at '%s'" % (perl,))
-    else:
-        print("No Perl installation was found. Existing Makefiles are used.")
-    sys.stdout.flush()
-    # Look for SSL 2 levels up from pcbuild - ie, same place zlib etc all live.
-    ssl_dir = find_best_ssl_dir(("..\\..",))
-    if ssl_dir is None:
-        sys.exit(1)
-
-    old_cd = os.getcwd()
-    try:
-        os.chdir(ssl_dir)
-        # rebuild makefile when we do the role over from 32 to 64 build
-        if arch == "amd64" and os.path.isfile(m32) and not os.path.isfile(makefile):
-            os.unlink(m32)
-
-        # If the ssl makefiles do not exist, we invoke Perl to generate them.
-        # Due to a bug in this script, the makefile sometimes ended up empty
-        # Force a regeneration if it is.
-        if not os.path.isfile(makefile) or os.path.getsize(makefile)==0:
-            if perl is None:
-                print("Perl is required to build the makefiles!")
-                sys.exit(1)
-
-            print("Creating the makefiles...")
-            sys.stdout.flush()
-            # Put our working Perl at the front of our path
-            os.environ["PATH"] = os.path.dirname(perl) + \
-                                          os.pathsep + \
-                                          os.environ["PATH"]
-            run_configure(configure, do_script)
-            if debug:
-                print("OpenSSL debug builds aren't supported.")
-            #if arch=="x86" and debug:
-            #    # the do_masm script in openssl doesn't generate a debug
-            #    # build makefile so we generate it here:
-            #    os.system("perl util\mk1mf.pl debug "+configure+" >"+makefile)
-
-            if arch == "amd64":
-                create_makefile64(makefile, m32)
-            fix_makefile(makefile)
-            copy(r"crypto\buildinf.h", r"crypto\buildinf_%s.h" % arch)
-            copy(r"crypto\opensslconf.h", r"crypto\opensslconf_%s.h" % arch)
-
-        # If the assembler files don't exist in tmpXX, copy them there
-        if perl is None and os.path.exists("asm"+dirsuffix):
-            if not os.path.exists("tmp"+dirsuffix):
-                os.mkdir("tmp"+dirsuffix)
-            for f in os.listdir("asm"+dirsuffix):
-                if not f.endswith(".asm"): continue
-                if os.path.isfile(r"tmp%s\%s" % (dirsuffix, f)): continue
-                shutil.copy(r"asm%s\%s" % (dirsuffix, f), "tmp"+dirsuffix)
-
-        # Now run make.
-        if arch == "amd64":
-            rc = os.system("ml64 -c -Foms\\uptable.obj ms\\uptable.asm")
-            if rc:
-                print("ml64 assembler has failed.")
-                sys.exit(rc)
-
-        copy(r"crypto\buildinf_%s.h" % arch, r"crypto\buildinf.h")
-        copy(r"crypto\opensslconf_%s.h" % arch, r"crypto\opensslconf.h")
-
-        #makeCommand = "nmake /nologo PERL=\"%s\" -f \"%s\"" %(perl, makefile)
-        makeCommand = "nmake /nologo -f \"%s\"" % makefile
-        print("Executing ssl makefiles: " + makeCommand)
-        sys.stdout.flush()
-        rc = os.system(makeCommand)
-        if rc:
-            print("Executing "+makefile+" failed")
-            print(rc)
-            sys.exit(rc)
-    finally:
-        os.chdir(old_cd)
-    sys.exit(rc)
-
-if __name__=='__main__':
-    main()
diff --git a/distutils2/_backport/_hashopenssl.c b/distutils2/_backport/_hashopenssl.c
deleted file mode 100644
--- a/distutils2/_backport/_hashopenssl.c
+++ /dev/null
@@ -1,524 +0,0 @@
-/* Module that wraps all OpenSSL hash algorithms */
-
-/*
- * Copyright (C) 2005   Gregory P. Smith (greg at krypto.org)
- * Licensed to PSF under a Contributor Agreement.
- *
- * Derived from a skeleton of shamodule.c containing work performed by:
- *
- * Andrew Kuchling (amk at amk.ca)
- * Greg Stein (gstein at lyra.org)
- *
- */
-
-#define PY_SSIZE_T_CLEAN
-
-#include "Python.h"
-#include "structmember.h"
-
-#if (PY_VERSION_HEX < 0x02050000)
-#define Py_ssize_t      int
-#endif
-
-/* EVP is the preferred interface to hashing in OpenSSL */
-#include <openssl/evp.h>
-
-#define MUNCH_SIZE INT_MAX
-
-
-#ifndef HASH_OBJ_CONSTRUCTOR
-#define HASH_OBJ_CONSTRUCTOR 0
-#endif
-
-typedef struct {
-    PyObject_HEAD
-    PyObject            *name;  /* name of this hash algorithm */
-    EVP_MD_CTX          ctx;    /* OpenSSL message digest context */
-} EVPobject;
-
-
-static PyTypeObject EVPtype;
-
-
-#define DEFINE_CONSTS_FOR_NEW(Name)  \
-    static PyObject *CONST_ ## Name ## _name_obj; \
-    static EVP_MD_CTX CONST_new_ ## Name ## _ctx; \
-    static EVP_MD_CTX *CONST_new_ ## Name ## _ctx_p = NULL;
-
-DEFINE_CONSTS_FOR_NEW(md5)
-DEFINE_CONSTS_FOR_NEW(sha1)
-DEFINE_CONSTS_FOR_NEW(sha224)
-DEFINE_CONSTS_FOR_NEW(sha256)
-DEFINE_CONSTS_FOR_NEW(sha384)
-DEFINE_CONSTS_FOR_NEW(sha512)
-
-
-static EVPobject *
-newEVPobject(PyObject *name)
-{
-    EVPobject *retval = (EVPobject *)PyObject_New(EVPobject, &EVPtype);
-
-    /* save the name for .name to return */
-    if (retval != NULL) {
-        Py_INCREF(name);
-        retval->name = name;
-    }
-
-    return retval;
-}
-
-/* Internal methods for a hash object */
-
-static void
-EVP_dealloc(PyObject *ptr)
-{
-    EVP_MD_CTX_cleanup(&((EVPobject *)ptr)->ctx);
-    Py_XDECREF(((EVPobject *)ptr)->name);
-    PyObject_Del(ptr);
-}
-
-
-/* External methods for a hash object */
-
-PyDoc_STRVAR(EVP_copy__doc__, "Return a copy of the hash object.");
-
-static PyObject *
-EVP_copy(EVPobject *self, PyObject *unused)
-{
-    EVPobject *newobj;
-
-    if ( (newobj = newEVPobject(self->name))==NULL)
-        return NULL;
-
-    EVP_MD_CTX_copy(&newobj->ctx, &self->ctx);
-    return (PyObject *)newobj;
-}
-
-PyDoc_STRVAR(EVP_digest__doc__,
-"Return the digest value as a string of binary data.");
-
-static PyObject *
-EVP_digest(EVPobject *self, PyObject *unused)
-{
-    unsigned char digest[EVP_MAX_MD_SIZE];
-    EVP_MD_CTX temp_ctx;
-    PyObject *retval;
-    unsigned int digest_size;
-
-    EVP_MD_CTX_copy(&temp_ctx, &self->ctx);
-    digest_size = EVP_MD_CTX_size(&temp_ctx);
-    EVP_DigestFinal(&temp_ctx, digest, NULL);
-
-    retval = PyString_FromStringAndSize((const char *)digest, digest_size);
-    EVP_MD_CTX_cleanup(&temp_ctx);
-    return retval;
-}
-
-PyDoc_STRVAR(EVP_hexdigest__doc__,
-"Return the digest value as a string of hexadecimal digits.");
-
-static PyObject *
-EVP_hexdigest(EVPobject *self, PyObject *unused)
-{
-    unsigned char digest[EVP_MAX_MD_SIZE];
-    EVP_MD_CTX temp_ctx;
-    PyObject *retval;
-    char *hex_digest;
-    unsigned int i, j, digest_size;
-
-    /* Get the raw (binary) digest value */
-    EVP_MD_CTX_copy(&temp_ctx, &self->ctx);
-    digest_size = EVP_MD_CTX_size(&temp_ctx);
-    EVP_DigestFinal(&temp_ctx, digest, NULL);
-
-    EVP_MD_CTX_cleanup(&temp_ctx);
-
-    /* Create a new string */
-    /* NOTE: not thread safe! modifying an already created string object */
-    /* (not a problem because we hold the GIL by default) */
-    retval = PyString_FromStringAndSize(NULL, digest_size * 2);
-    if (!retval)
-	    return NULL;
-    hex_digest = PyString_AsString(retval);
-    if (!hex_digest) {
-	    Py_DECREF(retval);
-	    return NULL;
-    }
-
-    /* Make hex version of the digest */
-    for(i=j=0; i<digest_size; i++) {
-        char c;
-        c = (digest[i] >> 4) & 0xf;
-	c = (c>9) ? c+'a'-10 : c + '0';
-        hex_digest[j++] = c;
-        c = (digest[i] & 0xf);
-	c = (c>9) ? c+'a'-10 : c + '0';
-        hex_digest[j++] = c;
-    }
-    return retval;
-}
-
-PyDoc_STRVAR(EVP_update__doc__,
-"Update this hash object's state with the provided string.");
-
-static PyObject *
-EVP_update(EVPobject *self, PyObject *args)
-{
-    unsigned char *cp;
-    Py_ssize_t len;
-
-    if (!PyArg_ParseTuple(args, "s#:update", &cp, &len))
-        return NULL;
-
-    if (len > 0 && len <= MUNCH_SIZE) {
-    EVP_DigestUpdate(&self->ctx, cp, Py_SAFE_DOWNCAST(len, Py_ssize_t,
-                                                      unsigned int));
-    } else {
-        Py_ssize_t offset = 0;
-        while (len) {
-            unsigned int process = len > MUNCH_SIZE ? MUNCH_SIZE : len;
-            EVP_DigestUpdate(&self->ctx, cp + offset, process);
-            len -= process;
-            offset += process;
-        }
-    }
-    Py_INCREF(Py_None);
-    return Py_None;
-}
-
-static PyMethodDef EVP_methods[] = {
-    {"update",	  (PyCFunction)EVP_update,    METH_VARARGS, EVP_update__doc__},
-    {"digest",	  (PyCFunction)EVP_digest,    METH_NOARGS,  EVP_digest__doc__},
-    {"hexdigest", (PyCFunction)EVP_hexdigest, METH_NOARGS,  EVP_hexdigest__doc__},
-    {"copy",	  (PyCFunction)EVP_copy,      METH_NOARGS,  EVP_copy__doc__},
-    {NULL,	  NULL}		/* sentinel */
-};
-
-static PyObject *
-EVP_get_block_size(EVPobject *self, void *closure)
-{
-    return PyInt_FromLong(EVP_MD_CTX_block_size(&((EVPobject *)self)->ctx));
-}
-
-static PyObject *
-EVP_get_digest_size(EVPobject *self, void *closure)
-{
-    return PyInt_FromLong(EVP_MD_CTX_size(&((EVPobject *)self)->ctx));
-}
-
-static PyMemberDef EVP_members[] = {
-    {"name", T_OBJECT, offsetof(EVPobject, name), READONLY, PyDoc_STR("algorithm name.")},
-    {NULL}  /* Sentinel */
-};
-
-static PyGetSetDef EVP_getseters[] = {
-    {"digest_size",
-     (getter)EVP_get_digest_size, NULL,
-     NULL,
-     NULL},
-    {"block_size",
-     (getter)EVP_get_block_size, NULL,
-     NULL,
-     NULL},
-    /* the old md5 and sha modules support 'digest_size' as in PEP 247.
-     * the old sha module also supported 'digestsize'.  ugh. */
-    {"digestsize",
-     (getter)EVP_get_digest_size, NULL,
-     NULL,
-     NULL},
-    {NULL}  /* Sentinel */
-};
-
-
-static PyObject *
-EVP_repr(PyObject *self)
-{
-    char buf[100];
-    PyOS_snprintf(buf, sizeof(buf), "<%s HASH object @ %p>",
-            PyString_AsString(((EVPobject *)self)->name), self);
-    return PyString_FromString(buf);
-}
-
-#if HASH_OBJ_CONSTRUCTOR
-static int
-EVP_tp_init(EVPobject *self, PyObject *args, PyObject *kwds)
-{
-    static char *kwlist[] = {"name", "string", NULL};
-    PyObject *name_obj = NULL;
-    char *nameStr;
-    unsigned char *cp = NULL;
-    Py_ssize_t len = 0;
-    const EVP_MD *digest;
-
-    if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|s#:HASH", kwlist,
-                                     &name_obj, &cp, &len)) {
-        return -1;
-    }
-
-    if (!PyArg_Parse(name_obj, "s", &nameStr)) {
-        PyErr_SetString(PyExc_TypeError, "name must be a string");
-        return -1;
-    }
-
-    digest = EVP_get_digestbyname(nameStr);
-    if (!digest) {
-        PyErr_SetString(PyExc_ValueError, "unknown hash function");
-        return -1;
-    }
-    EVP_DigestInit(&self->ctx, digest);
-
-    self->name = name_obj;
-    Py_INCREF(self->name);
-
-    if (cp && len) {
-        if (len > 0 && len <= MUNCH_SIZE) {
-        EVP_DigestUpdate(&self->ctx, cp, Py_SAFE_DOWNCAST(len, Py_ssize_t,
-                                                          unsigned int));
-        } else {
-            Py_ssize_t offset = 0;
-            while (len) {
-                unsigned int process = len > MUNCH_SIZE ? MUNCH_SIZE : len;
-                EVP_DigestUpdate(&self->ctx, cp + offset, process);
-                len -= process;
-                offset += process;
-            }
-        }
-    }
-    
-    return 0;
-}
-#endif
-
-
-PyDoc_STRVAR(hashtype_doc,
-"A hash represents the object used to calculate a checksum of a\n\
-string of information.\n\
-\n\
-Methods:\n\
-\n\
-update() -- updates the current digest with an additional string\n\
-digest() -- return the current digest value\n\
-hexdigest() -- return the current digest as a string of hexadecimal digits\n\
-copy() -- return a copy of the current hash object\n\
-\n\
-Attributes:\n\
-\n\
-name -- the hash algorithm being used by this object\n\
-digest_size -- number of bytes in this hashes output\n");
-
-static PyTypeObject EVPtype = {
-    PyObject_HEAD_INIT(NULL)
-    0,			/*ob_size*/
-    "_hashlib.HASH",    /*tp_name*/
-    sizeof(EVPobject),	/*tp_basicsize*/
-    0,			/*tp_itemsize*/
-    /* methods */
-    EVP_dealloc,	/*tp_dealloc*/
-    0,			/*tp_print*/
-    0,                  /*tp_getattr*/
-    0,                  /*tp_setattr*/
-    0,                  /*tp_compare*/
-    EVP_repr,           /*tp_repr*/
-    0,                  /*tp_as_number*/
-    0,                  /*tp_as_sequence*/
-    0,                  /*tp_as_mapping*/
-    0,                  /*tp_hash*/
-    0,                  /*tp_call*/
-    0,                  /*tp_str*/
-    0,                  /*tp_getattro*/
-    0,                  /*tp_setattro*/
-    0,                  /*tp_as_buffer*/
-    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
-    hashtype_doc,       /*tp_doc*/
-    0,                  /*tp_traverse*/
-    0,			/*tp_clear*/
-    0,			/*tp_richcompare*/
-    0,			/*tp_weaklistoffset*/
-    0,			/*tp_iter*/
-    0,			/*tp_iternext*/
-    EVP_methods,	/* tp_methods */
-    EVP_members,	/* tp_members */
-    EVP_getseters,      /* tp_getset */
-#if 1
-    0,                  /* tp_base */
-    0,                  /* tp_dict */
-    0,                  /* tp_descr_get */
-    0,                  /* tp_descr_set */
-    0,                  /* tp_dictoffset */
-#endif
-#if HASH_OBJ_CONSTRUCTOR
-    (initproc)EVP_tp_init, /* tp_init */
-#endif
-};
-
-static PyObject *
-EVPnew(PyObject *name_obj,
-       const EVP_MD *digest, const EVP_MD_CTX *initial_ctx,
-       const unsigned char *cp, Py_ssize_t len)
-{
-    EVPobject *self;
-
-    if (!digest && !initial_ctx) {
-        PyErr_SetString(PyExc_ValueError, "unsupported hash type");
-        return NULL;
-    }
-
-    if ((self = newEVPobject(name_obj)) == NULL)
-        return NULL;
-
-    if (initial_ctx) {
-        EVP_MD_CTX_copy(&self->ctx, initial_ctx);
-    } else {
-        EVP_DigestInit(&self->ctx, digest);
-    }
-
-    if (cp && len) {
-        if (len > 0 && len <= MUNCH_SIZE) {
-            EVP_DigestUpdate(&self->ctx, cp, Py_SAFE_DOWNCAST(len, Py_ssize_t,
-                                                              unsigned int));
-        } else {
-            Py_ssize_t offset = 0;
-            while (len) {
-                unsigned int process = len > MUNCH_SIZE ? MUNCH_SIZE : len;
-                EVP_DigestUpdate(&self->ctx, cp + offset, process);
-                len -= process;
-                offset += process;
-            }
-        }
-    }
-
-    return (PyObject *)self;
-}
-
-
-/* The module-level function: new() */
-
-PyDoc_STRVAR(EVP_new__doc__,
-"Return a new hash object using the named algorithm.\n\
-An optional string argument may be provided and will be\n\
-automatically hashed.\n\
-\n\
-The MD5 and SHA1 algorithms are always supported.\n");
-
-static PyObject *
-EVP_new(PyObject *self, PyObject *args, PyObject *kwdict)
-{
-    static char *kwlist[] = {"name", "string", NULL};
-    PyObject *name_obj = NULL;
-    char *name;
-    const EVP_MD *digest;
-    unsigned char *cp = NULL;
-    Py_ssize_t len = 0;
-
-    if (!PyArg_ParseTupleAndKeywords(args, kwdict, "O|s#:new", kwlist,
-                                     &name_obj, &cp, &len)) {
-        return NULL;
-    }
-
-    if (!PyArg_Parse(name_obj, "s", &name)) {
-        PyErr_SetString(PyExc_TypeError, "name must be a string");
-        return NULL;
-    }
-
-    digest = EVP_get_digestbyname(name);
-
-    return EVPnew(name_obj, digest, NULL, cp, len);
-}
-
-/*
- *  This macro generates constructor function definitions for specific
- *  hash algorithms.  These constructors are much faster than calling
- *  the generic one passing it a python string and are noticably
- *  faster than calling a python new() wrapper.  Thats important for
- *  code that wants to make hashes of a bunch of small strings.
- */
-#define GEN_CONSTRUCTOR(NAME)  \
-    static PyObject * \
-    EVP_new_ ## NAME (PyObject *self, PyObject *args) \
-    { \
-        unsigned char *cp = NULL; \
-        Py_ssize_t len = 0; \
-     \
-        if (!PyArg_ParseTuple(args, "|s#:" #NAME , &cp, &len)) { \
-            return NULL; \
-        } \
-     \
-        return EVPnew( \
-                CONST_ ## NAME ## _name_obj, \
-                NULL, \
-                CONST_new_ ## NAME ## _ctx_p, \
-                cp, len); \
-    }
-
-/* a PyMethodDef structure for the constructor */
-#define CONSTRUCTOR_METH_DEF(NAME)  \
-    {"openssl_" #NAME, (PyCFunction)EVP_new_ ## NAME, METH_VARARGS, \
-        PyDoc_STR("Returns a " #NAME \
-                  " hash object; optionally initialized with a string") \
-    }
-
-/* used in the init function to setup a constructor */
-#define INIT_CONSTRUCTOR_CONSTANTS(NAME)  do { \
-    CONST_ ## NAME ## _name_obj = PyString_FromString(#NAME); \
-    if (EVP_get_digestbyname(#NAME)) { \
-        CONST_new_ ## NAME ## _ctx_p = &CONST_new_ ## NAME ## _ctx; \
-        EVP_DigestInit(CONST_new_ ## NAME ## _ctx_p, EVP_get_digestbyname(#NAME)); \
-    } \
-} while (0);
-
-GEN_CONSTRUCTOR(md5)
-GEN_CONSTRUCTOR(sha1)
-GEN_CONSTRUCTOR(sha224)
-GEN_CONSTRUCTOR(sha256)
-GEN_CONSTRUCTOR(sha384)
-GEN_CONSTRUCTOR(sha512)
-
-/* List of functions exported by this module */
-
-static struct PyMethodDef EVP_functions[] = {
-    {"new", (PyCFunction)EVP_new, METH_VARARGS|METH_KEYWORDS, EVP_new__doc__},
-    CONSTRUCTOR_METH_DEF(md5),
-    CONSTRUCTOR_METH_DEF(sha1),
-    CONSTRUCTOR_METH_DEF(sha224),
-    CONSTRUCTOR_METH_DEF(sha256),
-    CONSTRUCTOR_METH_DEF(sha384),
-    CONSTRUCTOR_METH_DEF(sha512),
-    {NULL,	NULL}		 /* Sentinel */
-};
-
-
-/* Initialize this module. */
-
-PyMODINIT_FUNC
-init_hashlib(void)
-{
-    PyObject *m;
-
-    OpenSSL_add_all_digests();
-
-    /* TODO build EVP_functions openssl_* entries dynamically based
-     * on what hashes are supported rather than listing many
-     * but having some be unsupported.  Only init appropriate
-     * constants. */
-
-    EVPtype.ob_type = &PyType_Type;
-    if (PyType_Ready(&EVPtype) < 0)
-        return;
-
-    m = Py_InitModule("_hashlib", EVP_functions);
-    if (m == NULL)
-        return;
-
-#if HASH_OBJ_CONSTRUCTOR
-    Py_INCREF(&EVPtype);
-    PyModule_AddObject(m, "HASH", (PyObject *)&EVPtype);
-#endif
-
-    /* these constants are used by the convenience constructors */
-    INIT_CONSTRUCTOR_CONSTANTS(md5);
-    INIT_CONSTRUCTOR_CONSTANTS(sha1);
-    INIT_CONSTRUCTOR_CONSTANTS(sha224);
-    INIT_CONSTRUCTOR_CONSTANTS(sha256);
-    INIT_CONSTRUCTOR_CONSTANTS(sha384);
-    INIT_CONSTRUCTOR_CONSTANTS(sha512);
-}
diff --git a/distutils2/_backport/hashlib.py b/distutils2/_backport/hashlib.py
deleted file mode 100644
--- a/distutils2/_backport/hashlib.py
+++ /dev/null
@@ -1,143 +0,0 @@
-# $Id$
-#
-#  Copyright (C) 2005   Gregory P. Smith (greg at krypto.org)
-#  Licensed to PSF under a Contributor Agreement.
-#
-
-__doc__ = """hashlib module - A common interface to many hash functions.
-
-new(name, string='') - returns a new hash object implementing the
-                       given hash function; initializing the hash
-                       using the given string data.
-
-Named constructor functions are also available, these are much faster
-than using new():
-
-md5(), sha1(), sha224(), sha256(), sha384(), and sha512()
-
-More algorithms may be available on your platform but the above are
-guaranteed to exist.
-
-NOTE: If you want the adler32 or crc32 hash functions they are available in
-the zlib module.
-
-Choose your hash function wisely.  Some have known collision weaknesses.
-sha384 and sha512 will be slow on 32 bit platforms.
-
-Hash objects have these methods:
- - update(arg): Update the hash object with the string arg. Repeated calls
-                are equivalent to a single call with the concatenation of all
-                the arguments.
- - digest():    Return the digest of the strings passed to the update() method
-                so far. This may contain non-ASCII characters, including
-                NUL bytes.
- - hexdigest(): Like digest() except the digest is returned as a string of
-                double length, containing only hexadecimal digits.
- - copy():      Return a copy (clone) of the hash object. This can be used to
-                efficiently compute the digests of strings that share a common
-                initial substring.
-
-For example, to obtain the digest of the string 'Nobody inspects the
-spammish repetition':
-
-    >>> import hashlib
-    >>> m = hashlib.md5()
-    >>> m.update("Nobody inspects")
-    >>> m.update(" the spammish repetition")
-    >>> m.digest()
-    '\\xbbd\\x9c\\x83\\xdd\\x1e\\xa5\\xc9\\xd9\\xde\\xc9\\xa1\\x8d\\xf0\\xff\\xe9'
-
-More condensed:
-
-    >>> hashlib.sha224("Nobody inspects the spammish repetition").hexdigest()
-    'a4337bc45a8fc544c03f52dc550cd6e1e87021bc896588bd79e901e2'
-
-"""
-
-# This tuple and __get_builtin_constructor() must be modified if a new
-# always available algorithm is added.
-__always_supported = ('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512')
-
-algorithms = __always_supported
-
-__all__ = __always_supported + ('new', 'algorithms')
-
-
-def __get_builtin_constructor(name):
-    if name in ('SHA1', 'sha1'):
-        from distutils2._backport import _sha
-        return _sha.new
-    elif name in ('MD5', 'md5'):
-        from distutils2._backport import _md5
-        return _md5.new
-    elif name in ('SHA256', 'sha256', 'SHA224', 'sha224'):
-        from distutils2._backport import _sha256
-        bs = name[3:]
-        if bs == '256':
-            return _sha256.sha256
-        elif bs == '224':
-            return _sha256.sha224
-    elif name in ('SHA512', 'sha512', 'SHA384', 'sha384'):
-        from distutils2._backport import _sha512
-        bs = name[3:]
-        if bs == '512':
-            return _sha512.sha512
-        elif bs == '384':
-            return _sha512.sha384
-
-    raise ValueError('unsupported hash type %s' % name)
-
-
-def __get_openssl_constructor(name):
-    try:
-        f = getattr(_hashlib, 'openssl_' + name)
-        # Allow the C module to raise ValueError.  The function will be
-        # defined but the hash not actually available thanks to OpenSSL.
-        f()
-        # Use the C function directly (very fast)
-        return f
-    except (AttributeError, ValueError):
-        return __get_builtin_constructor(name)
-
-
-def __py_new(name, string=''):
-    """new(name, string='') - Return a new hashing object using the named algorithm;
-    optionally initialized with a string.
-    """
-    return __get_builtin_constructor(name)(string)
-
-
-def __hash_new(name, string=''):
-    """new(name, string='') - Return a new hashing object using the named algorithm;
-    optionally initialized with a string.
-    """
-    try:
-        return _hashlib.new(name, string)
-    except ValueError:
-        # If the _hashlib module (OpenSSL) doesn't support the named
-        # hash, try using our builtin implementations.
-        # This allows for SHA224/256 and SHA384/512 support even though
-        # the OpenSSL library prior to 0.9.8 doesn't provide them.
-        return __get_builtin_constructor(name)(string)
-
-
-try:
-    from distutils2._backport import _hashlib
-    new = __hash_new
-    __get_hash = __get_openssl_constructor
-except ImportError:
-    new = __py_new
-    __get_hash = __get_builtin_constructor
-
-for __func_name in __always_supported:
-    # try them all, some may not work due to the OpenSSL
-    # version not supporting that algorithm.
-    try:
-        globals()[__func_name] = __get_hash(__func_name)
-    except ValueError:
-        import logging
-        logging.exception('code for hash %s was not found.', __func_name)
-
-# Cleanup locals()
-del __always_supported, __func_name, __get_hash
-del __py_new, __hash_new, __get_openssl_constructor
diff --git a/distutils2/_backport/md5.c b/distutils2/_backport/md5.c
deleted file mode 100644
--- a/distutils2/_backport/md5.c
+++ /dev/null
@@ -1,381 +0,0 @@
-/*
-  Copyright (C) 1999, 2000, 2002 Aladdin Enterprises.  All rights reserved.
-
-  This software is provided 'as-is', without any express or implied
-  warranty.  In no event will the authors be held liable for any damages
-  arising from the use of this software.
-
-  Permission is granted to anyone to use this software for any purpose,
-  including commercial applications, and to alter it and redistribute it
-  freely, subject to the following restrictions:
-
-  1. The origin of this software must not be misrepresented; you must not
-     claim that you wrote the original software. If you use this software
-     in a product, an acknowledgment in the product documentation would be
-     appreciated but is not required.
-  2. Altered source versions must be plainly marked as such, and must not be
-     misrepresented as being the original software.
-  3. This notice may not be removed or altered from any source distribution.
-
-  L. Peter Deutsch
-  ghost at aladdin.com
-
- */
-/* $Id: md5.c,v 1.6 2002/04/13 19:20:28 lpd Exp $ */
-/*
-  Independent implementation of MD5 (RFC 1321).
-
-  This code implements the MD5 Algorithm defined in RFC 1321, whose
-  text is available at
-	http://www.ietf.org/rfc/rfc1321.txt
-  The code is derived from the text of the RFC, including the test suite
-  (section A.5) but excluding the rest of Appendix A.  It does not include
-  any code or documentation that is identified in the RFC as being
-  copyrighted.
-
-  The original and principal author of md5.c is L. Peter Deutsch
-  <ghost at aladdin.com>.  Other authors are noted in the change history
-  that follows (in reverse chronological order):
-
-  2002-04-13 lpd Clarified derivation from RFC 1321; now handles byte order
-	either statically or dynamically; added missing #include <string.h>
-	in library.
-  2002-03-11 lpd Corrected argument list for main(), and added int return
-	type, in test program and T value program.
-  2002-02-21 lpd Added missing #include <stdio.h> in test program.
-  2000-07-03 lpd Patched to eliminate warnings about "constant is
-	unsigned in ANSI C, signed in traditional"; made test program
-	self-checking.
-  1999-11-04 lpd Edited comments slightly for automatic TOC extraction.
-  1999-10-18 lpd Fixed typo in header comment (ansi2knr rather than md5).
-  1999-05-03 lpd Original version.
- */
-
-#include "md5.h"
-#include <string.h>
-
-#undef BYTE_ORDER	/* 1 = big-endian, -1 = little-endian, 0 = unknown */
-#ifdef ARCH_IS_BIG_ENDIAN
-#  define BYTE_ORDER (ARCH_IS_BIG_ENDIAN ? 1 : -1)
-#else
-#  define BYTE_ORDER 0
-#endif
-
-#define T_MASK ((md5_word_t)~0)
-#define T1 /* 0xd76aa478 */ (T_MASK ^ 0x28955b87)
-#define T2 /* 0xe8c7b756 */ (T_MASK ^ 0x173848a9)
-#define T3    0x242070db
-#define T4 /* 0xc1bdceee */ (T_MASK ^ 0x3e423111)
-#define T5 /* 0xf57c0faf */ (T_MASK ^ 0x0a83f050)
-#define T6    0x4787c62a
-#define T7 /* 0xa8304613 */ (T_MASK ^ 0x57cfb9ec)
-#define T8 /* 0xfd469501 */ (T_MASK ^ 0x02b96afe)
-#define T9    0x698098d8
-#define T10 /* 0x8b44f7af */ (T_MASK ^ 0x74bb0850)
-#define T11 /* 0xffff5bb1 */ (T_MASK ^ 0x0000a44e)
-#define T12 /* 0x895cd7be */ (T_MASK ^ 0x76a32841)
-#define T13    0x6b901122
-#define T14 /* 0xfd987193 */ (T_MASK ^ 0x02678e6c)
-#define T15 /* 0xa679438e */ (T_MASK ^ 0x5986bc71)
-#define T16    0x49b40821
-#define T17 /* 0xf61e2562 */ (T_MASK ^ 0x09e1da9d)
-#define T18 /* 0xc040b340 */ (T_MASK ^ 0x3fbf4cbf)
-#define T19    0x265e5a51
-#define T20 /* 0xe9b6c7aa */ (T_MASK ^ 0x16493855)
-#define T21 /* 0xd62f105d */ (T_MASK ^ 0x29d0efa2)
-#define T22    0x02441453
-#define T23 /* 0xd8a1e681 */ (T_MASK ^ 0x275e197e)
-#define T24 /* 0xe7d3fbc8 */ (T_MASK ^ 0x182c0437)
-#define T25    0x21e1cde6
-#define T26 /* 0xc33707d6 */ (T_MASK ^ 0x3cc8f829)
-#define T27 /* 0xf4d50d87 */ (T_MASK ^ 0x0b2af278)
-#define T28    0x455a14ed
-#define T29 /* 0xa9e3e905 */ (T_MASK ^ 0x561c16fa)
-#define T30 /* 0xfcefa3f8 */ (T_MASK ^ 0x03105c07)
-#define T31    0x676f02d9
-#define T32 /* 0x8d2a4c8a */ (T_MASK ^ 0x72d5b375)
-#define T33 /* 0xfffa3942 */ (T_MASK ^ 0x0005c6bd)
-#define T34 /* 0x8771f681 */ (T_MASK ^ 0x788e097e)
-#define T35    0x6d9d6122
-#define T36 /* 0xfde5380c */ (T_MASK ^ 0x021ac7f3)
-#define T37 /* 0xa4beea44 */ (T_MASK ^ 0x5b4115bb)
-#define T38    0x4bdecfa9
-#define T39 /* 0xf6bb4b60 */ (T_MASK ^ 0x0944b49f)
-#define T40 /* 0xbebfbc70 */ (T_MASK ^ 0x4140438f)
-#define T41    0x289b7ec6
-#define T42 /* 0xeaa127fa */ (T_MASK ^ 0x155ed805)
-#define T43 /* 0xd4ef3085 */ (T_MASK ^ 0x2b10cf7a)
-#define T44    0x04881d05
-#define T45 /* 0xd9d4d039 */ (T_MASK ^ 0x262b2fc6)
-#define T46 /* 0xe6db99e5 */ (T_MASK ^ 0x1924661a)
-#define T47    0x1fa27cf8
-#define T48 /* 0xc4ac5665 */ (T_MASK ^ 0x3b53a99a)
-#define T49 /* 0xf4292244 */ (T_MASK ^ 0x0bd6ddbb)
-#define T50    0x432aff97
-#define T51 /* 0xab9423a7 */ (T_MASK ^ 0x546bdc58)
-#define T52 /* 0xfc93a039 */ (T_MASK ^ 0x036c5fc6)
-#define T53    0x655b59c3
-#define T54 /* 0x8f0ccc92 */ (T_MASK ^ 0x70f3336d)
-#define T55 /* 0xffeff47d */ (T_MASK ^ 0x00100b82)
-#define T56 /* 0x85845dd1 */ (T_MASK ^ 0x7a7ba22e)
-#define T57    0x6fa87e4f
-#define T58 /* 0xfe2ce6e0 */ (T_MASK ^ 0x01d3191f)
-#define T59 /* 0xa3014314 */ (T_MASK ^ 0x5cfebceb)
-#define T60    0x4e0811a1
-#define T61 /* 0xf7537e82 */ (T_MASK ^ 0x08ac817d)
-#define T62 /* 0xbd3af235 */ (T_MASK ^ 0x42c50dca)
-#define T63    0x2ad7d2bb
-#define T64 /* 0xeb86d391 */ (T_MASK ^ 0x14792c6e)
-
-
-static void
-md5_process(md5_state_t *pms, const md5_byte_t *data /*[64]*/)
-{
-    md5_word_t
-	a = pms->abcd[0], b = pms->abcd[1],
-	c = pms->abcd[2], d = pms->abcd[3];
-    md5_word_t t;
-#if BYTE_ORDER > 0
-    /* Define storage only for big-endian CPUs. */
-    md5_word_t X[16];
-#else
-    /* Define storage for little-endian or both types of CPUs. */
-    md5_word_t xbuf[16];
-    const md5_word_t *X;
-#endif
-
-    {
-#if BYTE_ORDER == 0
-	/*
-	 * Determine dynamically whether this is a big-endian or
-	 * little-endian machine, since we can use a more efficient
-	 * algorithm on the latter.
-	 */
-	static const int w = 1;
-
-	if (*((const md5_byte_t *)&w)) /* dynamic little-endian */
-#endif
-#if BYTE_ORDER <= 0		/* little-endian */
-	{
-	    /*
-	     * On little-endian machines, we can process properly aligned
-	     * data without copying it.
-	     */
-	    if (!((data - (const md5_byte_t *)0) & 3)) {
-		/* data are properly aligned */
-		X = (const md5_word_t *)data;
-	    } else {
-		/* not aligned */
-		memcpy(xbuf, data, 64);
-		X = xbuf;
-	    }
-	}
-#endif
-#if BYTE_ORDER == 0
-	else			/* dynamic big-endian */
-#endif
-#if BYTE_ORDER >= 0		/* big-endian */
-	{
-	    /*
-	     * On big-endian machines, we must arrange the bytes in the
-	     * right order.
-	     */
-	    const md5_byte_t *xp = data;
-	    int i;
-
-#  if BYTE_ORDER == 0
-	    X = xbuf;		/* (dynamic only) */
-#  else
-#    define xbuf X		/* (static only) */
-#  endif
-	    for (i = 0; i < 16; ++i, xp += 4)
-		xbuf[i] = xp[0] + (xp[1] << 8) + (xp[2] << 16) + (xp[3] << 24);
-	}
-#endif
-    }
-
-#define ROTATE_LEFT(x, n) (((x) << (n)) | ((x) >> (32 - (n))))
-
-    /* Round 1. */
-    /* Let [abcd k s i] denote the operation
-       a = b + ((a + F(b,c,d) + X[k] + T[i]) <<< s). */
-#define F(x, y, z) (((x) & (y)) | (~(x) & (z)))
-#define SET(a, b, c, d, k, s, Ti)\
-  t = a + F(b,c,d) + X[k] + Ti;\
-  a = ROTATE_LEFT(t, s) + b
-    /* Do the following 16 operations. */
-    SET(a, b, c, d,  0,  7,  T1);
-    SET(d, a, b, c,  1, 12,  T2);
-    SET(c, d, a, b,  2, 17,  T3);
-    SET(b, c, d, a,  3, 22,  T4);
-    SET(a, b, c, d,  4,  7,  T5);
-    SET(d, a, b, c,  5, 12,  T6);
-    SET(c, d, a, b,  6, 17,  T7);
-    SET(b, c, d, a,  7, 22,  T8);
-    SET(a, b, c, d,  8,  7,  T9);
-    SET(d, a, b, c,  9, 12, T10);
-    SET(c, d, a, b, 10, 17, T11);
-    SET(b, c, d, a, 11, 22, T12);
-    SET(a, b, c, d, 12,  7, T13);
-    SET(d, a, b, c, 13, 12, T14);
-    SET(c, d, a, b, 14, 17, T15);
-    SET(b, c, d, a, 15, 22, T16);
-#undef SET
-
-     /* Round 2. */
-     /* Let [abcd k s i] denote the operation
-          a = b + ((a + G(b,c,d) + X[k] + T[i]) <<< s). */
-#define G(x, y, z) (((x) & (z)) | ((y) & ~(z)))
-#define SET(a, b, c, d, k, s, Ti)\
-  t = a + G(b,c,d) + X[k] + Ti;\
-  a = ROTATE_LEFT(t, s) + b
-     /* Do the following 16 operations. */
-    SET(a, b, c, d,  1,  5, T17);
-    SET(d, a, b, c,  6,  9, T18);
-    SET(c, d, a, b, 11, 14, T19);
-    SET(b, c, d, a,  0, 20, T20);
-    SET(a, b, c, d,  5,  5, T21);
-    SET(d, a, b, c, 10,  9, T22);
-    SET(c, d, a, b, 15, 14, T23);
-    SET(b, c, d, a,  4, 20, T24);
-    SET(a, b, c, d,  9,  5, T25);
-    SET(d, a, b, c, 14,  9, T26);
-    SET(c, d, a, b,  3, 14, T27);
-    SET(b, c, d, a,  8, 20, T28);
-    SET(a, b, c, d, 13,  5, T29);
-    SET(d, a, b, c,  2,  9, T30);
-    SET(c, d, a, b,  7, 14, T31);
-    SET(b, c, d, a, 12, 20, T32);
-#undef SET
-
-     /* Round 3. */
-     /* Let [abcd k s t] denote the operation
-          a = b + ((a + H(b,c,d) + X[k] + T[i]) <<< s). */
-#define H(x, y, z) ((x) ^ (y) ^ (z))
-#define SET(a, b, c, d, k, s, Ti)\
-  t = a + H(b,c,d) + X[k] + Ti;\
-  a = ROTATE_LEFT(t, s) + b
-     /* Do the following 16 operations. */
-    SET(a, b, c, d,  5,  4, T33);
-    SET(d, a, b, c,  8, 11, T34);
-    SET(c, d, a, b, 11, 16, T35);
-    SET(b, c, d, a, 14, 23, T36);
-    SET(a, b, c, d,  1,  4, T37);
-    SET(d, a, b, c,  4, 11, T38);
-    SET(c, d, a, b,  7, 16, T39);
-    SET(b, c, d, a, 10, 23, T40);
-    SET(a, b, c, d, 13,  4, T41);
-    SET(d, a, b, c,  0, 11, T42);
-    SET(c, d, a, b,  3, 16, T43);
-    SET(b, c, d, a,  6, 23, T44);
-    SET(a, b, c, d,  9,  4, T45);
-    SET(d, a, b, c, 12, 11, T46);
-    SET(c, d, a, b, 15, 16, T47);
-    SET(b, c, d, a,  2, 23, T48);
-#undef SET
-
-     /* Round 4. */
-     /* Let [abcd k s t] denote the operation
-          a = b + ((a + I(b,c,d) + X[k] + T[i]) <<< s). */
-#define I(x, y, z) ((y) ^ ((x) | ~(z)))
-#define SET(a, b, c, d, k, s, Ti)\
-  t = a + I(b,c,d) + X[k] + Ti;\
-  a = ROTATE_LEFT(t, s) + b
-     /* Do the following 16 operations. */
-    SET(a, b, c, d,  0,  6, T49);
-    SET(d, a, b, c,  7, 10, T50);
-    SET(c, d, a, b, 14, 15, T51);
-    SET(b, c, d, a,  5, 21, T52);
-    SET(a, b, c, d, 12,  6, T53);
-    SET(d, a, b, c,  3, 10, T54);
-    SET(c, d, a, b, 10, 15, T55);
-    SET(b, c, d, a,  1, 21, T56);
-    SET(a, b, c, d,  8,  6, T57);
-    SET(d, a, b, c, 15, 10, T58);
-    SET(c, d, a, b,  6, 15, T59);
-    SET(b, c, d, a, 13, 21, T60);
-    SET(a, b, c, d,  4,  6, T61);
-    SET(d, a, b, c, 11, 10, T62);
-    SET(c, d, a, b,  2, 15, T63);
-    SET(b, c, d, a,  9, 21, T64);
-#undef SET
-
-     /* Then perform the following additions. (That is increment each
-        of the four registers by the value it had before this block
-        was started.) */
-    pms->abcd[0] += a;
-    pms->abcd[1] += b;
-    pms->abcd[2] += c;
-    pms->abcd[3] += d;
-}
-
-void
-md5_init(md5_state_t *pms)
-{
-    pms->count[0] = pms->count[1] = 0;
-    pms->abcd[0] = 0x67452301;
-    pms->abcd[1] = /*0xefcdab89*/ T_MASK ^ 0x10325476;
-    pms->abcd[2] = /*0x98badcfe*/ T_MASK ^ 0x67452301;
-    pms->abcd[3] = 0x10325476;
-}
-
-void
-md5_append(md5_state_t *pms, const md5_byte_t *data, int nbytes)
-{
-    const md5_byte_t *p = data;
-    int left = nbytes;
-    int offset = (pms->count[0] >> 3) & 63;
-    md5_word_t nbits = (md5_word_t)(nbytes << 3);
-
-    if (nbytes <= 0)
-	return;
-
-    /* Update the message length. */
-    pms->count[1] += nbytes >> 29;
-    pms->count[0] += nbits;
-    if (pms->count[0] < nbits)
-	pms->count[1]++;
-
-    /* Process an initial partial block. */
-    if (offset) {
-	int copy = (offset + nbytes > 64 ? 64 - offset : nbytes);
-
-	memcpy(pms->buf + offset, p, copy);
-	if (offset + copy < 64)
-	    return;
-	p += copy;
-	left -= copy;
-	md5_process(pms, pms->buf);
-    }
-
-    /* Process full blocks. */
-    for (; left >= 64; p += 64, left -= 64)
-	md5_process(pms, p);
-
-    /* Process a final partial block. */
-    if (left)
-	memcpy(pms->buf, p, left);
-}
-
-void
-md5_finish(md5_state_t *pms, md5_byte_t digest[16])
-{
-    static const md5_byte_t pad[64] = {
-	0x80, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
-	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
-	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
-	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
-    };
-    md5_byte_t data[8];
-    int i;
-
-    /* Save the length before padding. */
-    for (i = 0; i < 8; ++i)
-	data[i] = (md5_byte_t)(pms->count[i >> 2] >> ((i & 3) << 3));
-    /* Pad to 56 bytes mod 64. */
-    md5_append(pms, pad, ((55 - (pms->count[0] >> 3)) & 63) + 1);
-    /* Append the length. */
-    md5_append(pms, data, 8);
-    for (i = 0; i < 16; ++i)
-	digest[i] = (md5_byte_t)(pms->abcd[i >> 2] >> ((i & 3) << 3));
-}
diff --git a/distutils2/_backport/md5.h b/distutils2/_backport/md5.h
deleted file mode 100644
--- a/distutils2/_backport/md5.h
+++ /dev/null
@@ -1,91 +0,0 @@
-/*
-  Copyright (C) 1999, 2002 Aladdin Enterprises.  All rights reserved.
-
-  This software is provided 'as-is', without any express or implied
-  warranty.  In no event will the authors be held liable for any damages
-  arising from the use of this software.
-
-  Permission is granted to anyone to use this software for any purpose,
-  including commercial applications, and to alter it and redistribute it
-  freely, subject to the following restrictions:
-
-  1. The origin of this software must not be misrepresented; you must not
-     claim that you wrote the original software. If you use this software
-     in a product, an acknowledgment in the product documentation would be
-     appreciated but is not required.
-  2. Altered source versions must be plainly marked as such, and must not be
-     misrepresented as being the original software.
-  3. This notice may not be removed or altered from any source distribution.
-
-  L. Peter Deutsch
-  ghost at aladdin.com
-
- */
-/* $Id: md5.h 43594 2006-04-03 16:27:50Z matthias.klose $ */
-/*
-  Independent implementation of MD5 (RFC 1321).
-
-  This code implements the MD5 Algorithm defined in RFC 1321, whose
-  text is available at
-	http://www.ietf.org/rfc/rfc1321.txt
-  The code is derived from the text of the RFC, including the test suite
-  (section A.5) but excluding the rest of Appendix A.  It does not include
-  any code or documentation that is identified in the RFC as being
-  copyrighted.
-
-  The original and principal author of md5.h is L. Peter Deutsch
-  <ghost at aladdin.com>.  Other authors are noted in the change history
-  that follows (in reverse chronological order):
-
-  2002-04-13 lpd Removed support for non-ANSI compilers; removed
-	references to Ghostscript; clarified derivation from RFC 1321;
-	now handles byte order either statically or dynamically.
-  1999-11-04 lpd Edited comments slightly for automatic TOC extraction.
-  1999-10-18 lpd Fixed typo in header comment (ansi2knr rather than md5);
-	added conditionalization for C++ compilation from Martin
-	Purschke <purschke at bnl.gov>.
-  1999-05-03 lpd Original version.
- */
-
-#ifndef md5_INCLUDED
-#  define md5_INCLUDED
-
-/*
- * This package supports both compile-time and run-time determination of CPU
- * byte order.  If ARCH_IS_BIG_ENDIAN is defined as 0, the code will be
- * compiled to run only on little-endian CPUs; if ARCH_IS_BIG_ENDIAN is
- * defined as non-zero, the code will be compiled to run only on big-endian
- * CPUs; if ARCH_IS_BIG_ENDIAN is not defined, the code will be compiled to
- * run on either big- or little-endian CPUs, but will run slightly less
- * efficiently on either one than if ARCH_IS_BIG_ENDIAN is defined.
- */
-
-typedef unsigned char md5_byte_t; /* 8-bit byte */
-typedef unsigned int md5_word_t; /* 32-bit word */
-
-/* Define the state of the MD5 Algorithm. */
-typedef struct md5_state_s {
-    md5_word_t count[2];	/* message length in bits, lsw first */
-    md5_word_t abcd[4];		/* digest buffer */
-    md5_byte_t buf[64];		/* accumulate block */
-} md5_state_t;
-
-#ifdef __cplusplus
-extern "C" 
-{
-#endif
-
-/* Initialize the algorithm. */
-void md5_init(md5_state_t *pms);
-
-/* Append a string to the message. */
-void md5_append(md5_state_t *pms, const md5_byte_t *data, int nbytes);
-
-/* Finish the message and return the digest. */
-void md5_finish(md5_state_t *pms, md5_byte_t digest[16]);
-
-#ifdef __cplusplus
-}  /* end extern "C" */
-#endif
-
-#endif /* md5_INCLUDED */
diff --git a/distutils2/_backport/md5module.c b/distutils2/_backport/md5module.c
deleted file mode 100644
--- a/distutils2/_backport/md5module.c
+++ /dev/null
@@ -1,312 +0,0 @@
-
-/* MD5 module */
-
-/* This module provides an interface to the RSA Data Security,
-   Inc. MD5 Message-Digest Algorithm, described in RFC 1321.
-   It requires the files md5c.c and md5.h (which are slightly changed
-   from the versions in the RFC to avoid the "global.h" file.) */
-
-
-/* MD5 objects */
-
-#include "Python.h"
-#include "structmember.h"
-#include "md5.h"
-
-typedef struct {
-	PyObject_HEAD
-        md5_state_t	md5;		/* the context holder */
-} md5object;
-
-static PyTypeObject MD5type;
-
-#define is_md5object(v)		((v)->ob_type == &MD5type)
-
-static md5object *
-newmd5object(void)
-{
-	md5object *md5p;
-
-	md5p = PyObject_New(md5object, &MD5type);
-	if (md5p == NULL)
-		return NULL;
-
-	md5_init(&md5p->md5);	/* actual initialisation */
-	return md5p;
-}
-
-
-/* MD5 methods */
-
-static void
-md5_dealloc(md5object *md5p)
-{
-	PyObject_Del(md5p);
-}
-
-
-/* MD5 methods-as-attributes */
-
-static PyObject *
-md5_update(md5object *self, PyObject *args)
-{
-	unsigned char *cp;
-	int len;
-
-	if (!PyArg_ParseTuple(args, "s#:update", &cp, &len))
-		return NULL;
-
-	md5_append(&self->md5, cp, len);
-
-	Py_INCREF(Py_None);
-	return Py_None;
-}
-
-PyDoc_STRVAR(update_doc,
-"update (arg)\n\
-\n\
-Update the md5 object with the string arg. Repeated calls are\n\
-equivalent to a single call with the concatenation of all the\n\
-arguments.");
-
-
-static PyObject *
-md5_digest(md5object *self)
-{
- 	md5_state_t mdContext;
-	unsigned char aDigest[16];
-
-	/* make a temporary copy, and perform the final */
-	mdContext = self->md5;
-	md5_finish(&mdContext, aDigest);
-
-	return PyString_FromStringAndSize((char *)aDigest, 16);
-}
-
-PyDoc_STRVAR(digest_doc,
-"digest() -> string\n\
-\n\
-Return the digest of the strings passed to the update() method so\n\
-far. This is a 16-byte string which may contain non-ASCII characters,\n\
-including null bytes.");
-
-
-static PyObject *
-md5_hexdigest(md5object *self)
-{
- 	md5_state_t mdContext;
-	unsigned char digest[16];
-	unsigned char hexdigest[32];
-	int i, j;
-
-	/* make a temporary copy, and perform the final */
-	mdContext = self->md5;
-	md5_finish(&mdContext, digest);
-
-	/* Make hex version of the digest */
-	for(i=j=0; i<16; i++) {
-		char c;
-		c = (digest[i] >> 4) & 0xf;
-		c = (c>9) ? c+'a'-10 : c + '0';
-		hexdigest[j++] = c;
-		c = (digest[i] & 0xf);
-		c = (c>9) ? c+'a'-10 : c + '0';
-		hexdigest[j++] = c;
-	}
-	return PyString_FromStringAndSize((char*)hexdigest, 32);
-}
-
-
-PyDoc_STRVAR(hexdigest_doc,
-"hexdigest() -> string\n\
-\n\
-Like digest(), but returns the digest as a string of hexadecimal digits.");
-
-
-static PyObject *
-md5_copy(md5object *self)
-{
-	md5object *md5p;
-
-	if ((md5p = newmd5object()) == NULL)
-		return NULL;
-
-	md5p->md5 = self->md5;
-
-	return (PyObject *)md5p;
-}
-
-PyDoc_STRVAR(copy_doc,
-"copy() -> md5 object\n\
-\n\
-Return a copy (``clone'') of the md5 object.");
-
-
-static PyMethodDef md5_methods[] = {
-	{"update",    (PyCFunction)md5_update,    METH_VARARGS, update_doc},
-	{"digest",    (PyCFunction)md5_digest,    METH_NOARGS,  digest_doc},
-	{"hexdigest", (PyCFunction)md5_hexdigest, METH_NOARGS,  hexdigest_doc},
-	{"copy",      (PyCFunction)md5_copy,      METH_NOARGS,  copy_doc},
-	{NULL, NULL}			     /* sentinel */
-};
-
-static PyObject *
-md5_get_block_size(PyObject *self, void *closure)
-{
-    return PyInt_FromLong(64);
-}
-
-static PyObject *
-md5_get_digest_size(PyObject *self, void *closure)
-{
-    return PyInt_FromLong(16);
-}
-
-static PyObject *
-md5_get_name(PyObject *self, void *closure)
-{
-    return PyString_FromStringAndSize("MD5", 3);
-}
-
-static PyGetSetDef md5_getseters[] = {
-    {"digest_size",
-     (getter)md5_get_digest_size, NULL,
-     NULL,
-     NULL},
-    {"block_size",
-     (getter)md5_get_block_size, NULL,
-     NULL,
-     NULL},
-    {"name",
-     (getter)md5_get_name, NULL,
-     NULL,
-     NULL},
-    /* the old md5 and sha modules support 'digest_size' as in PEP 247.
-     * the old sha module also supported 'digestsize'.  ugh. */
-    {"digestsize",
-     (getter)md5_get_digest_size, NULL,
-     NULL,
-     NULL},
-    {NULL}  /* Sentinel */
-};
-
-
-PyDoc_STRVAR(module_doc,
-"This module implements the interface to RSA's MD5 message digest\n\
-algorithm (see also Internet RFC 1321). Its use is quite\n\
-straightforward: use the new() to create an md5 object. You can now\n\
-feed this object with arbitrary strings using the update() method, and\n\
-at any point you can ask it for the digest (a strong kind of 128-bit\n\
-checksum, a.k.a. ``fingerprint'') of the concatenation of the strings\n\
-fed to it so far using the digest() method.\n\
-\n\
-Functions:\n\
-\n\
-new([arg]) -- return a new md5 object, initialized with arg if provided\n\
-md5([arg]) -- DEPRECATED, same as new, but for compatibility\n\
-\n\
-Special Objects:\n\
-\n\
-MD5Type -- type object for md5 objects");
-
-PyDoc_STRVAR(md5type_doc,
-"An md5 represents the object used to calculate the MD5 checksum of a\n\
-string of information.\n\
-\n\
-Methods:\n\
-\n\
-update() -- updates the current digest with an additional string\n\
-digest() -- return the current digest value\n\
-hexdigest() -- return the current digest as a string of hexadecimal digits\n\
-copy() -- return a copy of the current md5 object");
-
-static PyTypeObject MD5type = {
-	PyObject_HEAD_INIT(NULL)
-	0,			  /*ob_size*/
-	"_md5.md5",		  /*tp_name*/
-	sizeof(md5object),	  /*tp_size*/
-	0,			  /*tp_itemsize*/
-	/* methods */
-	(destructor)md5_dealloc,  /*tp_dealloc*/
-	0,			  /*tp_print*/
-	0,                        /*tp_getattr*/
-	0,			  /*tp_setattr*/
-	0,			  /*tp_compare*/
-	0,			  /*tp_repr*/
-        0,			  /*tp_as_number*/
-	0,                        /*tp_as_sequence*/
-	0,			  /*tp_as_mapping*/
-	0, 			  /*tp_hash*/
-	0,			  /*tp_call*/
-	0,			  /*tp_str*/
-	0,			  /*tp_getattro*/
-	0,			  /*tp_setattro*/
-	0,	                  /*tp_as_buffer*/
-	Py_TPFLAGS_DEFAULT,	  /*tp_flags*/
-	md5type_doc,		  /*tp_doc*/
-        0,                        /*tp_traverse*/
-        0,			  /*tp_clear*/
-        0,			  /*tp_richcompare*/
-        0,			  /*tp_weaklistoffset*/
-        0,			  /*tp_iter*/
-        0,			  /*tp_iternext*/
-        md5_methods,	          /*tp_methods*/
-        0,      	          /*tp_members*/
-        md5_getseters,            /*tp_getset*/
-};
-
-
-/* MD5 functions */
-
-static PyObject *
-MD5_new(PyObject *self, PyObject *args)
-{
-	md5object *md5p;
-	unsigned char *cp = NULL;
-	int len = 0;
-
-	if (!PyArg_ParseTuple(args, "|s#:new", &cp, &len))
-		return NULL;
-
-	if ((md5p = newmd5object()) == NULL)
-		return NULL;
-
-	if (cp)
-		md5_append(&md5p->md5, cp, len);
-
-	return (PyObject *)md5p;
-}
-
-PyDoc_STRVAR(new_doc,
-"new([arg]) -> md5 object\n\
-\n\
-Return a new md5 object. If arg is present, the method call update(arg)\n\
-is made.");
-
-
-/* List of functions exported by this module */
-
-static PyMethodDef md5_functions[] = {
-	{"new",		(PyCFunction)MD5_new, METH_VARARGS, new_doc},
-	{NULL,		NULL}	/* Sentinel */
-};
-
-
-/* Initialize this module. */
-
-PyMODINIT_FUNC
-init_md5(void)
-{
-	PyObject *m, *d;
-
-        MD5type.ob_type = &PyType_Type;
-        if (PyType_Ready(&MD5type) < 0)
-            return;
-	m = Py_InitModule3("_md5", md5_functions, module_doc);
-	if (m == NULL)
-	    return;
-	d = PyModule_GetDict(m);
-	PyDict_SetItemString(d, "MD5Type", (PyObject *)&MD5type);
-	PyModule_AddIntConstant(m, "digest_size", 16);
-	/* No need to check the error here, the caller will do that */
-}
diff --git a/distutils2/_backport/sha256module.c b/distutils2/_backport/sha256module.c
deleted file mode 100644
--- a/distutils2/_backport/sha256module.c
+++ /dev/null
@@ -1,701 +0,0 @@
-/* SHA256 module */
-
-/* This module provides an interface to NIST's SHA-256 and SHA-224 Algorithms */
-
-/* See below for information about the original code this module was
-   based upon. Additional work performed by:
-
-   Andrew Kuchling (amk at amk.ca)
-   Greg Stein (gstein at lyra.org)
-   Trevor Perrin (trevp at trevp.net)
-
-   Copyright (C) 2005   Gregory P. Smith (greg at krypto.org)
-   Licensed to PSF under a Contributor Agreement.
-
-*/
-
-/* SHA objects */
-
-#include "Python.h"
-#include "structmember.h"
-
-
-/* Endianness testing and definitions */
-#define TestEndianness(variable) {int i=1; variable=PCT_BIG_ENDIAN;\
-	if (*((char*)&i)==1) variable=PCT_LITTLE_ENDIAN;}
-
-#define PCT_LITTLE_ENDIAN 1
-#define PCT_BIG_ENDIAN 0
-
-/* Some useful types */
-
-typedef unsigned char SHA_BYTE;
-
-#if SIZEOF_INT == 4
-typedef unsigned int SHA_INT32;	/* 32-bit integer */
-#else
-/* not defined. compilation will die. */
-#endif
-
-/* The SHA block size and message digest sizes, in bytes */
-
-#define SHA_BLOCKSIZE    64
-#define SHA_DIGESTSIZE  32
-
-/* The structure for storing SHA info */
-
-typedef struct {
-    PyObject_HEAD
-    SHA_INT32 digest[8];		/* Message digest */
-    SHA_INT32 count_lo, count_hi;	/* 64-bit bit count */
-    SHA_BYTE data[SHA_BLOCKSIZE];	/* SHA data buffer */
-    int Endianness;
-    int local;				/* unprocessed amount in data */
-    int digestsize;
-} SHAobject;
-
-/* When run on a little-endian CPU we need to perform byte reversal on an
-   array of longwords. */
-
-static void longReverse(SHA_INT32 *buffer, int byteCount, int Endianness)
-{
-    SHA_INT32 value;
-
-    if ( Endianness == PCT_BIG_ENDIAN )
-	return;
-
-    byteCount /= sizeof(*buffer);
-    while (byteCount--) {
-        value = *buffer;
-        value = ( ( value & 0xFF00FF00L ) >> 8  ) | \
-                ( ( value & 0x00FF00FFL ) << 8 );
-        *buffer++ = ( value << 16 ) | ( value >> 16 );
-    }
-}
-
-static void SHAcopy(SHAobject *src, SHAobject *dest)
-{
-    dest->Endianness = src->Endianness;
-    dest->local = src->local;
-    dest->digestsize = src->digestsize;
-    dest->count_lo = src->count_lo;
-    dest->count_hi = src->count_hi;
-    memcpy(dest->digest, src->digest, sizeof(src->digest));
-    memcpy(dest->data, src->data, sizeof(src->data));
-}
-
-
-/* ------------------------------------------------------------------------
- *
- * This code for the SHA-256 algorithm was noted as public domain. The
- * original headers are pasted below.
- *
- * Several changes have been made to make it more compatible with the
- * Python environment and desired interface.
- *
- */
-
-/* LibTomCrypt, modular cryptographic library -- Tom St Denis
- *
- * LibTomCrypt is a library that provides various cryptographic
- * algorithms in a highly modular and flexible manner.
- *
- * The library is free for all purposes without any express
- * gurantee it works.
- *
- * Tom St Denis, tomstdenis at iahu.ca, http://libtomcrypt.org
- */
-
-
-/* SHA256 by Tom St Denis */
-
-/* Various logical functions */
-#define ROR(x, y)\
-( ((((unsigned long)(x)&0xFFFFFFFFUL)>>(unsigned long)((y)&31)) | \
-((unsigned long)(x)<<(unsigned long)(32-((y)&31)))) & 0xFFFFFFFFUL)
-#define Ch(x,y,z)       (z ^ (x & (y ^ z)))
-#define Maj(x,y,z)      (((x | y) & z) | (x & y)) 
-#define S(x, n)         ROR((x),(n))
-#define R(x, n)         (((x)&0xFFFFFFFFUL)>>(n))
-#define Sigma0(x)       (S(x, 2) ^ S(x, 13) ^ S(x, 22))
-#define Sigma1(x)       (S(x, 6) ^ S(x, 11) ^ S(x, 25))
-#define Gamma0(x)       (S(x, 7) ^ S(x, 18) ^ R(x, 3))
-#define Gamma1(x)       (S(x, 17) ^ S(x, 19) ^ R(x, 10))
-
-
-static void
-sha_transform(SHAobject *sha_info)
-{
-    int i;
-	SHA_INT32 S[8], W[64], t0, t1;
-
-    memcpy(W, sha_info->data, sizeof(sha_info->data));
-    longReverse(W, (int)sizeof(sha_info->data), sha_info->Endianness);
-
-    for (i = 16; i < 64; ++i) {
-		W[i] = Gamma1(W[i - 2]) + W[i - 7] + Gamma0(W[i - 15]) + W[i - 16];
-    }
-    for (i = 0; i < 8; ++i) {
-        S[i] = sha_info->digest[i];
-    }
-
-    /* Compress */
-#define RND(a,b,c,d,e,f,g,h,i,ki)                    \
-     t0 = h + Sigma1(e) + Ch(e, f, g) + ki + W[i];   \
-     t1 = Sigma0(a) + Maj(a, b, c);                  \
-     d += t0;                                        \
-     h  = t0 + t1;
-
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],0,0x428a2f98);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],1,0x71374491);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],2,0xb5c0fbcf);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],3,0xe9b5dba5);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],4,0x3956c25b);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],5,0x59f111f1);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],6,0x923f82a4);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],7,0xab1c5ed5);
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],8,0xd807aa98);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],9,0x12835b01);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],10,0x243185be);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],11,0x550c7dc3);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],12,0x72be5d74);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],13,0x80deb1fe);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],14,0x9bdc06a7);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],15,0xc19bf174);
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],16,0xe49b69c1);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],17,0xefbe4786);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],18,0x0fc19dc6);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],19,0x240ca1cc);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],20,0x2de92c6f);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],21,0x4a7484aa);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],22,0x5cb0a9dc);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],23,0x76f988da);
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],24,0x983e5152);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],25,0xa831c66d);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],26,0xb00327c8);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],27,0xbf597fc7);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],28,0xc6e00bf3);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],29,0xd5a79147);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],30,0x06ca6351);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],31,0x14292967);
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],32,0x27b70a85);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],33,0x2e1b2138);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],34,0x4d2c6dfc);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],35,0x53380d13);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],36,0x650a7354);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],37,0x766a0abb);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],38,0x81c2c92e);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],39,0x92722c85);
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],40,0xa2bfe8a1);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],41,0xa81a664b);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],42,0xc24b8b70);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],43,0xc76c51a3);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],44,0xd192e819);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],45,0xd6990624);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],46,0xf40e3585);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],47,0x106aa070);
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],48,0x19a4c116);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],49,0x1e376c08);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],50,0x2748774c);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],51,0x34b0bcb5);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],52,0x391c0cb3);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],53,0x4ed8aa4a);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],54,0x5b9cca4f);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],55,0x682e6ff3);
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],56,0x748f82ee);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],57,0x78a5636f);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],58,0x84c87814);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],59,0x8cc70208);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],60,0x90befffa);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],61,0xa4506ceb);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],62,0xbef9a3f7);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],63,0xc67178f2);
-
-#undef RND     
-    
-    /* feedback */
-    for (i = 0; i < 8; i++) {
-        sha_info->digest[i] = sha_info->digest[i] + S[i];
-    }
-
-}
-
-
-
-/* initialize the SHA digest */
-
-static void
-sha_init(SHAobject *sha_info)
-{
-    TestEndianness(sha_info->Endianness)
-    sha_info->digest[0] = 0x6A09E667L;
-    sha_info->digest[1] = 0xBB67AE85L;
-    sha_info->digest[2] = 0x3C6EF372L;
-    sha_info->digest[3] = 0xA54FF53AL;
-    sha_info->digest[4] = 0x510E527FL;
-    sha_info->digest[5] = 0x9B05688CL;
-    sha_info->digest[6] = 0x1F83D9ABL;
-    sha_info->digest[7] = 0x5BE0CD19L;
-    sha_info->count_lo = 0L;
-    sha_info->count_hi = 0L;
-    sha_info->local = 0;
-    sha_info->digestsize = 32;
-}
-
-static void
-sha224_init(SHAobject *sha_info)
-{
-    TestEndianness(sha_info->Endianness)
-    sha_info->digest[0] = 0xc1059ed8L;
-    sha_info->digest[1] = 0x367cd507L;
-    sha_info->digest[2] = 0x3070dd17L;
-    sha_info->digest[3] = 0xf70e5939L;
-    sha_info->digest[4] = 0xffc00b31L;
-    sha_info->digest[5] = 0x68581511L;
-    sha_info->digest[6] = 0x64f98fa7L;
-    sha_info->digest[7] = 0xbefa4fa4L;
-    sha_info->count_lo = 0L;
-    sha_info->count_hi = 0L;
-    sha_info->local = 0;
-    sha_info->digestsize = 28;
-}
-
-
-/* update the SHA digest */
-
-static void
-sha_update(SHAobject *sha_info, SHA_BYTE *buffer, int count)
-{
-    int i;
-    SHA_INT32 clo;
-
-    clo = sha_info->count_lo + ((SHA_INT32) count << 3);
-    if (clo < sha_info->count_lo) {
-        ++sha_info->count_hi;
-    }
-    sha_info->count_lo = clo;
-    sha_info->count_hi += (SHA_INT32) count >> 29;
-    if (sha_info->local) {
-        i = SHA_BLOCKSIZE - sha_info->local;
-        if (i > count) {
-            i = count;
-        }
-        memcpy(((SHA_BYTE *) sha_info->data) + sha_info->local, buffer, i);
-        count -= i;
-        buffer += i;
-        sha_info->local += i;
-        if (sha_info->local == SHA_BLOCKSIZE) {
-            sha_transform(sha_info);
-        }
-        else {
-            return;
-        }
-    }
-    while (count >= SHA_BLOCKSIZE) {
-        memcpy(sha_info->data, buffer, SHA_BLOCKSIZE);
-        buffer += SHA_BLOCKSIZE;
-        count -= SHA_BLOCKSIZE;
-        sha_transform(sha_info);
-    }
-    memcpy(sha_info->data, buffer, count);
-    sha_info->local = count;
-}
-
-/* finish computing the SHA digest */
-
-static void
-sha_final(unsigned char digest[SHA_DIGESTSIZE], SHAobject *sha_info)
-{
-    int count;
-    SHA_INT32 lo_bit_count, hi_bit_count;
-
-    lo_bit_count = sha_info->count_lo;
-    hi_bit_count = sha_info->count_hi;
-    count = (int) ((lo_bit_count >> 3) & 0x3f);
-    ((SHA_BYTE *) sha_info->data)[count++] = 0x80;
-    if (count > SHA_BLOCKSIZE - 8) {
-	memset(((SHA_BYTE *) sha_info->data) + count, 0,
-	       SHA_BLOCKSIZE - count);
-	sha_transform(sha_info);
-	memset((SHA_BYTE *) sha_info->data, 0, SHA_BLOCKSIZE - 8);
-    }
-    else {
-	memset(((SHA_BYTE *) sha_info->data) + count, 0,
-	       SHA_BLOCKSIZE - 8 - count);
-    }
-
-    /* GJS: note that we add the hi/lo in big-endian. sha_transform will
-       swap these values into host-order. */
-    sha_info->data[56] = (hi_bit_count >> 24) & 0xff;
-    sha_info->data[57] = (hi_bit_count >> 16) & 0xff;
-    sha_info->data[58] = (hi_bit_count >>  8) & 0xff;
-    sha_info->data[59] = (hi_bit_count >>  0) & 0xff;
-    sha_info->data[60] = (lo_bit_count >> 24) & 0xff;
-    sha_info->data[61] = (lo_bit_count >> 16) & 0xff;
-    sha_info->data[62] = (lo_bit_count >>  8) & 0xff;
-    sha_info->data[63] = (lo_bit_count >>  0) & 0xff;
-    sha_transform(sha_info);
-    digest[ 0] = (unsigned char) ((sha_info->digest[0] >> 24) & 0xff);
-    digest[ 1] = (unsigned char) ((sha_info->digest[0] >> 16) & 0xff);
-    digest[ 2] = (unsigned char) ((sha_info->digest[0] >>  8) & 0xff);
-    digest[ 3] = (unsigned char) ((sha_info->digest[0]      ) & 0xff);
-    digest[ 4] = (unsigned char) ((sha_info->digest[1] >> 24) & 0xff);
-    digest[ 5] = (unsigned char) ((sha_info->digest[1] >> 16) & 0xff);
-    digest[ 6] = (unsigned char) ((sha_info->digest[1] >>  8) & 0xff);
-    digest[ 7] = (unsigned char) ((sha_info->digest[1]      ) & 0xff);
-    digest[ 8] = (unsigned char) ((sha_info->digest[2] >> 24) & 0xff);
-    digest[ 9] = (unsigned char) ((sha_info->digest[2] >> 16) & 0xff);
-    digest[10] = (unsigned char) ((sha_info->digest[2] >>  8) & 0xff);
-    digest[11] = (unsigned char) ((sha_info->digest[2]      ) & 0xff);
-    digest[12] = (unsigned char) ((sha_info->digest[3] >> 24) & 0xff);
-    digest[13] = (unsigned char) ((sha_info->digest[3] >> 16) & 0xff);
-    digest[14] = (unsigned char) ((sha_info->digest[3] >>  8) & 0xff);
-    digest[15] = (unsigned char) ((sha_info->digest[3]      ) & 0xff);
-    digest[16] = (unsigned char) ((sha_info->digest[4] >> 24) & 0xff);
-    digest[17] = (unsigned char) ((sha_info->digest[4] >> 16) & 0xff);
-    digest[18] = (unsigned char) ((sha_info->digest[4] >>  8) & 0xff);
-    digest[19] = (unsigned char) ((sha_info->digest[4]      ) & 0xff);
-    digest[20] = (unsigned char) ((sha_info->digest[5] >> 24) & 0xff);
-    digest[21] = (unsigned char) ((sha_info->digest[5] >> 16) & 0xff);
-    digest[22] = (unsigned char) ((sha_info->digest[5] >>  8) & 0xff);
-    digest[23] = (unsigned char) ((sha_info->digest[5]      ) & 0xff);
-    digest[24] = (unsigned char) ((sha_info->digest[6] >> 24) & 0xff);
-    digest[25] = (unsigned char) ((sha_info->digest[6] >> 16) & 0xff);
-    digest[26] = (unsigned char) ((sha_info->digest[6] >>  8) & 0xff);
-    digest[27] = (unsigned char) ((sha_info->digest[6]      ) & 0xff);
-    digest[28] = (unsigned char) ((sha_info->digest[7] >> 24) & 0xff);
-    digest[29] = (unsigned char) ((sha_info->digest[7] >> 16) & 0xff);
-    digest[30] = (unsigned char) ((sha_info->digest[7] >>  8) & 0xff);
-    digest[31] = (unsigned char) ((sha_info->digest[7]      ) & 0xff);
-}
-
-/*
- * End of copied SHA code.
- *
- * ------------------------------------------------------------------------
- */
-
-static PyTypeObject SHA224type;
-static PyTypeObject SHA256type;
-
-
-static SHAobject *
-newSHA224object(void)
-{
-    return (SHAobject *)PyObject_New(SHAobject, &SHA224type);
-}
-
-static SHAobject *
-newSHA256object(void)
-{
-    return (SHAobject *)PyObject_New(SHAobject, &SHA256type);
-}
-
-/* Internal methods for a hash object */
-
-static void
-SHA_dealloc(PyObject *ptr)
-{
-    PyObject_Del(ptr);
-}
-
-
-/* External methods for a hash object */
-
-PyDoc_STRVAR(SHA256_copy__doc__, "Return a copy of the hash object.");
-
-static PyObject *
-SHA256_copy(SHAobject *self, PyObject *unused)
-{
-    SHAobject *newobj;
-
-    if (((PyObject*)self)->ob_type == &SHA256type) {
-        if ( (newobj = newSHA256object())==NULL)
-            return NULL;
-    } else {
-        if ( (newobj = newSHA224object())==NULL)
-            return NULL;
-    }
-
-    SHAcopy(self, newobj);
-    return (PyObject *)newobj;
-}
-
-PyDoc_STRVAR(SHA256_digest__doc__,
-"Return the digest value as a string of binary data.");
-
-static PyObject *
-SHA256_digest(SHAobject *self, PyObject *unused)
-{
-    unsigned char digest[SHA_DIGESTSIZE];
-    SHAobject temp;
-
-    SHAcopy(self, &temp);
-    sha_final(digest, &temp);
-    return PyString_FromStringAndSize((const char *)digest, self->digestsize);
-}
-
-PyDoc_STRVAR(SHA256_hexdigest__doc__,
-"Return the digest value as a string of hexadecimal digits.");
-
-static PyObject *
-SHA256_hexdigest(SHAobject *self, PyObject *unused)
-{
-    unsigned char digest[SHA_DIGESTSIZE];
-    SHAobject temp;
-    PyObject *retval;
-    char *hex_digest;
-    int i, j;
-
-    /* Get the raw (binary) digest value */
-    SHAcopy(self, &temp);
-    sha_final(digest, &temp);
-
-    /* Create a new string */
-    retval = PyString_FromStringAndSize(NULL, self->digestsize * 2);
-    if (!retval)
-	    return NULL;
-    hex_digest = PyString_AsString(retval);
-    if (!hex_digest) {
-	    Py_DECREF(retval);
-	    return NULL;
-    }
-
-    /* Make hex version of the digest */
-    for(i=j=0; i<self->digestsize; i++) {
-        char c;
-        c = (digest[i] >> 4) & 0xf;
-	c = (c>9) ? c+'a'-10 : c + '0';
-        hex_digest[j++] = c;
-        c = (digest[i] & 0xf);
-	c = (c>9) ? c+'a'-10 : c + '0';
-        hex_digest[j++] = c;
-    }
-    return retval;
-}
-
-PyDoc_STRVAR(SHA256_update__doc__,
-"Update this hash object's state with the provided string.");
-
-static PyObject *
-SHA256_update(SHAobject *self, PyObject *args)
-{
-    unsigned char *cp;
-    int len;
-
-    if (!PyArg_ParseTuple(args, "s#:update", &cp, &len))
-        return NULL;
-
-    sha_update(self, cp, len);
-
-    Py_INCREF(Py_None);
-    return Py_None;
-}
-
-static PyMethodDef SHA_methods[] = {
-    {"copy",	  (PyCFunction)SHA256_copy,      METH_NOARGS,  SHA256_copy__doc__},
-    {"digest",	  (PyCFunction)SHA256_digest,    METH_NOARGS,  SHA256_digest__doc__},
-    {"hexdigest", (PyCFunction)SHA256_hexdigest, METH_NOARGS,  SHA256_hexdigest__doc__},
-    {"update",	  (PyCFunction)SHA256_update,    METH_VARARGS, SHA256_update__doc__},
-    {NULL,	  NULL}		/* sentinel */
-};
-
-static PyObject *
-SHA256_get_block_size(PyObject *self, void *closure)
-{
-    return PyInt_FromLong(SHA_BLOCKSIZE);
-}
-
-static PyObject *
-SHA256_get_name(PyObject *self, void *closure)
-{
-    if (((SHAobject *)self)->digestsize == 32)
-        return PyString_FromStringAndSize("SHA256", 6);
-    else
-        return PyString_FromStringAndSize("SHA224", 6);
-}
-
-static PyGetSetDef SHA_getseters[] = {
-    {"block_size",
-     (getter)SHA256_get_block_size, NULL,
-     NULL,
-     NULL},
-    {"name",
-     (getter)SHA256_get_name, NULL,
-     NULL,
-     NULL},
-    {NULL}  /* Sentinel */
-};
-
-static PyMemberDef SHA_members[] = {
-    {"digest_size", T_INT, offsetof(SHAobject, digestsize), READONLY, NULL},
-    /* the old md5 and sha modules support 'digest_size' as in PEP 247.
-     * the old sha module also supported 'digestsize'.  ugh. */
-    {"digestsize", T_INT, offsetof(SHAobject, digestsize), READONLY, NULL},
-    {NULL}  /* Sentinel */
-};
-
-static PyTypeObject SHA224type = {
-    PyObject_HEAD_INIT(NULL)
-    0,			/*ob_size*/
-    "_sha256.sha224",	/*tp_name*/
-    sizeof(SHAobject),	/*tp_size*/
-    0,			/*tp_itemsize*/
-    /* methods */
-    SHA_dealloc,	/*tp_dealloc*/
-    0,			/*tp_print*/
-    0,          	/*tp_getattr*/
-    0,                  /*tp_setattr*/
-    0,                  /*tp_compare*/
-    0,                  /*tp_repr*/
-    0,                  /*tp_as_number*/
-    0,                  /*tp_as_sequence*/
-    0,                  /*tp_as_mapping*/
-    0,                  /*tp_hash*/
-    0,                  /*tp_call*/
-    0,                  /*tp_str*/
-    0,                  /*tp_getattro*/
-    0,                  /*tp_setattro*/
-    0,                  /*tp_as_buffer*/
-    Py_TPFLAGS_DEFAULT, /*tp_flags*/
-    0,                  /*tp_doc*/
-    0,                  /*tp_traverse*/
-    0,			/*tp_clear*/
-    0,			/*tp_richcompare*/
-    0,			/*tp_weaklistoffset*/
-    0,			/*tp_iter*/
-    0,			/*tp_iternext*/
-    SHA_methods,	/* tp_methods */
-    SHA_members,	/* tp_members */
-    SHA_getseters,      /* tp_getset */
-};
-
-static PyTypeObject SHA256type = {
-    PyObject_HEAD_INIT(NULL)
-    0,			/*ob_size*/
-    "_sha256.sha256",	/*tp_name*/
-    sizeof(SHAobject),	/*tp_size*/
-    0,			/*tp_itemsize*/
-    /* methods */
-    SHA_dealloc,	/*tp_dealloc*/
-    0,			/*tp_print*/
-    0,          	/*tp_getattr*/
-    0,                  /*tp_setattr*/
-    0,                  /*tp_compare*/
-    0,                  /*tp_repr*/
-    0,                  /*tp_as_number*/
-    0,                  /*tp_as_sequence*/
-    0,                  /*tp_as_mapping*/
-    0,                  /*tp_hash*/
-    0,                  /*tp_call*/
-    0,                  /*tp_str*/
-    0,                  /*tp_getattro*/
-    0,                  /*tp_setattro*/
-    0,                  /*tp_as_buffer*/
-    Py_TPFLAGS_DEFAULT, /*tp_flags*/
-    0,                  /*tp_doc*/
-    0,                  /*tp_traverse*/
-    0,			/*tp_clear*/
-    0,			/*tp_richcompare*/
-    0,			/*tp_weaklistoffset*/
-    0,			/*tp_iter*/
-    0,			/*tp_iternext*/
-    SHA_methods,	/* tp_methods */
-    SHA_members,	/* tp_members */
-    SHA_getseters,      /* tp_getset */
-};
-
-
-/* The single module-level function: new() */
-
-PyDoc_STRVAR(SHA256_new__doc__,
-"Return a new SHA-256 hash object; optionally initialized with a string.");
-
-static PyObject *
-SHA256_new(PyObject *self, PyObject *args, PyObject *kwdict)
-{
-    static char *kwlist[] = {"string", NULL};
-    SHAobject *new;
-    unsigned char *cp = NULL;
-    int len;
-
-    if (!PyArg_ParseTupleAndKeywords(args, kwdict, "|s#:new", kwlist,
-                                     &cp, &len)) {
-        return NULL;
-    }
-
-    if ((new = newSHA256object()) == NULL)
-        return NULL;
-
-    sha_init(new);
-
-    if (PyErr_Occurred()) {
-        Py_DECREF(new);
-        return NULL;
-    }
-    if (cp)
-        sha_update(new, cp, len);
-
-    return (PyObject *)new;
-}
-
-PyDoc_STRVAR(SHA224_new__doc__,
-"Return a new SHA-224 hash object; optionally initialized with a string.");
-
-static PyObject *
-SHA224_new(PyObject *self, PyObject *args, PyObject *kwdict)
-{
-    static char *kwlist[] = {"string", NULL};
-    SHAobject *new;
-    unsigned char *cp = NULL;
-    int len;
-
-    if (!PyArg_ParseTupleAndKeywords(args, kwdict, "|s#:new", kwlist,
-                                     &cp, &len)) {
-        return NULL;
-    }
-
-    if ((new = newSHA224object()) == NULL)
-        return NULL;
-
-    sha224_init(new);
-
-    if (PyErr_Occurred()) {
-        Py_DECREF(new);
-        return NULL;
-    }
-    if (cp)
-        sha_update(new, cp, len);
-
-    return (PyObject *)new;
-}
-
-
-/* List of functions exported by this module */
-
-static struct PyMethodDef SHA_functions[] = {
-    {"sha256", (PyCFunction)SHA256_new, METH_VARARGS|METH_KEYWORDS, SHA256_new__doc__},
-    {"sha224", (PyCFunction)SHA224_new, METH_VARARGS|METH_KEYWORDS, SHA224_new__doc__},
-    {NULL,	NULL}		 /* Sentinel */
-};
-
-
-/* Initialize this module. */
-
-#define insint(n,v) { PyModule_AddIntConstant(m,n,v); }
-
-PyMODINIT_FUNC
-init_sha256(void)
-{
-    PyObject *m;
-
-    SHA224type.ob_type = &PyType_Type;
-    if (PyType_Ready(&SHA224type) < 0)
-        return;
-    SHA256type.ob_type = &PyType_Type;
-    if (PyType_Ready(&SHA256type) < 0)
-        return;
-    m = Py_InitModule("_sha256", SHA_functions);
-    if (m == NULL)
-	return;
-}
diff --git a/distutils2/_backport/sha512module.c b/distutils2/_backport/sha512module.c
deleted file mode 100644
--- a/distutils2/_backport/sha512module.c
+++ /dev/null
@@ -1,769 +0,0 @@
-/* SHA512 module */
-
-/* This module provides an interface to NIST's SHA-512 and SHA-384 Algorithms */
-
-/* See below for information about the original code this module was
-   based upon. Additional work performed by:
-
-   Andrew Kuchling (amk at amk.ca)
-   Greg Stein (gstein at lyra.org)
-   Trevor Perrin (trevp at trevp.net)
-
-   Copyright (C) 2005   Gregory P. Smith (greg at krypto.org)
-   Licensed to PSF under a Contributor Agreement.
-
-*/
-
-/* SHA objects */
-
-#include "Python.h"
-#include "structmember.h"
-
-#ifdef PY_LONG_LONG /* If no PY_LONG_LONG, don't compile anything! */
-
-/* Endianness testing and definitions */
-#define TestEndianness(variable) {int i=1; variable=PCT_BIG_ENDIAN;\
-	if (*((char*)&i)==1) variable=PCT_LITTLE_ENDIAN;}
-
-#define PCT_LITTLE_ENDIAN 1
-#define PCT_BIG_ENDIAN 0
-
-/* Some useful types */
-
-typedef unsigned char SHA_BYTE;
-
-#if SIZEOF_INT == 4
-typedef unsigned int SHA_INT32;	/* 32-bit integer */
-typedef unsigned PY_LONG_LONG SHA_INT64;	/* 64-bit integer */
-#else
-/* not defined. compilation will die. */
-#endif
-
-/* The SHA block size and message digest sizes, in bytes */
-
-#define SHA_BLOCKSIZE   128
-#define SHA_DIGESTSIZE  64
-
-/* The structure for storing SHA info */
-
-typedef struct {
-    PyObject_HEAD
-    SHA_INT64 digest[8];		/* Message digest */
-    SHA_INT32 count_lo, count_hi;	/* 64-bit bit count */
-    SHA_BYTE data[SHA_BLOCKSIZE];	/* SHA data buffer */
-    int Endianness;
-    int local;				/* unprocessed amount in data */
-    int digestsize;
-} SHAobject;
-
-/* When run on a little-endian CPU we need to perform byte reversal on an
-   array of longwords. */
-
-static void longReverse(SHA_INT64 *buffer, int byteCount, int Endianness)
-{
-    SHA_INT64 value;
-
-    if ( Endianness == PCT_BIG_ENDIAN )
-	return;
-
-    byteCount /= sizeof(*buffer);
-    while (byteCount--) {
-        value = *buffer;
-
-		((unsigned char*)buffer)[0] = (unsigned char)(value >> 56) & 0xff;
-		((unsigned char*)buffer)[1] = (unsigned char)(value >> 48) & 0xff;
-		((unsigned char*)buffer)[2] = (unsigned char)(value >> 40) & 0xff;
-		((unsigned char*)buffer)[3] = (unsigned char)(value >> 32) & 0xff;
-		((unsigned char*)buffer)[4] = (unsigned char)(value >> 24) & 0xff;
-		((unsigned char*)buffer)[5] = (unsigned char)(value >> 16) & 0xff;
-		((unsigned char*)buffer)[6] = (unsigned char)(value >>  8) & 0xff;
-		((unsigned char*)buffer)[7] = (unsigned char)(value      ) & 0xff;
-        
-		buffer++;
-    }
-}
-
-static void SHAcopy(SHAobject *src, SHAobject *dest)
-{
-    dest->Endianness = src->Endianness;
-    dest->local = src->local;
-    dest->digestsize = src->digestsize;
-    dest->count_lo = src->count_lo;
-    dest->count_hi = src->count_hi;
-    memcpy(dest->digest, src->digest, sizeof(src->digest));
-    memcpy(dest->data, src->data, sizeof(src->data));
-}
-
-
-/* ------------------------------------------------------------------------
- *
- * This code for the SHA-512 algorithm was noted as public domain. The
- * original headers are pasted below.
- *
- * Several changes have been made to make it more compatible with the
- * Python environment and desired interface.
- *
- */
-
-/* LibTomCrypt, modular cryptographic library -- Tom St Denis
- *
- * LibTomCrypt is a library that provides various cryptographic
- * algorithms in a highly modular and flexible manner.
- *
- * The library is free for all purposes without any express
- * gurantee it works.
- *
- * Tom St Denis, tomstdenis at iahu.ca, http://libtomcrypt.org
- */
-
-
-/* SHA512 by Tom St Denis */
-
-/* Various logical functions */
-#define ROR64(x, y) \
-    ( ((((x) & 0xFFFFFFFFFFFFFFFFULL)>>((unsigned PY_LONG_LONG)(y) & 63)) | \
-      ((x)<<((unsigned PY_LONG_LONG)(64-((y) & 63))))) & 0xFFFFFFFFFFFFFFFFULL)
-#define Ch(x,y,z)       (z ^ (x & (y ^ z)))
-#define Maj(x,y,z)      (((x | y) & z) | (x & y)) 
-#define S(x, n)         ROR64((x),(n))
-#define R(x, n)         (((x) & 0xFFFFFFFFFFFFFFFFULL) >> ((unsigned PY_LONG_LONG)n))
-#define Sigma0(x)       (S(x, 28) ^ S(x, 34) ^ S(x, 39))
-#define Sigma1(x)       (S(x, 14) ^ S(x, 18) ^ S(x, 41))
-#define Gamma0(x)       (S(x, 1) ^ S(x, 8) ^ R(x, 7))
-#define Gamma1(x)       (S(x, 19) ^ S(x, 61) ^ R(x, 6))
-
-
-static void
-sha512_transform(SHAobject *sha_info)
-{
-    int i;
-    SHA_INT64 S[8], W[80], t0, t1;
-
-    memcpy(W, sha_info->data, sizeof(sha_info->data));
-    longReverse(W, (int)sizeof(sha_info->data), sha_info->Endianness);
-
-    for (i = 16; i < 80; ++i) {
-		W[i] = Gamma1(W[i - 2]) + W[i - 7] + Gamma0(W[i - 15]) + W[i - 16];
-    }
-    for (i = 0; i < 8; ++i) {
-        S[i] = sha_info->digest[i];
-    }
-
-    /* Compress */
-#define RND(a,b,c,d,e,f,g,h,i,ki)                    \
-     t0 = h + Sigma1(e) + Ch(e, f, g) + ki + W[i];   \
-     t1 = Sigma0(a) + Maj(a, b, c);                  \
-     d += t0;                                        \
-     h  = t0 + t1;
-
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],0,0x428a2f98d728ae22ULL);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],1,0x7137449123ef65cdULL);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],2,0xb5c0fbcfec4d3b2fULL);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],3,0xe9b5dba58189dbbcULL);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],4,0x3956c25bf348b538ULL);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],5,0x59f111f1b605d019ULL);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],6,0x923f82a4af194f9bULL);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],7,0xab1c5ed5da6d8118ULL);
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],8,0xd807aa98a3030242ULL);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],9,0x12835b0145706fbeULL);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],10,0x243185be4ee4b28cULL);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],11,0x550c7dc3d5ffb4e2ULL);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],12,0x72be5d74f27b896fULL);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],13,0x80deb1fe3b1696b1ULL);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],14,0x9bdc06a725c71235ULL);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],15,0xc19bf174cf692694ULL);
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],16,0xe49b69c19ef14ad2ULL);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],17,0xefbe4786384f25e3ULL);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],18,0x0fc19dc68b8cd5b5ULL);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],19,0x240ca1cc77ac9c65ULL);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],20,0x2de92c6f592b0275ULL);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],21,0x4a7484aa6ea6e483ULL);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],22,0x5cb0a9dcbd41fbd4ULL);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],23,0x76f988da831153b5ULL);
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],24,0x983e5152ee66dfabULL);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],25,0xa831c66d2db43210ULL);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],26,0xb00327c898fb213fULL);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],27,0xbf597fc7beef0ee4ULL);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],28,0xc6e00bf33da88fc2ULL);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],29,0xd5a79147930aa725ULL);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],30,0x06ca6351e003826fULL);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],31,0x142929670a0e6e70ULL);
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],32,0x27b70a8546d22ffcULL);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],33,0x2e1b21385c26c926ULL);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],34,0x4d2c6dfc5ac42aedULL);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],35,0x53380d139d95b3dfULL);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],36,0x650a73548baf63deULL);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],37,0x766a0abb3c77b2a8ULL);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],38,0x81c2c92e47edaee6ULL);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],39,0x92722c851482353bULL);
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],40,0xa2bfe8a14cf10364ULL);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],41,0xa81a664bbc423001ULL);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],42,0xc24b8b70d0f89791ULL);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],43,0xc76c51a30654be30ULL);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],44,0xd192e819d6ef5218ULL);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],45,0xd69906245565a910ULL);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],46,0xf40e35855771202aULL);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],47,0x106aa07032bbd1b8ULL);
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],48,0x19a4c116b8d2d0c8ULL);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],49,0x1e376c085141ab53ULL);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],50,0x2748774cdf8eeb99ULL);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],51,0x34b0bcb5e19b48a8ULL);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],52,0x391c0cb3c5c95a63ULL);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],53,0x4ed8aa4ae3418acbULL);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],54,0x5b9cca4f7763e373ULL);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],55,0x682e6ff3d6b2b8a3ULL);
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],56,0x748f82ee5defb2fcULL);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],57,0x78a5636f43172f60ULL);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],58,0x84c87814a1f0ab72ULL);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],59,0x8cc702081a6439ecULL);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],60,0x90befffa23631e28ULL);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],61,0xa4506cebde82bde9ULL);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],62,0xbef9a3f7b2c67915ULL);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],63,0xc67178f2e372532bULL);
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],64,0xca273eceea26619cULL);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],65,0xd186b8c721c0c207ULL);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],66,0xeada7dd6cde0eb1eULL);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],67,0xf57d4f7fee6ed178ULL);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],68,0x06f067aa72176fbaULL);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],69,0x0a637dc5a2c898a6ULL);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],70,0x113f9804bef90daeULL);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],71,0x1b710b35131c471bULL);
-    RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],72,0x28db77f523047d84ULL);
-    RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],73,0x32caab7b40c72493ULL);
-    RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],74,0x3c9ebe0a15c9bebcULL);
-    RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],75,0x431d67c49c100d4cULL);
-    RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],76,0x4cc5d4becb3e42b6ULL);
-    RND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],77,0x597f299cfc657e2aULL);
-    RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],78,0x5fcb6fab3ad6faecULL);
-    RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],79,0x6c44198c4a475817ULL);
-
-#undef RND     
-    
-    /* feedback */
-    for (i = 0; i < 8; i++) {
-        sha_info->digest[i] = sha_info->digest[i] + S[i];
-    }
-
-}
-
-
-
-/* initialize the SHA digest */
-
-static void
-sha512_init(SHAobject *sha_info)
-{
-    TestEndianness(sha_info->Endianness)
-    sha_info->digest[0] = 0x6a09e667f3bcc908ULL;
-    sha_info->digest[1] = 0xbb67ae8584caa73bULL;
-    sha_info->digest[2] = 0x3c6ef372fe94f82bULL;
-    sha_info->digest[3] = 0xa54ff53a5f1d36f1ULL;
-    sha_info->digest[4] = 0x510e527fade682d1ULL;
-    sha_info->digest[5] = 0x9b05688c2b3e6c1fULL;
-    sha_info->digest[6] = 0x1f83d9abfb41bd6bULL;
-    sha_info->digest[7] = 0x5be0cd19137e2179ULL;
-    sha_info->count_lo = 0L;
-    sha_info->count_hi = 0L;
-    sha_info->local = 0;
-    sha_info->digestsize = 64;
-}
-
-static void
-sha384_init(SHAobject *sha_info)
-{
-    TestEndianness(sha_info->Endianness)
-    sha_info->digest[0] = 0xcbbb9d5dc1059ed8ULL;
-    sha_info->digest[1] = 0x629a292a367cd507ULL;
-    sha_info->digest[2] = 0x9159015a3070dd17ULL;
-    sha_info->digest[3] = 0x152fecd8f70e5939ULL;
-    sha_info->digest[4] = 0x67332667ffc00b31ULL;
-    sha_info->digest[5] = 0x8eb44a8768581511ULL;
-    sha_info->digest[6] = 0xdb0c2e0d64f98fa7ULL;
-    sha_info->digest[7] = 0x47b5481dbefa4fa4ULL;
-    sha_info->count_lo = 0L;
-    sha_info->count_hi = 0L;
-    sha_info->local = 0;
-    sha_info->digestsize = 48;
-}
-
-
-/* update the SHA digest */
-
-static void
-sha512_update(SHAobject *sha_info, SHA_BYTE *buffer, int count)
-{
-    int i;
-    SHA_INT32 clo;
-
-    clo = sha_info->count_lo + ((SHA_INT32) count << 3);
-    if (clo < sha_info->count_lo) {
-        ++sha_info->count_hi;
-    }
-    sha_info->count_lo = clo;
-    sha_info->count_hi += (SHA_INT32) count >> 29;
-    if (sha_info->local) {
-        i = SHA_BLOCKSIZE - sha_info->local;
-        if (i > count) {
-            i = count;
-        }
-        memcpy(((SHA_BYTE *) sha_info->data) + sha_info->local, buffer, i);
-        count -= i;
-        buffer += i;
-        sha_info->local += i;
-        if (sha_info->local == SHA_BLOCKSIZE) {
-            sha512_transform(sha_info);
-        }
-        else {
-            return;
-        }
-    }
-    while (count >= SHA_BLOCKSIZE) {
-        memcpy(sha_info->data, buffer, SHA_BLOCKSIZE);
-        buffer += SHA_BLOCKSIZE;
-        count -= SHA_BLOCKSIZE;
-        sha512_transform(sha_info);
-    }
-    memcpy(sha_info->data, buffer, count);
-    sha_info->local = count;
-}
-
-/* finish computing the SHA digest */
-
-static void
-sha512_final(unsigned char digest[SHA_DIGESTSIZE], SHAobject *sha_info)
-{
-    int count;
-    SHA_INT32 lo_bit_count, hi_bit_count;
-
-    lo_bit_count = sha_info->count_lo;
-    hi_bit_count = sha_info->count_hi;
-    count = (int) ((lo_bit_count >> 3) & 0x7f);
-    ((SHA_BYTE *) sha_info->data)[count++] = 0x80;
-    if (count > SHA_BLOCKSIZE - 16) {
-	memset(((SHA_BYTE *) sha_info->data) + count, 0,
-	       SHA_BLOCKSIZE - count);
-	sha512_transform(sha_info);
-	memset((SHA_BYTE *) sha_info->data, 0, SHA_BLOCKSIZE - 16);
-    }
-    else {
-	memset(((SHA_BYTE *) sha_info->data) + count, 0,
-	       SHA_BLOCKSIZE - 16 - count);
-    }
-
-    /* GJS: note that we add the hi/lo in big-endian. sha512_transform will
-       swap these values into host-order. */
-    sha_info->data[112] = 0;
-    sha_info->data[113] = 0;
-    sha_info->data[114] = 0;
-    sha_info->data[115] = 0;
-    sha_info->data[116] = 0;
-    sha_info->data[117] = 0;
-    sha_info->data[118] = 0;
-    sha_info->data[119] = 0;
-    sha_info->data[120] = (hi_bit_count >> 24) & 0xff;
-    sha_info->data[121] = (hi_bit_count >> 16) & 0xff;
-    sha_info->data[122] = (hi_bit_count >>  8) & 0xff;
-    sha_info->data[123] = (hi_bit_count >>  0) & 0xff;
-    sha_info->data[124] = (lo_bit_count >> 24) & 0xff;
-    sha_info->data[125] = (lo_bit_count >> 16) & 0xff;
-    sha_info->data[126] = (lo_bit_count >>  8) & 0xff;
-    sha_info->data[127] = (lo_bit_count >>  0) & 0xff;
-    sha512_transform(sha_info);
-    digest[ 0] = (unsigned char) ((sha_info->digest[0] >> 56) & 0xff);
-    digest[ 1] = (unsigned char) ((sha_info->digest[0] >> 48) & 0xff);
-    digest[ 2] = (unsigned char) ((sha_info->digest[0] >> 40) & 0xff);
-    digest[ 3] = (unsigned char) ((sha_info->digest[0] >> 32) & 0xff);
-    digest[ 4] = (unsigned char) ((sha_info->digest[0] >> 24) & 0xff);
-    digest[ 5] = (unsigned char) ((sha_info->digest[0] >> 16) & 0xff);
-    digest[ 6] = (unsigned char) ((sha_info->digest[0] >>  8) & 0xff);
-    digest[ 7] = (unsigned char) ((sha_info->digest[0]      ) & 0xff);
-    digest[ 8] = (unsigned char) ((sha_info->digest[1] >> 56) & 0xff);
-    digest[ 9] = (unsigned char) ((sha_info->digest[1] >> 48) & 0xff);
-    digest[10] = (unsigned char) ((sha_info->digest[1] >> 40) & 0xff);
-    digest[11] = (unsigned char) ((sha_info->digest[1] >> 32) & 0xff);
-    digest[12] = (unsigned char) ((sha_info->digest[1] >> 24) & 0xff);
-    digest[13] = (unsigned char) ((sha_info->digest[1] >> 16) & 0xff);
-    digest[14] = (unsigned char) ((sha_info->digest[1] >>  8) & 0xff);
-    digest[15] = (unsigned char) ((sha_info->digest[1]      ) & 0xff);
-    digest[16] = (unsigned char) ((sha_info->digest[2] >> 56) & 0xff);
-    digest[17] = (unsigned char) ((sha_info->digest[2] >> 48) & 0xff);
-    digest[18] = (unsigned char) ((sha_info->digest[2] >> 40) & 0xff);
-    digest[19] = (unsigned char) ((sha_info->digest[2] >> 32) & 0xff);
-    digest[20] = (unsigned char) ((sha_info->digest[2] >> 24) & 0xff);
-    digest[21] = (unsigned char) ((sha_info->digest[2] >> 16) & 0xff);
-    digest[22] = (unsigned char) ((sha_info->digest[2] >>  8) & 0xff);
-    digest[23] = (unsigned char) ((sha_info->digest[2]      ) & 0xff);
-    digest[24] = (unsigned char) ((sha_info->digest[3] >> 56) & 0xff);
-    digest[25] = (unsigned char) ((sha_info->digest[3] >> 48) & 0xff);
-    digest[26] = (unsigned char) ((sha_info->digest[3] >> 40) & 0xff);
-    digest[27] = (unsigned char) ((sha_info->digest[3] >> 32) & 0xff);
-    digest[28] = (unsigned char) ((sha_info->digest[3] >> 24) & 0xff);
-    digest[29] = (unsigned char) ((sha_info->digest[3] >> 16) & 0xff);
-    digest[30] = (unsigned char) ((sha_info->digest[3] >>  8) & 0xff);
-    digest[31] = (unsigned char) ((sha_info->digest[3]      ) & 0xff);
-    digest[32] = (unsigned char) ((sha_info->digest[4] >> 56) & 0xff);
-    digest[33] = (unsigned char) ((sha_info->digest[4] >> 48) & 0xff);
-    digest[34] = (unsigned char) ((sha_info->digest[4] >> 40) & 0xff);
-    digest[35] = (unsigned char) ((sha_info->digest[4] >> 32) & 0xff);
-    digest[36] = (unsigned char) ((sha_info->digest[4] >> 24) & 0xff);
-    digest[37] = (unsigned char) ((sha_info->digest[4] >> 16) & 0xff);
-    digest[38] = (unsigned char) ((sha_info->digest[4] >>  8) & 0xff);
-    digest[39] = (unsigned char) ((sha_info->digest[4]      ) & 0xff);
-    digest[40] = (unsigned char) ((sha_info->digest[5] >> 56) & 0xff);
-    digest[41] = (unsigned char) ((sha_info->digest[5] >> 48) & 0xff);
-    digest[42] = (unsigned char) ((sha_info->digest[5] >> 40) & 0xff);
-    digest[43] = (unsigned char) ((sha_info->digest[5] >> 32) & 0xff);
-    digest[44] = (unsigned char) ((sha_info->digest[5] >> 24) & 0xff);
-    digest[45] = (unsigned char) ((sha_info->digest[5] >> 16) & 0xff);
-    digest[46] = (unsigned char) ((sha_info->digest[5] >>  8) & 0xff);
-    digest[47] = (unsigned char) ((sha_info->digest[5]      ) & 0xff);
-    digest[48] = (unsigned char) ((sha_info->digest[6] >> 56) & 0xff);
-    digest[49] = (unsigned char) ((sha_info->digest[6] >> 48) & 0xff);
-    digest[50] = (unsigned char) ((sha_info->digest[6] >> 40) & 0xff);
-    digest[51] = (unsigned char) ((sha_info->digest[6] >> 32) & 0xff);
-    digest[52] = (unsigned char) ((sha_info->digest[6] >> 24) & 0xff);
-    digest[53] = (unsigned char) ((sha_info->digest[6] >> 16) & 0xff);
-    digest[54] = (unsigned char) ((sha_info->digest[6] >>  8) & 0xff);
-    digest[55] = (unsigned char) ((sha_info->digest[6]      ) & 0xff);
-    digest[56] = (unsigned char) ((sha_info->digest[7] >> 56) & 0xff);
-    digest[57] = (unsigned char) ((sha_info->digest[7] >> 48) & 0xff);
-    digest[58] = (unsigned char) ((sha_info->digest[7] >> 40) & 0xff);
-    digest[59] = (unsigned char) ((sha_info->digest[7] >> 32) & 0xff);
-    digest[60] = (unsigned char) ((sha_info->digest[7] >> 24) & 0xff);
-    digest[61] = (unsigned char) ((sha_info->digest[7] >> 16) & 0xff);
-    digest[62] = (unsigned char) ((sha_info->digest[7] >>  8) & 0xff);
-    digest[63] = (unsigned char) ((sha_info->digest[7]      ) & 0xff);
-}
-
-/*
- * End of copied SHA code.
- *
- * ------------------------------------------------------------------------
- */
-
-static PyTypeObject SHA384type;
-static PyTypeObject SHA512type;
-
-
-static SHAobject *
-newSHA384object(void)
-{
-    return (SHAobject *)PyObject_New(SHAobject, &SHA384type);
-}
-
-static SHAobject *
-newSHA512object(void)
-{
-    return (SHAobject *)PyObject_New(SHAobject, &SHA512type);
-}
-
-/* Internal methods for a hash object */
-
-static void
-SHA512_dealloc(PyObject *ptr)
-{
-    PyObject_Del(ptr);
-}
-
-
-/* External methods for a hash object */
-
-PyDoc_STRVAR(SHA512_copy__doc__, "Return a copy of the hash object.");
-
-static PyObject *
-SHA512_copy(SHAobject *self, PyObject *unused)
-{
-    SHAobject *newobj;
-
-    if (((PyObject*)self)->ob_type == &SHA512type) {
-        if ( (newobj = newSHA512object())==NULL)
-            return NULL;
-    } else {
-        if ( (newobj = newSHA384object())==NULL)
-            return NULL;
-    }
-
-    SHAcopy(self, newobj);
-    return (PyObject *)newobj;
-}
-
-PyDoc_STRVAR(SHA512_digest__doc__,
-"Return the digest value as a string of binary data.");
-
-static PyObject *
-SHA512_digest(SHAobject *self, PyObject *unused)
-{
-    unsigned char digest[SHA_DIGESTSIZE];
-    SHAobject temp;
-
-    SHAcopy(self, &temp);
-    sha512_final(digest, &temp);
-    return PyString_FromStringAndSize((const char *)digest, self->digestsize);
-}
-
-PyDoc_STRVAR(SHA512_hexdigest__doc__,
-"Return the digest value as a string of hexadecimal digits.");
-
-static PyObject *
-SHA512_hexdigest(SHAobject *self, PyObject *unused)
-{
-    unsigned char digest[SHA_DIGESTSIZE];
-    SHAobject temp;
-    PyObject *retval;
-    char *hex_digest;
-    int i, j;
-
-    /* Get the raw (binary) digest value */
-    SHAcopy(self, &temp);
-    sha512_final(digest, &temp);
-
-    /* Create a new string */
-    retval = PyString_FromStringAndSize(NULL, self->digestsize * 2);
-    if (!retval)
-	    return NULL;
-    hex_digest = PyString_AsString(retval);
-    if (!hex_digest) {
-	    Py_DECREF(retval);
-	    return NULL;
-    }
-
-    /* Make hex version of the digest */
-    for (i=j=0; i<self->digestsize; i++) {
-        char c;
-        c = (digest[i] >> 4) & 0xf;
-	c = (c>9) ? c+'a'-10 : c + '0';
-        hex_digest[j++] = c;
-        c = (digest[i] & 0xf);
-	c = (c>9) ? c+'a'-10 : c + '0';
-        hex_digest[j++] = c;
-    }
-    return retval;
-}
-
-PyDoc_STRVAR(SHA512_update__doc__,
-"Update this hash object's state with the provided string.");
-
-static PyObject *
-SHA512_update(SHAobject *self, PyObject *args)
-{
-    unsigned char *cp;
-    int len;
-
-    if (!PyArg_ParseTuple(args, "s#:update", &cp, &len))
-        return NULL;
-
-    sha512_update(self, cp, len);
-
-    Py_INCREF(Py_None);
-    return Py_None;
-}
-
-static PyMethodDef SHA_methods[] = {
-    {"copy",	  (PyCFunction)SHA512_copy,      METH_NOARGS, SHA512_copy__doc__},
-    {"digest",	  (PyCFunction)SHA512_digest,    METH_NOARGS, SHA512_digest__doc__},
-    {"hexdigest", (PyCFunction)SHA512_hexdigest, METH_NOARGS, SHA512_hexdigest__doc__},
-    {"update",	  (PyCFunction)SHA512_update,    METH_VARARGS, SHA512_update__doc__},
-    {NULL,	  NULL}		/* sentinel */
-};
-
-static PyObject *
-SHA512_get_block_size(PyObject *self, void *closure)
-{
-    return PyInt_FromLong(SHA_BLOCKSIZE);
-}
-
-static PyObject *
-SHA512_get_name(PyObject *self, void *closure)
-{
-    if (((SHAobject *)self)->digestsize == 64)
-        return PyString_FromStringAndSize("SHA512", 6);
-    else
-        return PyString_FromStringAndSize("SHA384", 6);
-}
-
-static PyGetSetDef SHA_getseters[] = {
-    {"block_size",
-     (getter)SHA512_get_block_size, NULL,
-     NULL,
-     NULL},
-    {"name",
-     (getter)SHA512_get_name, NULL,
-     NULL,
-     NULL},
-    {NULL}  /* Sentinel */
-};
-
-static PyMemberDef SHA_members[] = {
-    {"digest_size", T_INT, offsetof(SHAobject, digestsize), READONLY, NULL},
-    /* the old md5 and sha modules support 'digest_size' as in PEP 247.
-     * the old sha module also supported 'digestsize'.  ugh. */
-    {"digestsize", T_INT, offsetof(SHAobject, digestsize), READONLY, NULL},
-    {NULL}  /* Sentinel */
-};
-
-static PyTypeObject SHA384type = {
-    PyObject_HEAD_INIT(NULL)
-    0,			/*ob_size*/
-    "_sha512.sha384",	/*tp_name*/
-    sizeof(SHAobject),	/*tp_size*/
-    0,			/*tp_itemsize*/
-    /* methods */
-    SHA512_dealloc,	/*tp_dealloc*/
-    0,			/*tp_print*/
-    0,          	/*tp_getattr*/
-    0,                  /*tp_setattr*/
-    0,                  /*tp_compare*/
-    0,                  /*tp_repr*/
-    0,                  /*tp_as_number*/
-    0,                  /*tp_as_sequence*/
-    0,                  /*tp_as_mapping*/
-    0,                  /*tp_hash*/
-    0,                  /*tp_call*/
-    0,                  /*tp_str*/
-    0,                  /*tp_getattro*/
-    0,                  /*tp_setattro*/
-    0,                  /*tp_as_buffer*/
-    Py_TPFLAGS_DEFAULT, /*tp_flags*/
-    0,                  /*tp_doc*/
-    0,                  /*tp_traverse*/
-    0,			/*tp_clear*/
-    0,			/*tp_richcompare*/
-    0,			/*tp_weaklistoffset*/
-    0,			/*tp_iter*/
-    0,			/*tp_iternext*/
-    SHA_methods,	/* tp_methods */
-    SHA_members,	/* tp_members */
-    SHA_getseters,      /* tp_getset */
-};
-
-static PyTypeObject SHA512type = {
-    PyObject_HEAD_INIT(NULL)
-    0,			/*ob_size*/
-    "_sha512.sha512",	/*tp_name*/
-    sizeof(SHAobject),	/*tp_size*/
-    0,			/*tp_itemsize*/
-    /* methods */
-    SHA512_dealloc,	/*tp_dealloc*/
-    0,			/*tp_print*/
-    0,          	/*tp_getattr*/
-    0,                  /*tp_setattr*/
-    0,                  /*tp_compare*/
-    0,                  /*tp_repr*/
-    0,                  /*tp_as_number*/
-    0,                  /*tp_as_sequence*/
-    0,                  /*tp_as_mapping*/
-    0,                  /*tp_hash*/
-    0,                  /*tp_call*/
-    0,                  /*tp_str*/
-    0,                  /*tp_getattro*/
-    0,                  /*tp_setattro*/
-    0,                  /*tp_as_buffer*/
-    Py_TPFLAGS_DEFAULT, /*tp_flags*/
-    0,                  /*tp_doc*/
-    0,                  /*tp_traverse*/
-    0,			/*tp_clear*/
-    0,			/*tp_richcompare*/
-    0,			/*tp_weaklistoffset*/
-    0,			/*tp_iter*/
-    0,			/*tp_iternext*/
-    SHA_methods,	/* tp_methods */
-    SHA_members,	/* tp_members */
-    SHA_getseters,      /* tp_getset */
-};
-
-
-/* The single module-level function: new() */
-
-PyDoc_STRVAR(SHA512_new__doc__,
-"Return a new SHA-512 hash object; optionally initialized with a string.");
-
-static PyObject *
-SHA512_new(PyObject *self, PyObject *args, PyObject *kwdict)
-{
-    static char *kwlist[] = {"string", NULL};
-    SHAobject *new;
-    unsigned char *cp = NULL;
-    int len;
-
-    if (!PyArg_ParseTupleAndKeywords(args, kwdict, "|s#:new", kwlist,
-                                     &cp, &len)) {
-        return NULL;
-    }
-
-    if ((new = newSHA512object()) == NULL)
-        return NULL;
-
-    sha512_init(new);
-
-    if (PyErr_Occurred()) {
-        Py_DECREF(new);
-        return NULL;
-    }
-    if (cp)
-        sha512_update(new, cp, len);
-
-    return (PyObject *)new;
-}
-
-PyDoc_STRVAR(SHA384_new__doc__,
-"Return a new SHA-384 hash object; optionally initialized with a string.");
-
-static PyObject *
-SHA384_new(PyObject *self, PyObject *args, PyObject *kwdict)
-{
-    static char *kwlist[] = {"string", NULL};
-    SHAobject *new;
-    unsigned char *cp = NULL;
-    int len;
-
-    if (!PyArg_ParseTupleAndKeywords(args, kwdict, "|s#:new", kwlist,
-                                     &cp, &len)) {
-        return NULL;
-    }
-
-    if ((new = newSHA384object()) == NULL)
-        return NULL;
-
-    sha384_init(new);
-
-    if (PyErr_Occurred()) {
-        Py_DECREF(new);
-        return NULL;
-    }
-    if (cp)
-        sha512_update(new, cp, len);
-
-    return (PyObject *)new;
-}
-
-
-/* List of functions exported by this module */
-
-static struct PyMethodDef SHA_functions[] = {
-    {"sha512", (PyCFunction)SHA512_new, METH_VARARGS|METH_KEYWORDS, SHA512_new__doc__},
-    {"sha384", (PyCFunction)SHA384_new, METH_VARARGS|METH_KEYWORDS, SHA384_new__doc__},
-    {NULL,	NULL}		 /* Sentinel */
-};
-
-
-/* Initialize this module. */
-
-#define insint(n,v) { PyModule_AddIntConstant(m,n,v); }
-
-PyMODINIT_FUNC
-init_sha512(void)
-{
-    PyObject *m;
-
-    SHA384type.ob_type = &PyType_Type;
-    if (PyType_Ready(&SHA384type) < 0)
-        return;
-    SHA512type.ob_type = &PyType_Type;
-    if (PyType_Ready(&SHA512type) < 0)
-        return;
-    m = Py_InitModule("_sha512", SHA_functions);
-    if (m == NULL)
-	return;
-}
-
-#endif
diff --git a/distutils2/_backport/shamodule.c b/distutils2/_backport/shamodule.c
deleted file mode 100644
--- a/distutils2/_backport/shamodule.c
+++ /dev/null
@@ -1,593 +0,0 @@
-/* SHA module */
-
-/* This module provides an interface to NIST's Secure Hash Algorithm */
-
-/* See below for information about the original code this module was
-   based upon. Additional work performed by:
-
-   Andrew Kuchling (amk at amk.ca)
-   Greg Stein (gstein at lyra.org)
-
-   Copyright (C) 2005   Gregory P. Smith (greg at krypto.org)
-   Licensed to PSF under a Contributor Agreement.
-
-*/
-
-/* SHA objects */
-
-#include "Python.h"
-#include "structmember.h"
-
-
-/* Endianness testing and definitions */
-#define TestEndianness(variable) {int i=1; variable=PCT_BIG_ENDIAN;\
-	if (*((char*)&i)==1) variable=PCT_LITTLE_ENDIAN;}
-
-#define PCT_LITTLE_ENDIAN 1
-#define PCT_BIG_ENDIAN 0
-
-/* Some useful types */
-
-typedef unsigned char SHA_BYTE;
-
-#if SIZEOF_INT == 4
-typedef unsigned int SHA_INT32;	/* 32-bit integer */
-#else
-/* not defined. compilation will die. */
-#endif
-
-/* The SHA block size and message digest sizes, in bytes */
-
-#define SHA_BLOCKSIZE    64
-#define SHA_DIGESTSIZE  20
-
-/* The structure for storing SHS info */
-
-typedef struct {
-    PyObject_HEAD
-    SHA_INT32 digest[5];		/* Message digest */
-    SHA_INT32 count_lo, count_hi;	/* 64-bit bit count */
-    SHA_BYTE data[SHA_BLOCKSIZE];	/* SHA data buffer */
-    int Endianness;
-    int local;				/* unprocessed amount in data */
-} SHAobject;
-
-/* When run on a little-endian CPU we need to perform byte reversal on an
-   array of longwords. */
-
-static void longReverse(SHA_INT32 *buffer, int byteCount, int Endianness)
-{
-    SHA_INT32 value;
-
-    if ( Endianness == PCT_BIG_ENDIAN )
-	return;
-
-    byteCount /= sizeof(*buffer);
-    while (byteCount--) {
-        value = *buffer;
-        value = ( ( value & 0xFF00FF00L ) >> 8  ) | \
-                ( ( value & 0x00FF00FFL ) << 8 );
-        *buffer++ = ( value << 16 ) | ( value >> 16 );
-    }
-}
-
-static void SHAcopy(SHAobject *src, SHAobject *dest)
-{
-    dest->Endianness = src->Endianness;
-    dest->local = src->local;
-    dest->count_lo = src->count_lo;
-    dest->count_hi = src->count_hi;
-    memcpy(dest->digest, src->digest, sizeof(src->digest));
-    memcpy(dest->data, src->data, sizeof(src->data));
-}
-
-
-/* ------------------------------------------------------------------------
- *
- * This code for the SHA algorithm was noted as public domain. The original
- * headers are pasted below.
- *
- * Several changes have been made to make it more compatible with the
- * Python environment and desired interface.
- *
- */
-
-/* NIST Secure Hash Algorithm */
-/* heavily modified by Uwe Hollerbach <uh at alumni.caltech edu> */
-/* from Peter C. Gutmann's implementation as found in */
-/* Applied Cryptography by Bruce Schneier */
-/* Further modifications to include the "UNRAVEL" stuff, below */
-
-/* This code is in the public domain */
-
-/* UNRAVEL should be fastest & biggest */
-/* UNROLL_LOOPS should be just as big, but slightly slower */
-/* both undefined should be smallest and slowest */
-
-#define UNRAVEL
-/* #define UNROLL_LOOPS */
-
-/* The SHA f()-functions.  The f1 and f3 functions can be optimized to
-   save one boolean operation each - thanks to Rich Schroeppel,
-   rcs at cs.arizona.edu for discovering this */
-
-/*#define f1(x,y,z)	((x & y) | (~x & z))		// Rounds  0-19 */
-#define f1(x,y,z)	(z ^ (x & (y ^ z)))		/* Rounds  0-19 */
-#define f2(x,y,z)	(x ^ y ^ z)			/* Rounds 20-39 */
-/*#define f3(x,y,z)	((x & y) | (x & z) | (y & z))	// Rounds 40-59 */
-#define f3(x,y,z)	((x & y) | (z & (x | y)))	/* Rounds 40-59 */
-#define f4(x,y,z)	(x ^ y ^ z)			/* Rounds 60-79 */
-
-/* SHA constants */
-
-#define CONST1		0x5a827999L			/* Rounds  0-19 */
-#define CONST2		0x6ed9eba1L			/* Rounds 20-39 */
-#define CONST3		0x8f1bbcdcL			/* Rounds 40-59 */
-#define CONST4		0xca62c1d6L			/* Rounds 60-79 */
-
-/* 32-bit rotate */
-
-#define R32(x,n)	((x << n) | (x >> (32 - n)))
-
-/* the generic case, for when the overall rotation is not unraveled */
-
-#define FG(n)	\
-    T = R32(A,5) + f##n(B,C,D) + E + *WP++ + CONST##n;	\
-    E = D; D = C; C = R32(B,30); B = A; A = T
-
-/* specific cases, for when the overall rotation is unraveled */
-
-#define FA(n)	\
-    T = R32(A,5) + f##n(B,C,D) + E + *WP++ + CONST##n; B = R32(B,30)
-
-#define FB(n)	\
-    E = R32(T,5) + f##n(A,B,C) + D + *WP++ + CONST##n; A = R32(A,30)
-
-#define FC(n)	\
-    D = R32(E,5) + f##n(T,A,B) + C + *WP++ + CONST##n; T = R32(T,30)
-
-#define FD(n)	\
-    C = R32(D,5) + f##n(E,T,A) + B + *WP++ + CONST##n; E = R32(E,30)
-
-#define FE(n)	\
-    B = R32(C,5) + f##n(D,E,T) + A + *WP++ + CONST##n; D = R32(D,30)
-
-#define FT(n)	\
-    A = R32(B,5) + f##n(C,D,E) + T + *WP++ + CONST##n; C = R32(C,30)
-
-/* do SHA transformation */
-
-static void
-sha_transform(SHAobject *sha_info)
-{
-    int i;
-    SHA_INT32 T, A, B, C, D, E, W[80], *WP;
-
-    memcpy(W, sha_info->data, sizeof(sha_info->data));
-    longReverse(W, (int)sizeof(sha_info->data), sha_info->Endianness);
-
-    for (i = 16; i < 80; ++i) {
-	W[i] = W[i-3] ^ W[i-8] ^ W[i-14] ^ W[i-16];
-
-	/* extra rotation fix */
-	W[i] = R32(W[i], 1);
-    }
-    A = sha_info->digest[0];
-    B = sha_info->digest[1];
-    C = sha_info->digest[2];
-    D = sha_info->digest[3];
-    E = sha_info->digest[4];
-    WP = W;
-#ifdef UNRAVEL
-    FA(1); FB(1); FC(1); FD(1); FE(1); FT(1); FA(1); FB(1); FC(1); FD(1);
-    FE(1); FT(1); FA(1); FB(1); FC(1); FD(1); FE(1); FT(1); FA(1); FB(1);
-    FC(2); FD(2); FE(2); FT(2); FA(2); FB(2); FC(2); FD(2); FE(2); FT(2);
-    FA(2); FB(2); FC(2); FD(2); FE(2); FT(2); FA(2); FB(2); FC(2); FD(2);
-    FE(3); FT(3); FA(3); FB(3); FC(3); FD(3); FE(3); FT(3); FA(3); FB(3);
-    FC(3); FD(3); FE(3); FT(3); FA(3); FB(3); FC(3); FD(3); FE(3); FT(3);
-    FA(4); FB(4); FC(4); FD(4); FE(4); FT(4); FA(4); FB(4); FC(4); FD(4);
-    FE(4); FT(4); FA(4); FB(4); FC(4); FD(4); FE(4); FT(4); FA(4); FB(4);
-    sha_info->digest[0] += E;
-    sha_info->digest[1] += T;
-    sha_info->digest[2] += A;
-    sha_info->digest[3] += B;
-    sha_info->digest[4] += C;
-#else /* !UNRAVEL */
-#ifdef UNROLL_LOOPS
-    FG(1); FG(1); FG(1); FG(1); FG(1); FG(1); FG(1); FG(1); FG(1); FG(1);
-    FG(1); FG(1); FG(1); FG(1); FG(1); FG(1); FG(1); FG(1); FG(1); FG(1);
-    FG(2); FG(2); FG(2); FG(2); FG(2); FG(2); FG(2); FG(2); FG(2); FG(2);
-    FG(2); FG(2); FG(2); FG(2); FG(2); FG(2); FG(2); FG(2); FG(2); FG(2);
-    FG(3); FG(3); FG(3); FG(3); FG(3); FG(3); FG(3); FG(3); FG(3); FG(3);
-    FG(3); FG(3); FG(3); FG(3); FG(3); FG(3); FG(3); FG(3); FG(3); FG(3);
-    FG(4); FG(4); FG(4); FG(4); FG(4); FG(4); FG(4); FG(4); FG(4); FG(4);
-    FG(4); FG(4); FG(4); FG(4); FG(4); FG(4); FG(4); FG(4); FG(4); FG(4);
-#else /* !UNROLL_LOOPS */
-    for (i =  0; i < 20; ++i) { FG(1); }
-    for (i = 20; i < 40; ++i) { FG(2); }
-    for (i = 40; i < 60; ++i) { FG(3); }
-    for (i = 60; i < 80; ++i) { FG(4); }
-#endif /* !UNROLL_LOOPS */
-    sha_info->digest[0] += A;
-    sha_info->digest[1] += B;
-    sha_info->digest[2] += C;
-    sha_info->digest[3] += D;
-    sha_info->digest[4] += E;
-#endif /* !UNRAVEL */
-}
-
-/* initialize the SHA digest */
-
-static void
-sha_init(SHAobject *sha_info)
-{
-    TestEndianness(sha_info->Endianness)
-
-    sha_info->digest[0] = 0x67452301L;
-    sha_info->digest[1] = 0xefcdab89L;
-    sha_info->digest[2] = 0x98badcfeL;
-    sha_info->digest[3] = 0x10325476L;
-    sha_info->digest[4] = 0xc3d2e1f0L;
-    sha_info->count_lo = 0L;
-    sha_info->count_hi = 0L;
-    sha_info->local = 0;
-}
-
-/* update the SHA digest */
-
-static void
-sha_update(SHAobject *sha_info, SHA_BYTE *buffer, int count)
-{
-    int i;
-    SHA_INT32 clo;
-
-    clo = sha_info->count_lo + ((SHA_INT32) count << 3);
-    if (clo < sha_info->count_lo) {
-        ++sha_info->count_hi;
-    }
-    sha_info->count_lo = clo;
-    sha_info->count_hi += (SHA_INT32) count >> 29;
-    if (sha_info->local) {
-        i = SHA_BLOCKSIZE - sha_info->local;
-        if (i > count) {
-            i = count;
-        }
-        memcpy(((SHA_BYTE *) sha_info->data) + sha_info->local, buffer, i);
-        count -= i;
-        buffer += i;
-        sha_info->local += i;
-        if (sha_info->local == SHA_BLOCKSIZE) {
-            sha_transform(sha_info);
-        }
-        else {
-            return;
-        }
-    }
-    while (count >= SHA_BLOCKSIZE) {
-        memcpy(sha_info->data, buffer, SHA_BLOCKSIZE);
-        buffer += SHA_BLOCKSIZE;
-        count -= SHA_BLOCKSIZE;
-        sha_transform(sha_info);
-    }
-    memcpy(sha_info->data, buffer, count);
-    sha_info->local = count;
-}
-
-/* finish computing the SHA digest */
-
-static void
-sha_final(unsigned char digest[20], SHAobject *sha_info)
-{
-    int count;
-    SHA_INT32 lo_bit_count, hi_bit_count;
-
-    lo_bit_count = sha_info->count_lo;
-    hi_bit_count = sha_info->count_hi;
-    count = (int) ((lo_bit_count >> 3) & 0x3f);
-    ((SHA_BYTE *) sha_info->data)[count++] = 0x80;
-    if (count > SHA_BLOCKSIZE - 8) {
-	memset(((SHA_BYTE *) sha_info->data) + count, 0,
-	       SHA_BLOCKSIZE - count);
-	sha_transform(sha_info);
-	memset((SHA_BYTE *) sha_info->data, 0, SHA_BLOCKSIZE - 8);
-    }
-    else {
-	memset(((SHA_BYTE *) sha_info->data) + count, 0,
-	       SHA_BLOCKSIZE - 8 - count);
-    }
-
-    /* GJS: note that we add the hi/lo in big-endian. sha_transform will
-       swap these values into host-order. */
-    sha_info->data[56] = (hi_bit_count >> 24) & 0xff;
-    sha_info->data[57] = (hi_bit_count >> 16) & 0xff;
-    sha_info->data[58] = (hi_bit_count >>  8) & 0xff;
-    sha_info->data[59] = (hi_bit_count >>  0) & 0xff;
-    sha_info->data[60] = (lo_bit_count >> 24) & 0xff;
-    sha_info->data[61] = (lo_bit_count >> 16) & 0xff;
-    sha_info->data[62] = (lo_bit_count >>  8) & 0xff;
-    sha_info->data[63] = (lo_bit_count >>  0) & 0xff;
-    sha_transform(sha_info);
-    digest[ 0] = (unsigned char) ((sha_info->digest[0] >> 24) & 0xff);
-    digest[ 1] = (unsigned char) ((sha_info->digest[0] >> 16) & 0xff);
-    digest[ 2] = (unsigned char) ((sha_info->digest[0] >>  8) & 0xff);
-    digest[ 3] = (unsigned char) ((sha_info->digest[0]      ) & 0xff);
-    digest[ 4] = (unsigned char) ((sha_info->digest[1] >> 24) & 0xff);
-    digest[ 5] = (unsigned char) ((sha_info->digest[1] >> 16) & 0xff);
-    digest[ 6] = (unsigned char) ((sha_info->digest[1] >>  8) & 0xff);
-    digest[ 7] = (unsigned char) ((sha_info->digest[1]      ) & 0xff);
-    digest[ 8] = (unsigned char) ((sha_info->digest[2] >> 24) & 0xff);
-    digest[ 9] = (unsigned char) ((sha_info->digest[2] >> 16) & 0xff);
-    digest[10] = (unsigned char) ((sha_info->digest[2] >>  8) & 0xff);
-    digest[11] = (unsigned char) ((sha_info->digest[2]      ) & 0xff);
-    digest[12] = (unsigned char) ((sha_info->digest[3] >> 24) & 0xff);
-    digest[13] = (unsigned char) ((sha_info->digest[3] >> 16) & 0xff);
-    digest[14] = (unsigned char) ((sha_info->digest[3] >>  8) & 0xff);
-    digest[15] = (unsigned char) ((sha_info->digest[3]      ) & 0xff);
-    digest[16] = (unsigned char) ((sha_info->digest[4] >> 24) & 0xff);
-    digest[17] = (unsigned char) ((sha_info->digest[4] >> 16) & 0xff);
-    digest[18] = (unsigned char) ((sha_info->digest[4] >>  8) & 0xff);
-    digest[19] = (unsigned char) ((sha_info->digest[4]      ) & 0xff);
-}
-
-/*
- * End of copied SHA code.
- *
- * ------------------------------------------------------------------------
- */
-
-static PyTypeObject SHAtype;
-
-
-static SHAobject *
-newSHAobject(void)
-{
-    return (SHAobject *)PyObject_New(SHAobject, &SHAtype);
-}
-
-/* Internal methods for a hashing object */
-
-static void
-SHA_dealloc(PyObject *ptr)
-{
-    PyObject_Del(ptr);
-}
-
-
-/* External methods for a hashing object */
-
-PyDoc_STRVAR(SHA_copy__doc__, "Return a copy of the hashing object.");
-
-static PyObject *
-SHA_copy(SHAobject *self, PyObject *unused)
-{
-    SHAobject *newobj;
-
-    if ( (newobj = newSHAobject())==NULL)
-        return NULL;
-
-    SHAcopy(self, newobj);
-    return (PyObject *)newobj;
-}
-
-PyDoc_STRVAR(SHA_digest__doc__,
-"Return the digest value as a string of binary data.");
-
-static PyObject *
-SHA_digest(SHAobject *self, PyObject *unused)
-{
-    unsigned char digest[SHA_DIGESTSIZE];
-    SHAobject temp;
-
-    SHAcopy(self, &temp);
-    sha_final(digest, &temp);
-    return PyString_FromStringAndSize((const char *)digest, sizeof(digest));
-}
-
-PyDoc_STRVAR(SHA_hexdigest__doc__,
-"Return the digest value as a string of hexadecimal digits.");
-
-static PyObject *
-SHA_hexdigest(SHAobject *self, PyObject *unused)
-{
-    unsigned char digest[SHA_DIGESTSIZE];
-    SHAobject temp;
-    PyObject *retval;
-    char *hex_digest;
-    int i, j;
-
-    /* Get the raw (binary) digest value */
-    SHAcopy(self, &temp);
-    sha_final(digest, &temp);
-
-    /* Create a new string */
-    retval = PyString_FromStringAndSize(NULL, sizeof(digest) * 2);
-    if (!retval)
-	    return NULL;
-    hex_digest = PyString_AsString(retval);
-    if (!hex_digest) {
-	    Py_DECREF(retval);
-	    return NULL;
-    }
-
-    /* Make hex version of the digest */
-    for(i=j=0; i<sizeof(digest); i++) {
-        char c;
-        c = (digest[i] >> 4) & 0xf;
-	c = (c>9) ? c+'a'-10 : c + '0';
-        hex_digest[j++] = c;
-        c = (digest[i] & 0xf);
-	c = (c>9) ? c+'a'-10 : c + '0';
-        hex_digest[j++] = c;
-    }
-    return retval;
-}
-
-PyDoc_STRVAR(SHA_update__doc__,
-"Update this hashing object's state with the provided string.");
-
-static PyObject *
-SHA_update(SHAobject *self, PyObject *args)
-{
-    unsigned char *cp;
-    int len;
-
-    if (!PyArg_ParseTuple(args, "s#:update", &cp, &len))
-        return NULL;
-
-    sha_update(self, cp, len);
-
-    Py_INCREF(Py_None);
-    return Py_None;
-}
-
-static PyMethodDef SHA_methods[] = {
-    {"copy",	  (PyCFunction)SHA_copy,      METH_NOARGS,  SHA_copy__doc__},
-    {"digest",	  (PyCFunction)SHA_digest,    METH_NOARGS,  SHA_digest__doc__},
-    {"hexdigest", (PyCFunction)SHA_hexdigest, METH_NOARGS,  SHA_hexdigest__doc__},
-    {"update",	  (PyCFunction)SHA_update,    METH_VARARGS, SHA_update__doc__},
-    {NULL,	  NULL}		/* sentinel */
-};
-
-static PyObject *
-SHA_get_block_size(PyObject *self, void *closure)
-{
-    return PyInt_FromLong(SHA_BLOCKSIZE);
-}
-
-static PyObject *
-SHA_get_digest_size(PyObject *self, void *closure)
-{
-    return PyInt_FromLong(SHA_DIGESTSIZE);
-}
-
-static PyObject *
-SHA_get_name(PyObject *self, void *closure)
-{
-    return PyString_FromStringAndSize("SHA1", 4);
-}
-
-static PyGetSetDef SHA_getseters[] = {
-    {"digest_size",
-     (getter)SHA_get_digest_size, NULL,
-     NULL,
-     NULL},
-    {"block_size",
-     (getter)SHA_get_block_size, NULL,
-     NULL,
-     NULL},
-    {"name",
-     (getter)SHA_get_name, NULL,
-     NULL,
-     NULL},
-    /* the old md5 and sha modules support 'digest_size' as in PEP 247.
-     * the old sha module also supported 'digestsize'.  ugh. */
-    {"digestsize",
-     (getter)SHA_get_digest_size, NULL,
-     NULL,
-     NULL},
-    {NULL}  /* Sentinel */
-};
-
-static PyTypeObject SHAtype = {
-    PyObject_HEAD_INIT(NULL)
-    0,			/*ob_size*/
-    "_sha.sha",		/*tp_name*/
-    sizeof(SHAobject),	/*tp_size*/
-    0,			/*tp_itemsize*/
-    /* methods */
-    SHA_dealloc,	/*tp_dealloc*/
-    0,			/*tp_print*/
-    0,                  /*tp_getattr*/
-    0,                  /*tp_setattr*/
-    0,                  /*tp_compare*/
-    0,                  /*tp_repr*/
-    0,                  /*tp_as_number*/
-    0,                  /*tp_as_sequence*/
-    0,                  /*tp_as_mapping*/
-    0,                  /*tp_hash*/
-    0,                  /*tp_call*/
-    0,                  /*tp_str*/
-    0,                  /*tp_getattro*/
-    0,                  /*tp_setattro*/
-    0,                  /*tp_as_buffer*/
-    Py_TPFLAGS_DEFAULT, /*tp_flags*/
-    0,                  /*tp_doc*/
-    0,                  /*tp_traverse*/
-    0,			/*tp_clear*/
-    0,			/*tp_richcompare*/
-    0,			/*tp_weaklistoffset*/
-    0,			/*tp_iter*/
-    0,			/*tp_iternext*/
-    SHA_methods,	/* tp_methods */
-    0,                  /* tp_members */
-    SHA_getseters,      /* tp_getset */
-};
-
-
-/* The single module-level function: new() */
-
-PyDoc_STRVAR(SHA_new__doc__,
-"Return a new SHA hashing object.  An optional string argument\n\
-may be provided; if present, this string will be automatically\n\
-hashed.");
-
-static PyObject *
-SHA_new(PyObject *self, PyObject *args, PyObject *kwdict)
-{
-    static char *kwlist[] = {"string", NULL};
-    SHAobject *new;
-    unsigned char *cp = NULL;
-    int len;
-
-    if (!PyArg_ParseTupleAndKeywords(args, kwdict, "|s#:new", kwlist,
-                                     &cp, &len)) {
-        return NULL;
-    }
-
-    if ((new = newSHAobject()) == NULL)
-        return NULL;
-
-    sha_init(new);
-
-    if (PyErr_Occurred()) {
-        Py_DECREF(new);
-        return NULL;
-    }
-    if (cp)
-        sha_update(new, cp, len);
-
-    return (PyObject *)new;
-}
-
-
-/* List of functions exported by this module */
-
-static struct PyMethodDef SHA_functions[] = {
-    {"new", (PyCFunction)SHA_new, METH_VARARGS|METH_KEYWORDS, SHA_new__doc__},
-    {NULL,	NULL}		 /* Sentinel */
-};
-
-
-/* Initialize this module. */
-
-#define insint(n,v) { PyModule_AddIntConstant(m,n,v); }
-
-PyMODINIT_FUNC
-init_sha(void)
-{
-    PyObject *m;
-
-    SHAtype.ob_type = &PyType_Type;
-    if (PyType_Ready(&SHAtype) < 0)
-        return;
-    m = Py_InitModule("_sha", SHA_functions);
-    if (m == NULL)
-	return;
-
-    /* Add some symbolic constants to the module */
-    insint("blocksize", 1);  /* For future use, in case some hash
-                                functions require an integral number of
-                                blocks */ 
-    insint("digestsize", 20);
-    insint("digest_size", 20);
-}
diff --git a/distutils2/_backport/shutil.py b/distutils2/_backport/shutil.py
--- a/distutils2/_backport/shutil.py
+++ b/distutils2/_backport/shutil.py
@@ -9,6 +9,7 @@
 import stat
 from os.path import abspath
 import fnmatch
+import collections
 import errno
 
 try:
@@ -93,15 +94,9 @@
             if stat.S_ISFIFO(st.st_mode):
                 raise SpecialFileError("`%s` is a named pipe" % fn)
 
-    fsrc = open(src, 'rb')
-    try:
-        fdst = open(dst, 'wb')
-        try:
+    with open(src, 'rb') as fsrc:
+        with open(dst, 'wb') as fdst:
             copyfileobj(fsrc, fdst)
-        finally:
-            fdst.close()
-    finally:
-        fsrc.close()
 
 def copymode(src, dst):
     """Copy mode bits from src to dst"""
@@ -121,7 +116,7 @@
     if hasattr(os, 'chflags') and hasattr(st, 'st_flags'):
         try:
             os.chflags(dst, st.st_flags)
-        except OSError, why:
+        except OSError as why:
             if (not hasattr(errno, 'EOPNOTSUPP') or
                 why.errno != errno.EOPNOTSUPP):
                 raise
@@ -204,7 +199,7 @@
 
     try:
         os.makedirs(dst)
-    except OSError, e:
+    except OSError as e:
         if e.errno != errno.EEXIST:
             raise
 
@@ -232,13 +227,13 @@
                 copy_function(srcname, dstname)
         # catch the Error from the recursive copytree so that we can
         # continue with other files
-        except Error, err:
+        except Error as err:
             errors.extend(err.args[0])
-        except EnvironmentError, why:
+        except EnvironmentError as why:
             errors.append((srcname, dstname, str(why)))
     try:
         copystat(src, dst)
-    except OSError, why:
+    except OSError as why:
         if WindowsError is not None and isinstance(why, WindowsError):
             # Copying file access times may fail on Windows
             pass
@@ -400,7 +395,7 @@
     # flags for compression program, each element of list will be an argument
     if compress is not None and compress not in compress_ext:
         raise ValueError("bad value for 'compress', or compression format not "
-                         "supported: %s" % compress)
+                         "supported : {0}".format(compress))
 
     archive_name = base_name + '.tar' + compress_ext.get(compress, '')
     archive_dir = os.path.dirname(archive_name)
@@ -521,7 +516,7 @@
     Each element of the returned sequence is a tuple (name, description)
     """
     formats = [(name, registry[2]) for name, registry in
-               _ARCHIVE_FORMATS.iteritems()]
+               _ARCHIVE_FORMATS.items()]
     formats.sort()
     return formats
 
@@ -536,7 +531,7 @@
     """
     if extra_args is None:
         extra_args = []
-    if not callable(function):
+    if not isinstance(function, collections.Callable):
         raise TypeError('The %s object is not callable' % function)
     if not isinstance(extra_args, (tuple, list)):
         raise TypeError('extra_args needs to be a sequence')
@@ -611,7 +606,7 @@
     (name, extensions, description)
     """
     formats = [(name, info[0], info[3]) for name, info in
-               _UNPACK_FORMATS.iteritems()]
+               _UNPACK_FORMATS.items()]
     formats.sort()
     return formats
 
@@ -619,7 +614,7 @@
     """Checks what gets registered as an unpacker."""
     # first make sure no other unpacker is registered for this extension
     existing_extensions = {}
-    for name, info in _UNPACK_FORMATS.iteritems():
+    for name, info in _UNPACK_FORMATS.items():
         for ext in info[0]:
             existing_extensions[ext] = name
 
@@ -629,7 +624,7 @@
             raise RegistryError(msg % (extension,
                                        existing_extensions[extension]))
 
-    if not callable(function):
+    if not isinstance(function, collections.Callable):
         raise TypeError('The registered function must be a callable')
 
 
@@ -728,7 +723,7 @@
                                 "bzip2'ed tar-file")
 
 def _find_unpack_format(filename):
-    for name, info in _UNPACK_FORMATS.iteritems():
+    for name, info in _UNPACK_FORMATS.items():
         for extension in info[0]:
             if filename.endswith(extension):
                 return name
@@ -756,7 +751,7 @@
         try:
             format_info = _UNPACK_FORMATS[format]
         except KeyError:
-            raise ValueError("Unknown unpack format '%s'" % format)
+            raise ValueError("Unknown unpack format '{0}'".format(format))
 
         func = format_info[1]
         func(filename, extract_dir, **dict(format_info[2]))
@@ -764,7 +759,7 @@
         # we need to look at the registered unpackers supported extensions
         format = _find_unpack_format(filename)
         if format is None:
-            raise ReadError("Unknown archive format '%s'" % filename)
+            raise ReadError("Unknown archive format '{0}'".format(filename))
 
         func = _UNPACK_FORMATS[format][1]
         kwargs = dict(_UNPACK_FORMATS[format][2])
diff --git a/distutils2/_backport/sysconfig.cfg b/distutils2/_backport/sysconfig.cfg
--- a/distutils2/_backport/sysconfig.cfg
+++ b/distutils2/_backport/sysconfig.cfg
@@ -40,8 +40,8 @@
 platstdlib = {platbase}/lib/python{py_version_short}
 purelib = {base}/lib/python{py_version_short}/site-packages
 platlib = {platbase}/lib/python{py_version_short}/site-packages
-include = {base}/include/python{py_version_short}
-platinclude = {platbase}/include/python{py_version_short}
+include = {base}/include/python{py_version_short}{abiflags}
+platinclude = {platbase}/include/python{py_version_short}{abiflags}
 data = {base}
 
 [posix_home]
diff --git a/distutils2/_backport/sysconfig.py b/distutils2/_backport/sysconfig.py
--- a/distutils2/_backport/sysconfig.py
+++ b/distutils2/_backport/sysconfig.py
@@ -4,7 +4,7 @@
 import re
 import sys
 from os.path import pardir, realpath
-from ConfigParser import RawConfigParser
+from configparser import RawConfigParser
 
 __all__ = [
     'get_config_h_filename',
@@ -209,11 +209,8 @@
     done = {}
     notdone = {}
 
-    f = open(filename)
-    try:
+    with open(filename, errors="surrogateescape") as f:
         lines = f.readlines()
-    finally:
-        f.close()
 
     for line in lines:
         if line.startswith('#') or line.strip() == '':
@@ -237,7 +234,7 @@
                     done[n] = v
 
     # do variable interpolation here
-    variables = notdone.keys()
+    variables = list(notdone.keys())
 
     # Variables with a 'PY_' prefix in the makefile. These need to
     # be made available without that prefix through sysconfig.
@@ -316,7 +313,10 @@
     """Return the path of the Makefile."""
     if _PYTHON_BUILD:
         return os.path.join(_PROJECT_BASE, "Makefile")
-    config_dir_name = 'config'
+    if hasattr(sys, 'abiflags'):
+        config_dir_name = 'config-%s%s' % (_PY_VERSION_SHORT, sys.abiflags)
+    else:
+        config_dir_name = 'config'
     return os.path.join(get_path('stdlib'), config_dir_name, 'Makefile')
 
 
@@ -326,7 +326,7 @@
     makefile = get_makefile_filename()
     try:
         _parse_makefile(makefile, vars)
-    except IOError, e:
+    except IOError as e:
         msg = "invalid Python installation: unable to open %s" % makefile
         if hasattr(e, "strerror"):
             msg = msg + " (%s)" % e.strerror
@@ -334,12 +334,9 @@
     # load the installed pyconfig.h:
     config_h = get_config_h_filename()
     try:
-        f = open(config_h)
-        try:
+        with open(config_h) as f:
             parse_config_h(f, vars)
-        finally:
-            f.close()
-    except IOError, e:
+    except IOError as e:
         msg = "invalid Python installation: unable to open %s" % config_h
         if hasattr(e, "strerror"):
             msg = msg + " (%s)" % e.strerror
@@ -465,6 +462,11 @@
         _CONFIG_VARS['base'] = _PREFIX
         _CONFIG_VARS['platbase'] = _EXEC_PREFIX
         _CONFIG_VARS['projectbase'] = _PROJECT_BASE
+        try:
+            _CONFIG_VARS['abiflags'] = sys.abiflags
+        except AttributeError:
+            # sys.abiflags may not be defined on all platforms.
+            _CONFIG_VARS['abiflags'] = ''
 
         if os.name in ('nt', 'os2'):
             _init_non_posix(_CONFIG_VARS)
@@ -724,13 +726,13 @@
                 # On OSX the machine type returned by uname is always the
                 # 32-bit variant, even if the executable architecture is
                 # the 64-bit variant
-                if sys.maxint >= 2**32:
+                if sys.maxsize >= 2**32:
                     machine = 'x86_64'
 
             elif machine in ('PowerPC', 'Power_Macintosh'):
                 # Pick a sane name for the PPC architecture.
                 # See 'i386' case
-                if sys.maxint >= 2**32:
+                if sys.maxsize >= 2**32:
                     machine = 'ppc64'
                 else:
                     machine = 'ppc'
@@ -745,18 +747,18 @@
 def _print_dict(title, data):
     for index, (key, value) in enumerate(sorted(data.items())):
         if index == 0:
-            print '%s: ' % (title)
-        print '\t%s = "%s"' % (key, value)
+            print('%s: ' % (title))
+        print('\t%s = "%s"' % (key, value))
 
 
 def _main():
     """Display all information sysconfig detains."""
-    print 'Platform: "%s"' % get_platform()
-    print 'Python version: "%s"' % get_python_version()
-    print 'Current installation scheme: "%s"' % _get_default_scheme()
-    print
+    print('Platform: "%s"' % get_platform())
+    print('Python version: "%s"' % get_python_version())
+    print('Current installation scheme: "%s"' % _get_default_scheme())
+    print()
     _print_dict('Paths', get_paths())
-    print
+    print()
     _print_dict('Variables', get_config_vars())
 
 
diff --git a/distutils2/_backport/tarfile.py b/distutils2/_backport/tarfile.py
--- a/distutils2/_backport/tarfile.py
+++ b/distutils2/_backport/tarfile.py
@@ -1,9 +1,8 @@
-#!/usr/bin/env python
-# encoding: utf-8
+#!/usr/bin/env python3
 #-------------------------------------------------------------------
 # tarfile.py
 #-------------------------------------------------------------------
-# Copyright (C) 2002 Lars Gust?bel <lars at gustaebel.de>
+# Copyright (C) 2002 Lars Gustaebel <lars at gustaebel.de>
 # All rights reserved.
 #
 # Permission  is  hereby granted,  free  of charge,  to  any person
@@ -33,10 +32,10 @@
 __version__ = "$Revision$"
 
 version     = "0.9.0"
-__author__  = u"Lars Gust\u00e4bel (lars at gustaebel.de)"
+__author__  = "Lars Gust\u00e4bel (lars at gustaebel.de)"
 __date__    = "$Date: 2011-02-25 17:42:01 +0200 (Fri, 25 Feb 2011) $"
 __cvsid__   = "$Id: tarfile.py 88586 2011-02-25 15:42:01Z marc-andre.lemburg $"
-__credits__ = u"Gustavo Niemeyer, Niels Gust\u00e4bel, Richard Townsend."
+__credits__ = "Gustavo Niemeyer, Niels Gust\u00e4bel, Richard Townsend."
 
 #---------
 # Imports
@@ -68,38 +67,38 @@
 # from tarfile import *
 __all__ = ["TarFile", "TarInfo", "is_tarfile", "TarError"]
 
-from __builtin__ import open as _open # Since 'open' is TarFile.open
+from builtins import open as _open # Since 'open' is TarFile.open
 
 #---------------------------------------------------------
 # tar constants
 #---------------------------------------------------------
-NUL = "\0"                      # the null character
+NUL = b"\0"                     # the null character
 BLOCKSIZE = 512                 # length of processing blocks
 RECORDSIZE = BLOCKSIZE * 20     # length of records
-GNU_MAGIC = "ustar  \0"         # magic gnu tar string
-POSIX_MAGIC = "ustar\x0000"     # magic posix tar string
+GNU_MAGIC = b"ustar  \0"        # magic gnu tar string
+POSIX_MAGIC = b"ustar\x0000"    # magic posix tar string
 
 LENGTH_NAME = 100               # maximum length of a filename
 LENGTH_LINK = 100               # maximum length of a linkname
 LENGTH_PREFIX = 155             # maximum length of the prefix field
 
-REGTYPE = "0"                   # regular file
-AREGTYPE = "\0"                 # regular file
-LNKTYPE = "1"                   # link (inside tarfile)
-SYMTYPE = "2"                   # symbolic link
-CHRTYPE = "3"                   # character special device
-BLKTYPE = "4"                   # block special device
-DIRTYPE = "5"                   # directory
-FIFOTYPE = "6"                  # fifo special device
-CONTTYPE = "7"                  # contiguous file
+REGTYPE = b"0"                  # regular file
+AREGTYPE = b"\0"                # regular file
+LNKTYPE = b"1"                  # link (inside tarfile)
+SYMTYPE = b"2"                  # symbolic link
+CHRTYPE = b"3"                  # character special device
+BLKTYPE = b"4"                  # block special device
+DIRTYPE = b"5"                  # directory
+FIFOTYPE = b"6"                 # fifo special device
+CONTTYPE = b"7"                 # contiguous file
 
-GNUTYPE_LONGNAME = "L"          # GNU tar longname
-GNUTYPE_LONGLINK = "K"          # GNU tar longlink
-GNUTYPE_SPARSE = "S"            # GNU tar sparse file
+GNUTYPE_LONGNAME = b"L"         # GNU tar longname
+GNUTYPE_LONGLINK = b"K"         # GNU tar longlink
+GNUTYPE_SPARSE = b"S"           # GNU tar sparse file
 
-XHDTYPE = "x"                   # POSIX.1-2001 extended header
-XGLTYPE = "g"                   # POSIX.1-2001 global header
-SOLARIS_XHDTYPE = "X"           # Solaris extended header
+XHDTYPE = b"x"                  # POSIX.1-2001 extended header
+XGLTYPE = b"g"                  # POSIX.1-2001 global header
+SOLARIS_XHDTYPE = b"X"          # Solaris extended header
 
 USTAR_FORMAT = 0                # POSIX.1-1988 (ustar) format
 GNU_FORMAT = 1                  # GNU tar format
@@ -129,7 +128,7 @@
               "uid", "gid", "uname", "gname")
 
 # Fields from a pax header that are affected by hdrcharset.
-PAX_NAME_FIELDS = set(("path", "linkpath", "uname", "gname"))
+PAX_NAME_FIELDS = {"path", "linkpath", "uname", "gname"}
 
 # Fields in a pax header that are numbers, all other fields
 # are treated as strings.
@@ -145,26 +144,26 @@
 #---------------------------------------------------------
 # Bits used in the mode field, values in octal.
 #---------------------------------------------------------
-S_IFLNK = 0120000        # symbolic link
-S_IFREG = 0100000        # regular file
-S_IFBLK = 0060000        # block device
-S_IFDIR = 0040000        # directory
-S_IFCHR = 0020000        # character device
-S_IFIFO = 0010000        # fifo
+S_IFLNK = 0o120000        # symbolic link
+S_IFREG = 0o100000        # regular file
+S_IFBLK = 0o060000        # block device
+S_IFDIR = 0o040000        # directory
+S_IFCHR = 0o020000        # character device
+S_IFIFO = 0o010000        # fifo
 
-TSUID   = 04000          # set UID on execution
-TSGID   = 02000          # set GID on execution
-TSVTX   = 01000          # reserved
+TSUID   = 0o4000          # set UID on execution
+TSGID   = 0o2000          # set GID on execution
+TSVTX   = 0o1000          # reserved
 
-TUREAD  = 0400           # read by owner
-TUWRITE = 0200           # write by owner
-TUEXEC  = 0100           # execute/search by owner
-TGREAD  = 0040           # read by group
-TGWRITE = 0020           # write by group
-TGEXEC  = 0010           # execute/search by group
-TOREAD  = 0004           # read by other
-TOWRITE = 0002           # write by other
-TOEXEC  = 0001           # execute/search by other
+TUREAD  = 0o400           # read by owner
+TUWRITE = 0o200           # write by owner
+TUEXEC  = 0o100           # execute/search by owner
+TGREAD  = 0o040           # read by group
+TGWRITE = 0o020           # write by group
+TGEXEC  = 0o010           # execute/search by group
+TOREAD  = 0o004           # read by other
+TOWRITE = 0o002           # write by other
+TOEXEC  = 0o001           # execute/search by other
 
 #---------------------------------------------------------
 # initialization
@@ -187,7 +186,7 @@
 def nts(s, encoding, errors):
     """Convert a null-terminated bytes object to a string.
     """
-    p = s.find("\0")
+    p = s.find(b"\0")
     if p != -1:
         s = s[:p]
     return s.decode(encoding, errors)
@@ -197,14 +196,14 @@
     """
     # There are two possible encodings for a number field, see
     # itn() below.
-    if s[0] != chr(0200):
+    if s[0] != chr(0o200):
         try:
             n = int(nts(s, "ascii", "strict") or "0", 8)
         except ValueError:
             raise InvalidHeaderError("invalid header")
     else:
-        n = 0L
-        for i in xrange(len(s) - 1):
+        n = 0
+        for i in range(len(s) - 1):
             n <<= 8
             n += ord(s[i + 1])
     return n
@@ -215,11 +214,11 @@
     # POSIX 1003.1-1988 requires numbers to be encoded as a string of
     # octal digits followed by a null-byte, this allows values up to
     # (8**(digits-1))-1. GNU tar allows storing numbers greater than
-    # that if necessary. A leading 0200 byte indicates this particular
+    # that if necessary. A leading 0o200 byte indicates this particular
     # encoding, the following digits-1 bytes are a big-endian
     # representation. This allows values up to (256**(digits-1))-1.
     if 0 <= n < 8 ** (digits - 1):
-        s = "%0*o" % (digits - 1, n) + NUL
+        s = bytes("%0*o" % (digits - 1, n), "ascii") + NUL
     else:
         if format != GNU_FORMAT or n >= 256 ** (digits - 1):
             raise ValueError("overflow in number field")
@@ -229,11 +228,11 @@
             # this could raise OverflowError.
             n = struct.unpack("L", struct.pack("l", n))[0]
 
-        s = ""
-        for i in xrange(digits - 1):
-            s = chr(n & 0377) + s
+        s = bytearray()
+        for i in range(digits - 1):
+            s.insert(0, n & 0o377)
             n >>= 8
-        s = chr(0200) + s
+        s.insert(0, 0o200)
     return s
 
 def calc_chksums(buf):
@@ -261,7 +260,7 @@
 
     BUFSIZE = 16 * 1024
     blocks, remainder = divmod(length, BUFSIZE)
-    for b in xrange(blocks):
+    for b in range(blocks):
         buf = src.read(BUFSIZE)
         if len(buf) < BUFSIZE:
             raise IOError("end of file reached")
@@ -353,7 +352,7 @@
 #---------------------------
 # internal stream interface
 #---------------------------
-class _LowLevelFile(object):
+class _LowLevelFile:
     """Low-level file object. Supports reading and writing.
        It is used instead of a regular file object for streaming
        access.
@@ -366,7 +365,7 @@
         }[mode]
         if hasattr(os, "O_BINARY"):
             mode |= os.O_BINARY
-        self.fd = os.open(name, mode, 0666)
+        self.fd = os.open(name, mode, 0o666)
 
     def close(self):
         os.close(self.fd)
@@ -377,7 +376,7 @@
     def write(self, s):
         os.write(self.fd, s)
 
-class _Stream(object):
+class _Stream:
     """Class that serves as an adapter between TarFile and
        a stream-like object.  The stream-like object only
        needs to have a read() or write() method and is accessed
@@ -407,8 +406,8 @@
         self.comptype = comptype
         self.fileobj  = fileobj
         self.bufsize  = bufsize
-        self.buf      = ""
-        self.pos      = 0L
+        self.buf      = b""
+        self.pos      = 0
         self.closed   = False
 
         try:
@@ -418,7 +417,7 @@
                 except ImportError:
                     raise CompressionError("zlib module is not available")
                 self.zlib = zlib
-                self.crc = zlib.crc32("")
+                self.crc = zlib.crc32(b"")
                 if mode == "r":
                     self._init_read_gz()
                 else:
@@ -430,7 +429,7 @@
                 except ImportError:
                     raise CompressionError("bz2 module is not available")
                 if mode == "r":
-                    self.dbuf = ""
+                    self.dbuf = b""
                     self.cmp = bz2.BZ2Decompressor()
                 else:
                     self.cmp = bz2.BZ2Compressor()
@@ -451,8 +450,8 @@
                                             -self.zlib.MAX_WBITS,
                                             self.zlib.DEF_MEM_LEVEL,
                                             0)
-        timestamp = struct.pack("<L", long(time.time()))
-        self.__write("\037\213\010\010%s\002\377" % timestamp)
+        timestamp = struct.pack("<L", int(time.time()))
+        self.__write(b"\037\213\010\010" + timestamp + b"\002\377")
         if self.name.endswith(".gz"):
             self.name = self.name[:-3]
         # RFC1952 says we must use ISO-8859-1 for the FNAME field.
@@ -462,7 +461,7 @@
         """Write string s to the stream.
         """
         if self.comptype == "gz":
-            self.crc = self.zlib.crc32(s, self.crc) & 0xffffffffL
+            self.crc = self.zlib.crc32(s, self.crc)
         self.pos += len(s)
         if self.comptype != "tar":
             s = self.cmp.compress(s)
@@ -489,7 +488,7 @@
 
         if self.mode == "w" and self.buf:
             self.fileobj.write(self.buf)
-            self.buf = ""
+            self.buf = b""
             if self.comptype == "gz":
                 # The native zlib crc is an unsigned 32-bit integer, but
                 # the Python wrapper implicitly casts that to a signed C
@@ -497,8 +496,8 @@
                 # while the same crc on a 64-bit box may "look positive".
                 # To avoid irksome warnings from the `struct` module, force
                 # it to look positive on all boxes.
-                self.fileobj.write(struct.pack("<L", self.crc & 0xffffffffL))
-                self.fileobj.write(struct.pack("<L", self.pos & 0xffffFFFFL))
+                self.fileobj.write(struct.pack("<L", self.crc & 0xffffffff))
+                self.fileobj.write(struct.pack("<L", self.pos & 0xffffFFFF))
 
         if not self._extfileobj:
             self.fileobj.close()
@@ -509,12 +508,12 @@
         """Initialize for reading a gzip compressed fileobj.
         """
         self.cmp = self.zlib.decompressobj(-self.zlib.MAX_WBITS)
-        self.dbuf = ""
+        self.dbuf = b""
 
         # taken from gzip.GzipFile with some alterations
-        if self.__read(2) != "\037\213":
+        if self.__read(2) != b"\037\213":
             raise ReadError("not a gzip file")
-        if self.__read(1) != "\010":
+        if self.__read(1) != b"\010":
             raise CompressionError("unsupported compression method")
 
         flag = ord(self.__read(1))
@@ -547,7 +546,7 @@
         """
         if pos - self.pos >= 0:
             blocks, remainder = divmod(pos - self.pos, self.bufsize)
-            for i in xrange(blocks):
+            for i in range(blocks):
                 self.read(self.bufsize)
             self.read(remainder)
         else:
@@ -609,7 +608,7 @@
         return buf
 # class _Stream
 
-class _StreamProxy(object):
+class _StreamProxy:
     """Small proxy class that enables transparent compression
        detection for the Stream interface (mode 'r|*').
     """
@@ -623,9 +622,9 @@
         return self.buf
 
     def getcomptype(self):
-        if self.buf.startswith("\037\213\010"):
+        if self.buf.startswith(b"\037\213\010"):
             return "gz"
-        if self.buf.startswith("BZh91"):
+        if self.buf.startswith(b"BZh91"):
             return "bz2"
         return "tar"
 
@@ -633,7 +632,7 @@
         self.fileobj.close()
 # class StreamProxy
 
-class _BZ2Proxy(object):
+class _BZ2Proxy:
     """Small proxy class that enables external file object
        support for "r:bz2" and "w:bz2" modes. This is actually
        a workaround for a limitation in bz2 module's BZ2File
@@ -655,7 +654,7 @@
         if self.mode == "r":
             self.bz2obj = bz2.BZ2Decompressor()
             self.fileobj.seek(0)
-            self.buf = ""
+            self.buf = b""
         else:
             self.bz2obj = bz2.BZ2Compressor()
 
@@ -696,7 +695,7 @@
 #------------------------
 # Extraction file object
 #------------------------
-class _FileInFile(object):
+class _FileInFile:
     """A thin wrapper around an existing file object that
        provides a part of its data as an individual file
        object.
@@ -749,7 +748,7 @@
         else:
             size = min(size, self.size - self.position)
 
-        buf = ""
+        buf = b""
         while size > 0:
             while True:
                 data, start, stop, offset = self.map[self.map_index]
@@ -771,7 +770,7 @@
 #class _FileInFile
 
 
-class ExFileObject(object):
+class ExFileObject:
     """File-like object for reading an archive member.
        Is returned by TarFile.extractfile().
     """
@@ -788,7 +787,7 @@
         self.size = tarinfo.size
 
         self.position = 0
-        self.buffer = ""
+        self.buffer = b""
 
     def readable(self):
         return True
@@ -806,11 +805,11 @@
         if self.closed:
             raise ValueError("I/O operation on closed file")
 
-        buf = ""
+        buf = b""
         if self.buffer:
             if size is None:
                 buf = self.buffer
-                self.buffer = ""
+                self.buffer = b""
             else:
                 buf = self.buffer[:size]
                 self.buffer = self.buffer[size:]
@@ -834,14 +833,14 @@
         if self.closed:
             raise ValueError("I/O operation on closed file")
 
-        pos = self.buffer.find("\n") + 1
+        pos = self.buffer.find(b"\n") + 1
         if pos == 0:
             # no newline found.
             while True:
                 buf = self.fileobj.read(self.blocksize)
                 self.buffer += buf
-                if not buf or "\n" in buf:
-                    pos = self.buffer.find("\n") + 1
+                if not buf or b"\n" in buf:
+                    pos = self.buffer.find(b"\n") + 1
                     if pos == 0:
                         # no newline found.
                         pos = len(self.buffer)
@@ -873,25 +872,25 @@
 
         return self.position
 
-    def seek(self, pos, whence=0):
+    def seek(self, pos, whence=os.SEEK_SET):
         """Seek to a position in the file.
         """
         if self.closed:
             raise ValueError("I/O operation on closed file")
 
-        if whence == 0:  # os.SEEK_SET
+        if whence == os.SEEK_SET:
             self.position = min(max(pos, 0), self.size)
-        elif whence == 1:  # os.SEEK_CUR
+        elif whence == os.SEEK_CUR:
             if pos < 0:
                 self.position = max(self.position + pos, 0)
             else:
                 self.position = min(self.position + pos, self.size)
-        elif whence == 2:  # os.SEEK_END
+        elif whence == os.SEEK_END:
             self.position = max(min(self.size + pos, self.size), 0)
         else:
             raise ValueError("Invalid argument")
 
-        self.buffer = ""
+        self.buffer = b""
         self.fileobj.seek(self.position)
 
     def close(self):
@@ -912,7 +911,7 @@
 #------------------
 # Exported Classes
 #------------------
-class TarInfo(object):
+class TarInfo:
     """Informational class which holds the details about an
        archive member given by a tar header block.
        TarInfo objects are returned by TarFile.getmember(),
@@ -931,7 +930,7 @@
            of the member.
         """
         self.name = name        # member name
-        self.mode = 0644        # file permissions
+        self.mode = 0o644       # file permissions
         self.uid = 0            # user id
         self.gid = 0            # group id
         self.size = 0           # file size
@@ -972,7 +971,7 @@
         """
         info = {
             "name":     self.name,
-            "mode":     self.mode & 07777,
+            "mode":     self.mode & 0o7777,
             "uid":      self.uid,
             "gid":      self.gid,
             "size":     self.size,
@@ -991,7 +990,7 @@
 
         return info
 
-    def tobuf(self, format=DEFAULT_FORMAT, encoding=ENCODING, errors="strict"):
+    def tobuf(self, format=DEFAULT_FORMAT, encoding=ENCODING, errors="surrogateescape"):
         """Return a tar header as a string of 512 byte blocks.
         """
         info = self.get_info()
@@ -1023,7 +1022,7 @@
         """
         info["magic"] = GNU_MAGIC
 
-        buf = ""
+        buf = b""
         if len(info["linkname"]) > LENGTH_LINK:
             buf += self._create_gnu_long_header(info["linkname"], GNUTYPE_LONGLINK, encoding, errors)
 
@@ -1070,14 +1069,14 @@
 
             val = info[name]
             if not 0 <= val < 8 ** (digits - 1) or isinstance(val, float):
-                pax_headers[name] = unicode(val)
+                pax_headers[name] = str(val)
                 info[name] = 0
 
         # Create a pax extended header if necessary.
         if pax_headers:
             buf = self._create_pax_generic_header(pax_headers, XHDTYPE, encoding)
         else:
-            buf = ""
+            buf = b""
 
         return buf + self._create_header(info, USTAR_FORMAT, "ascii", "replace")
 
@@ -1109,12 +1108,12 @@
         """
         parts = [
             stn(info.get("name", ""), 100, encoding, errors),
-            itn(info.get("mode", 0) & 07777, 8, format),
+            itn(info.get("mode", 0) & 0o7777, 8, format),
             itn(info.get("uid", 0), 8, format),
             itn(info.get("gid", 0), 8, format),
             itn(info.get("size", 0), 12, format),
             itn(info.get("mtime", 0), 12, format),
-            "        ", # checksum field
+            b"        ", # checksum field
             info.get("type", REGTYPE),
             stn(info.get("linkname", ""), 100, encoding, errors),
             info.get("magic", POSIX_MAGIC),
@@ -1125,9 +1124,9 @@
             stn(info.get("prefix", ""), 155, encoding, errors)
         ]
 
-        buf = struct.pack("%ds" % BLOCKSIZE, "".join(parts))
+        buf = struct.pack("%ds" % BLOCKSIZE, b"".join(parts))
         chksum = calc_chksums(buf[-BLOCKSIZE:])[0]
-        buf = buf[:-364] + "%06o\0" % chksum + buf[-357:]
+        buf = buf[:-364] + bytes("%06o\0" % chksum, "ascii") + buf[-357:]
         return buf
 
     @staticmethod
@@ -1161,7 +1160,7 @@
     def _create_pax_generic_header(cls, pax_headers, type, encoding):
         """Return a POSIX.1-2008 extended or global header sequence
            that contains a list of keyword, value pairs. The values
-           must be unicode objects.
+           must be strings.
         """
         # Check if one of the fields contains surrogate characters and thereby
         # forces hdrcharset=BINARY, see _proc_pax() for more information.
@@ -1173,10 +1172,10 @@
                 binary = True
                 break
 
-        records = ""
+        records = b""
         if binary:
             # Put the hdrcharset field at the beginning of the header.
-            records += "21 hdrcharset=BINARY\n"
+            records += b"21 hdrcharset=BINARY\n"
 
         for keyword, value in pax_headers.items():
             keyword = keyword.encode("utf8")
@@ -1194,7 +1193,7 @@
                 if n == p:
                     break
                 p = n
-            records += bytes(str(p), "ascii") + " " + keyword + "=" + value + "\n"
+            records += bytes(str(p), "ascii") + b" " + keyword + b"=" + value + b"\n"
 
         # We use a hardcoded "././@PaxHeader" name like star does
         # instead of the one that POSIX recommends.
@@ -1355,7 +1354,7 @@
         while isextended:
             buf = tarfile.fileobj.read(BLOCKSIZE)
             pos = 0
-            for i in xrange(21):
+            for i in range(21):
                 try:
                     offset = nti(buf[pos:pos + 12])
                     numbytes = nti(buf[pos + 12:pos + 24])
@@ -1392,7 +1391,7 @@
         # these fields are UTF-8 encoded but since POSIX.1-2008 tar
         # implementations are allowed to store them as raw binary strings if
         # the translation to UTF-8 fails.
-        match = re.search(r"\d+ hdrcharset=([^\n]+)\n", buf)
+        match = re.search(br"\d+ hdrcharset=([^\n]+)\n", buf)
         if match is not None:
             pax_headers["hdrcharset"] = match.group(1).decode("utf8")
 
@@ -1409,7 +1408,7 @@
         # "%d %s=%s\n" % (length, keyword, value). length is the size
         # of the complete record including the length field itself and
         # the newline. keyword and value are both UTF-8 encoded strings.
-        regex = re.compile(r"(\d+) ([^=]+)=")
+        regex = re.compile(br"(\d+) ([^=]+)=")
         pos = 0
         while True:
             match = regex.match(buf, pos)
@@ -1478,10 +1477,10 @@
         """Process a GNU tar extended sparse header, version 0.0.
         """
         offsets = []
-        for match in re.finditer(r"\d+ GNU.sparse.offset=(\d+)\n", buf):
+        for match in re.finditer(br"\d+ GNU.sparse.offset=(\d+)\n", buf):
             offsets.append(int(match.group(1)))
         numbytes = []
-        for match in re.finditer(r"\d+ GNU.sparse.numbytes=(\d+)\n", buf):
+        for match in re.finditer(br"\d+ GNU.sparse.numbytes=(\d+)\n", buf):
             numbytes.append(int(match.group(1)))
         next.sparse = list(zip(offsets, numbytes))
 
@@ -1497,12 +1496,12 @@
         fields = None
         sparse = []
         buf = tarfile.fileobj.read(BLOCKSIZE)
-        fields, buf = buf.split("\n", 1)
+        fields, buf = buf.split(b"\n", 1)
         fields = int(fields)
         while len(sparse) < fields * 2:
-            if "\n" not in buf:
+            if b"\n" not in buf:
                 buf += tarfile.fileobj.read(BLOCKSIZE)
-            number, buf = buf.split("\n", 1)
+            number, buf = buf.split(b"\n", 1)
             sparse.append(int(number))
         next.offset_data = tarfile.fileobj.tell()
         next.sparse = list(zip(sparse[::2], sparse[1::2]))
@@ -1569,7 +1568,7 @@
         return self.type in (CHRTYPE, BLKTYPE, FIFOTYPE)
 # class TarInfo
 
-class TarFile(object):
+class TarFile:
     """The TarFile Class provides an interface to tar archives.
     """
 
@@ -1597,7 +1596,7 @@
 
     def __init__(self, name=None, mode="r", fileobj=None, format=None,
             tarinfo=None, dereference=None, ignore_zeros=None, encoding=None,
-            errors="strict", pax_headers=None, debug=None, errorlevel=None):
+            errors="surrogateescape", pax_headers=None, debug=None, errorlevel=None):
         """Open an (uncompressed) tar archive `name'. `mode' is either 'r' to
            read from an existing archive, 'a' to append data to an existing
            file or 'w' to create a new file overwriting an existing one. `mode'
@@ -1624,10 +1623,7 @@
             if hasattr(fileobj, "mode"):
                 self._mode = fileobj.mode
             self._extfileobj = True
-        if name:
-            self.name = os.path.abspath(name)
-        else:
-            self.name = None
+        self.name = os.path.abspath(name) if name else None
         self.fileobj = fileobj
 
         # Init attributes.
@@ -1678,7 +1674,7 @@
                     except EOFHeaderError:
                         self.fileobj.seek(self.offset)
                         break
-                    except HeaderError, e:
+                    except HeaderError as e:
                         raise ReadError(str(e))
 
             if self.mode in "aw":
@@ -1740,7 +1736,7 @@
                     saved_pos = fileobj.tell()
                 try:
                     return func(name, "r", fileobj, **kwargs)
-                except (ReadError, CompressionError), e:
+                except (ReadError, CompressionError) as e:
                     if fileobj is not None:
                         fileobj.seek(saved_pos)
                     continue
@@ -2010,31 +2006,27 @@
 
         for tarinfo in self:
             if verbose:
-                print filemode(tarinfo.mode),
-                print "%s/%s" % (tarinfo.uname or tarinfo.uid,
-                                 tarinfo.gname or tarinfo.gid),
+                print(filemode(tarinfo.mode), end=' ')
+                print("%s/%s" % (tarinfo.uname or tarinfo.uid,
+                                 tarinfo.gname or tarinfo.gid), end=' ')
                 if tarinfo.ischr() or tarinfo.isblk():
-                    print "%10s" % ("%d,%d" \
-                                    % (tarinfo.devmajor, tarinfo.devminor)),
+                    print("%10s" % ("%d,%d" \
+                                    % (tarinfo.devmajor, tarinfo.devminor)), end=' ')
                 else:
-                    print "%10d" % tarinfo.size,
-                print "%d-%02d-%02d %02d:%02d:%02d" \
-                      % time.localtime(tarinfo.mtime)[:6],
+                    print("%10d" % tarinfo.size, end=' ')
+                print("%d-%02d-%02d %02d:%02d:%02d" \
+                      % time.localtime(tarinfo.mtime)[:6], end=' ')
 
-            if tarinfo.isdir():
-                sep = "/"
-            else:
-                sep = ""
-            print tarinfo.name + sep,
+            print(tarinfo.name + ("/" if tarinfo.isdir() else ""), end=' ')
 
             if verbose:
                 if tarinfo.issym():
-                    print "->", tarinfo.linkname,
+                    print("->", tarinfo.linkname, end=' ')
                 if tarinfo.islnk():
-                    print "link to", tarinfo.linkname,
-            print
+                    print("link to", tarinfo.linkname, end=' ')
+            print()
 
-    def add(self, name, arcname=None, recursive=True, exclude=None, filter=None):
+    def add(self, name, arcname=None, recursive=True, exclude=None, *, filter=None):
         """Add the file `name' to the archive. `name' may be any type of file
            (directory, fifo, symbolic link, etc.). If given, `arcname'
            specifies an alternative name for the file in the archive.
@@ -2139,7 +2131,7 @@
                 # Extract directories with a safe mode.
                 directories.append(tarinfo)
                 tarinfo = copy.copy(tarinfo)
-                tarinfo.mode = 0700
+                tarinfo.mode = 0o700
             # Do not set_attrs directories, as we will do that further down
             self.extract(tarinfo, path, set_attrs=not tarinfo.isdir())
 
@@ -2154,7 +2146,7 @@
                 self.chown(tarinfo, dirpath)
                 self.utime(tarinfo, dirpath)
                 self.chmod(tarinfo, dirpath)
-            except ExtractError, e:
+            except ExtractError as e:
                 if self.errorlevel > 1:
                     raise
                 else:
@@ -2169,7 +2161,7 @@
         """
         self._check("r")
 
-        if isinstance(member, basestring):
+        if isinstance(member, str):
             tarinfo = self.getmember(member)
         else:
             tarinfo = member
@@ -2181,7 +2173,7 @@
         try:
             self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
                                  set_attrs=set_attrs)
-        except EnvironmentError, e:
+        except EnvironmentError as e:
             if self.errorlevel > 0:
                 raise
             else:
@@ -2189,7 +2181,7 @@
                     self._dbg(1, "tarfile: %s" % e.strerror)
                 else:
                     self._dbg(1, "tarfile: %s %r" % (e.strerror, e.filename))
-        except ExtractError, e:
+        except ExtractError as e:
             if self.errorlevel > 1:
                 raise
             else:
@@ -2206,7 +2198,7 @@
         """
         self._check("r")
 
-        if isinstance(member, basestring):
+        if isinstance(member, str):
             tarinfo = self.getmember(member)
         else:
             tarinfo = member
@@ -2287,8 +2279,8 @@
         try:
             # Use a safe mode for the directory, the real mode is set
             # later in _extract_member().
-            os.mkdir(targetpath, 0700)
-        except EnvironmentError, e:
+            os.mkdir(targetpath, 0o700)
+        except EnvironmentError as e:
             if e.errno != errno.EEXIST:
                 raise
 
@@ -2387,7 +2379,7 @@
                 else:
                     if sys.platform != "os2emx":
                         os.chown(targetpath, u, g)
-            except EnvironmentError, e:
+            except EnvironmentError as e:
                 raise ExtractError("could not change owner")
 
     def chmod(self, tarinfo, targetpath):
@@ -2396,7 +2388,7 @@
         if hasattr(os, 'chmod'):
             try:
                 os.chmod(targetpath, tarinfo.mode)
-            except EnvironmentError, e:
+            except EnvironmentError as e:
                 raise ExtractError("could not change mode")
 
     def utime(self, tarinfo, targetpath):
@@ -2406,7 +2398,7 @@
             return
         try:
             os.utime(targetpath, (tarinfo.mtime, tarinfo.mtime))
-        except EnvironmentError, e:
+        except EnvironmentError as e:
             raise ExtractError("could not change modification time")
 
     #--------------------------------------------------------------------------
@@ -2427,12 +2419,12 @@
         while True:
             try:
                 tarinfo = self.tarinfo.fromtarfile(self)
-            except EOFHeaderError, e:
+            except EOFHeaderError as e:
                 if self.ignore_zeros:
                     self._dbg(2, "0x%X: %s" % (self.offset, e))
                     self.offset += BLOCKSIZE
                     continue
-            except InvalidHeaderError, e:
+            except InvalidHeaderError as e:
                 if self.ignore_zeros:
                     self._dbg(2, "0x%X: %s" % (self.offset, e))
                     self.offset += BLOCKSIZE
@@ -2442,10 +2434,10 @@
             except EmptyHeaderError:
                 if self.offset == 0:
                     raise ReadError("empty file")
-            except TruncatedHeaderError, e:
+            except TruncatedHeaderError as e:
                 if self.offset == 0:
                     raise ReadError(str(e))
-            except SubsequentHeaderError, e:
+            except SubsequentHeaderError as e:
                 raise ReadError(str(e))
             break
 
@@ -2532,7 +2524,7 @@
         """Write debugging output to sys.stderr.
         """
         if level <= self.debug:
-            print >> sys.stderr, msg
+            print(msg, file=sys.stderr)
 
     def __enter__(self):
         self._check()
@@ -2549,7 +2541,7 @@
             self.closed = True
 # class TarFile
 
-class TarIter(object):
+class TarIter:
     """Iterator Class.
 
        for tarinfo in TarFile(...):
@@ -2565,7 +2557,7 @@
         """Return iterator object.
         """
         return self
-    def next(self):
+    def __next__(self):
         """Return the next item using TarFile's next() method.
            When all members have been read, set TarFile as _loaded.
         """
diff --git a/distutils2/_backport/tests/test_shutil.py b/distutils2/_backport/tests/test_shutil.py
--- a/distutils2/_backport/tests/test_shutil.py
+++ b/distutils2/_backport/tests/test_shutil.py
@@ -2,11 +2,11 @@
 import sys
 import stat
 import tempfile
+from io import StringIO
 from os.path import splitdrive
-from StringIO import StringIO
+from functools import wraps
+from distutils.spawn import find_executable, spawn
 
-from distutils.spawn import find_executable, spawn
-from distutils2.compat import wraps
 from distutils2._backport import shutil, tarfile
 from distutils2._backport.shutil import (
     _make_tarball, _make_zipfile, make_archive, unpack_archive,
@@ -15,7 +15,7 @@
     Error, RegistryError)
 
 from distutils2.tests import unittest, support
-from test.test_support import TESTFN
+from test.support import TESTFN
 
 
 try:
@@ -370,7 +370,7 @@
                 os.mkfifo(pipe)
                 try:
                     shutil.copytree(TESTFN, TESTFN2)
-                except shutil.Error, e:
+                except shutil.Error as e:
                     errors = e.args[0]
                     self.assertEqual(len(errors), 1)
                     src, dst, error_msg = errors[0]
@@ -756,7 +756,7 @@
         self.dst_file = os.path.join(self.dst_dir, filename)
         f = open(self.src_file, "wb")
         try:
-            f.write("spam")
+            f.write(b"spam")
         finally:
             f.close()
 
@@ -877,7 +877,7 @@
 
     _delete = False
 
-    class Faux(object):
+    class Faux:
         _entered = False
         _exited_with = None
         _raised = False
@@ -917,7 +917,6 @@
 
         self.assertRaises(IOError, shutil.copyfile, 'srcfile', 'destfile')
 
-    @unittest.skip("can't use the with statement and support 2.4")
     def test_w_dest_open_fails(self):
 
         srcfile = self.Faux()
@@ -937,7 +936,6 @@
         self.assertEqual(srcfile._exited_with[1].args,
                          ('Cannot open "destfile"',))
 
-    @unittest.skip("can't use the with statement and support 2.4")
     def test_w_dest_close_fails(self):
 
         srcfile = self.Faux()
@@ -960,7 +958,6 @@
         self.assertEqual(srcfile._exited_with[1].args,
                          ('Cannot close',))
 
-    @unittest.skip("can't use the with statement and support 2.4")
     def test_w_source_close_fails(self):
 
         srcfile = self.Faux(True)
diff --git a/distutils2/_backport/tests/test_sysconfig.py b/distutils2/_backport/tests/test_sysconfig.py
--- a/distutils2/_backport/tests/test_sysconfig.py
+++ b/distutils2/_backport/tests/test_sysconfig.py
@@ -3,8 +3,8 @@
 import subprocess
 import shutil
 from copy import copy
-from ConfigParser import RawConfigParser
-from StringIO import StringIO
+from configparser import RawConfigParser
+from io import StringIO
 
 from distutils2._backport import sysconfig
 from distutils2._backport.sysconfig import (
@@ -15,7 +15,7 @@
 from distutils2.tests import unittest
 from distutils2.tests.support import skip_unless_symlink
 
-from test.test_support import TESTFN, unlink
+from test.support import TESTFN, unlink
 
 
 class TestSysConfig(unittest.TestCase):
@@ -141,14 +141,14 @@
         get_config_vars()['CFLAGS'] = ('-fno-strict-aliasing -DNDEBUG -g '
                                        '-fwrapv -O3 -Wall -Wstrict-prototypes')
 
-        maxint = sys.maxint
+        maxint = sys.maxsize
         try:
-            sys.maxint = 2147483647
+            sys.maxsize = 2147483647
             self.assertEqual(get_platform(), 'macosx-10.3-ppc')
-            sys.maxint = 9223372036854775807
+            sys.maxsize = 9223372036854775807
             self.assertEqual(get_platform(), 'macosx-10.3-ppc64')
         finally:
-            sys.maxint = maxint
+            sys.maxsize = maxint
 
         self._set_uname(('Darwin', 'macziade', '8.11.1',
                    ('Darwin Kernel Version 8.11.1: '
@@ -159,14 +159,14 @@
 
         get_config_vars()['CFLAGS'] = ('-fno-strict-aliasing -DNDEBUG -g '
                                        '-fwrapv -O3 -Wall -Wstrict-prototypes')
-        maxint = sys.maxint
+        maxint = sys.maxsize
         try:
-            sys.maxint = 2147483647
+            sys.maxsize = 2147483647
             self.assertEqual(get_platform(), 'macosx-10.3-i386')
-            sys.maxint = 9223372036854775807
+            sys.maxsize = 9223372036854775807
             self.assertEqual(get_platform(), 'macosx-10.3-x86_64')
         finally:
-            sys.maxint = maxint
+            sys.maxsize = maxint
 
         # macbook with fat binaries (fat, universal or fat64)
         get_config_vars()['MACOSX_DEPLOYMENT_TARGET'] = '10.4'
@@ -244,7 +244,7 @@
         def get(python):
             cmd = [python, '-c',
                    'from distutils2._backport import sysconfig; '
-                   'print sysconfig.get_platform()']
+                   'print(sysconfig.get_platform())']
             p = subprocess.Popen(cmd, stdout=subprocess.PIPE, env=os.environ)
             return p.communicate()
         real = os.path.realpath(sys.executable)
@@ -255,7 +255,6 @@
         finally:
             unlink(link)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_user_similar(self):
         # Issue #8759: make sure the posix scheme for the users
         # is similar to the global posix_prefix one
@@ -290,18 +289,15 @@
         if 'MACOSX_DEPLOYMENT_TARGET' in env:
             del env['MACOSX_DEPLOYMENT_TARGET']
 
-        devnull_fp = open('/dev/null', 'w')
-        try:
+        with open('/dev/null', 'w') as devnull_fp:
             p = subprocess.Popen([
                     sys.executable, '-c',
                     'from distutils2._backport import sysconfig; '
-                    'print sysconfig.get_platform()',
+                    'print(sysconfig.get_platform())',
                 ],
                 stdout=subprocess.PIPE,
                 stderr=devnull_fp,
                 env=env)
-        finally:
-            devnull_fp.close()
         test_platform = p.communicate()[0].strip()
         test_platform = test_platform.decode('utf-8')
         status = p.wait()
@@ -314,12 +310,11 @@
         env = os.environ.copy()
         env['MACOSX_DEPLOYMENT_TARGET'] = '10.1'
 
-        dev_null = open('/dev/null')
-        try:
+        with open('/dev/null') as dev_null:
             p = subprocess.Popen([
                     sys.executable, '-c',
                     'from distutils2._backport import sysconfig; '
-                    'print sysconfig.get_platform()',
+                    'print(sysconfig.get_platform())',
                 ],
                 stdout=subprocess.PIPE,
                 stderr=dev_null,
@@ -330,8 +325,6 @@
 
             self.assertEqual(status, 0)
             self.assertEqual(my_platform, test_platform)
-        finally:
-            dev_null.close()
 
 
 class MakefileTests(unittest.TestCase):
@@ -344,15 +337,12 @@
 
     def test_parse_makefile(self):
         self.addCleanup(unlink, TESTFN)
-        makefile = open(TESTFN, "w")
-        try:
-            print >> makefile, "var1=a$(VAR2)"
-            print >> makefile, "VAR2=b$(var3)"
-            print >> makefile, "var3=42"
-            print >> makefile, "var4=$/invalid"
-            print >> makefile, "var5=dollar$$5"
-        finally:
-            makefile.close()
+        with open(TESTFN, "w") as makefile:
+            print("var1=a$(VAR2)", file=makefile)
+            print("VAR2=b$(var3)", file=makefile)
+            print("var3=42", file=makefile)
+            print("var4=$/invalid", file=makefile)
+            print("var5=dollar$$5", file=makefile)
         vars = sysconfig._parse_makefile(TESTFN)
         self.assertEqual(vars, {
             'var1': 'ab42',
diff --git a/distutils2/command/bdist_msi.py b/distutils2/command/bdist_msi.py
--- a/distutils2/command/bdist_msi.py
+++ b/distutils2/command/bdist_msi.py
@@ -394,8 +394,7 @@
         #     entries for each version as the above code does
         if self.pre_install_script:
             scriptfn = os.path.join(self.bdist_dir, "preinstall.bat")
-            f = open(scriptfn, "w")
-            try:
+            with open(scriptfn, "w") as f:
                 # The batch file will be executed with [PYTHON], so that %1
                 # is the path to the Python interpreter; %0 will be the path
                 # of the batch file.
@@ -405,13 +404,8 @@
                 # """
                 # <actual script>
                 f.write('rem ="""\n%1 %0\nexit\n"""\n')
-                fp = open(self.pre_install_script)
-                try:
+                with open(self.pre_install_script) as fp:
                     f.write(fp.read())
-                finally:
-                    fp.close()
-            finally:
-                f.close()
             add_data(self.db, "Binary",
                      [("PreInstall", msilib.Binary(scriptfn)),
                      ])
diff --git a/distutils2/command/bdist_wininst.py b/distutils2/command/bdist_wininst.py
--- a/distutils2/command/bdist_wininst.py
+++ b/distutils2/command/bdist_wininst.py
@@ -248,40 +248,33 @@
         logger.info("creating %s", installer_name)
 
         if bitmap:
-            fp = open(bitmap, "rb")
-            try:
+            with open(bitmap, "rb") as fp:
                 bitmapdata = fp.read()
-            finally:
-                fp.close()
             bitmaplen = len(bitmapdata)
         else:
             bitmaplen = 0
 
-        file = open(installer_name, "wb")
-        try:
+        with open(installer_name, "wb") as file:
             file.write(self.get_exe_bytes())
             if bitmap:
                 file.write(bitmapdata)
 
             # Convert cfgdata from unicode to ascii, mbcs encoded
-            if isinstance(cfgdata, unicode):
+            if isinstance(cfgdata, str):
                 cfgdata = cfgdata.encode("mbcs")
 
             # Append the pre-install script
-            cfgdata = cfgdata + "\0"
+            cfgdata = cfgdata + b"\0"
             if self.pre_install_script:
                 # We need to normalize newlines, so we open in text mode and
                 # convert back to bytes. "latin-1" simply avoids any possible
                 # failures.
-                fp = codecs.open(self.pre_install_script, encoding="latin-1")
-                try:
+                with open(self.pre_install_script, encoding="latin-1") as fp:
                     script_data = fp.read().encode("latin-1")
-                finally:
-                    fp.close()
-                cfgdata = cfgdata + script_data + "\n\0"
+                cfgdata = cfgdata + script_data + b"\n\0"
             else:
                 # empty pre-install script
-                cfgdata = cfgdata + "\0"
+                cfgdata = cfgdata + b"\0"
             file.write(cfgdata)
 
             # The 'magic number' 0x1234567B is used to make sure that the
@@ -295,13 +288,8 @@
                                  bitmaplen,        # number of bytes in bitmap
                                  )
             file.write(header)
-            fp = open(arcname, "rb")
-            try:
+            with open(arcname, "rb") as fp:
                 file.write(fp.read())
-            finally:
-                fp.close()
-        finally:
-            file.close()
 
     def get_installer_filename(self, fullname):
         # Factored out to allow overriding in subclasses
@@ -356,9 +344,5 @@
             sfix = ''
 
         filename = os.path.join(directory, "wininst-%.1f%s.exe" % (bv, sfix))
-        fp = open(filename, "rb")
-        try:
-            content = fp.read()
-        finally:
-            fp.close()
-        return content
+        with open(filename, "rb") as fp:
+            return fp.read()
diff --git a/distutils2/command/build_clib.py b/distutils2/command/build_clib.py
--- a/distutils2/command/build_clib.py
+++ b/distutils2/command/build_clib.py
@@ -82,7 +82,7 @@
 
         if self.include_dirs is None:
             self.include_dirs = self.distribution.include_dirs or []
-        if isinstance(self.include_dirs, basestring):
+        if isinstance(self.include_dirs, str):
             self.include_dirs = self.include_dirs.split(os.pathsep)
 
         # XXX same as for build_ext -- what about 'self.define' and
@@ -130,7 +130,7 @@
 
             name, build_info = lib
 
-            if not isinstance(name, basestring):
+            if not isinstance(name, str):
                 raise PackagingSetupError("first element of each tuple in 'libraries' " + \
                       "must be a string (the library name)")
             if '/' in name or (os.sep != '/' and os.sep in name):
diff --git a/distutils2/command/build_ext.py b/distutils2/command/build_ext.py
--- a/distutils2/command/build_ext.py
+++ b/distutils2/command/build_ext.py
@@ -3,6 +3,7 @@
 import os
 import re
 import sys
+import site
 import logging
 
 from distutils2._backport import sysconfig
@@ -15,12 +16,6 @@
 from distutils2.compiler.extension import Extension
 from distutils2 import logger
 
-import site
-if sys.version_info[:2] >= (2, 6):
-    HAS_USER_SITE = True
-else:
-    HAS_USER_SITE = False
-
 if os.name == 'nt':
     from distutils2.compiler.msvccompiler import get_build_version
     MSVC_VERSION = int(get_build_version())
@@ -65,6 +60,8 @@
         ('inplace', 'i',
          "ignore build-lib and put compiled extensions into the source " +
          "directory alongside your pure Python modules"),
+        ('user', None,
+         "add user include, library and rpath"),
         ('include-dirs=', 'I',
          "list of directories to search for header files" + sep_by),
         ('define=', 'D',
@@ -91,12 +88,8 @@
          "path to the SWIG executable"),
         ]
 
-    boolean_options = ['inplace', 'debug', 'force']
+    boolean_options = ['inplace', 'debug', 'force', 'user']
 
-    if HAS_USER_SITE:
-        user_options.append(('user', None,
-                             "add user include, library and rpath"))
-        boolean_options.append('user')
 
     help_options = [
         ('help-compiler', None,
@@ -123,8 +116,7 @@
         self.compiler = None
         self.swig = None
         self.swig_opts = None
-        if HAS_USER_SITE:
-            self.user = None
+        self.user = None
 
     def finalize_options(self):
         self.set_undefined_options('build',
@@ -159,7 +151,7 @@
         plat_py_include = sysconfig.get_path('platinclude')
         if self.include_dirs is None:
             self.include_dirs = self.distribution.include_dirs or []
-        if isinstance(self.include_dirs, basestring):
+        if isinstance(self.include_dirs, str):
             self.include_dirs = self.include_dirs.split(os.pathsep)
 
         # Put the Python "system" include dir at the end, so that
@@ -168,7 +160,7 @@
         if plat_py_include != py_include:
             self.include_dirs.append(plat_py_include)
 
-        if isinstance(self.libraries, basestring):
+        if isinstance(self.libraries, str):
             self.libraries = [self.libraries]
 
         # Life is easier if we're not forever checking for None, so
@@ -177,12 +169,12 @@
             self.libraries = []
         if self.library_dirs is None:
             self.library_dirs = []
-        elif isinstance(self.library_dirs, basestring):
+        elif isinstance(self.library_dirs, str):
             self.library_dirs = self.library_dirs.split(os.pathsep)
 
         if self.rpath is None:
             self.rpath = []
-        elif isinstance(self.rpath, basestring):
+        elif isinstance(self.rpath, str):
             self.rpath = self.rpath.split(os.pathsep)
 
         # for extensions under windows use different directories
@@ -243,8 +235,7 @@
         # for extensions under Linux or Solaris with a shared Python library,
         # Python's library directory must be appended to library_dirs
         sysconfig.get_config_var('Py_ENABLE_SHARED')
-        if ((sys.platform.startswith('linux') or sys.platform.startswith('gnu')
-             or sys.platform.startswith('sunos'))
+        if (sys.platform.startswith(('linux', 'gnu', 'sunos'))
             and sysconfig.get_config_var('Py_ENABLE_SHARED')):
             if sys.executable.startswith(os.path.join(sys.exec_prefix, "bin")):
                 # building third party extensions
@@ -274,7 +265,7 @@
             self.swig_opts = self.swig_opts.split(' ')
 
         # Finally add the user include and library directories if requested
-        if HAS_USER_SITE and self.user:
+        if self.user:
             user_include = os.path.join(site.USER_BASE, "include")
             user_lib = os.path.join(site.USER_BASE, "lib")
             if os.path.isdir(user_include):
@@ -362,7 +353,7 @@
         for ext in self.extensions:
             try:
                 self.build_extension(ext)
-            except (CCompilerError, PackagingError, CompileError), e:
+            except (CCompilerError, PackagingError, CompileError) as e:
                 if not ext.optional:
                     raise
                 logger.warning('%s: building extension %r failed: %s',
@@ -650,7 +641,7 @@
 
         else:
             if sysconfig.get_config_var('Py_ENABLE_SHARED'):
-                template = 'python%d.%d'
+                template = 'python%d.%d' + getattr(sys, 'abiflags', '')
                 pythonlib = template % sys.version_info[:2]
                 return ext.libraries + [pythonlib]
             else:
diff --git a/distutils2/command/build_py.py b/distutils2/command/build_py.py
--- a/distutils2/command/build_py.py
+++ b/distutils2/command/build_py.py
@@ -342,7 +342,7 @@
         return outputs
 
     def build_module(self, module, module_file, package):
-        if isinstance(package, basestring):
+        if isinstance(package, str):
             package = package.split('.')
         elif not isinstance(package, (list, tuple)):
             raise TypeError(
@@ -388,7 +388,7 @@
                 self.build_module(module, module_file, package)
 
     def byte_compile(self, files):
-        if getattr(sys, 'dont_write_bytecode', False):
+        if sys.dont_write_bytecode:
             logger.warning('%s: byte-compiling is disabled, skipping.',
                            self.get_command_name())
             return
diff --git a/distutils2/command/build_scripts.py b/distutils2/command/build_scripts.py
--- a/distutils2/command/build_scripts.py
+++ b/distutils2/command/build_scripts.py
@@ -2,17 +2,18 @@
 
 import os
 import re
+from tokenize import detect_encoding
 
 from distutils2.command.cmd import Command
 from distutils2.util import convert_path, newer
 from distutils2 import logger
 from distutils2.compat import Mixin2to3
-from distutils2.compat import detect_encoding, fsencode
+from distutils2.compat import fsencode
 from distutils2._backport import sysconfig
 
 
 # check if Python is called on the first line with this expression
-first_line_re = re.compile('^#!.*python[0-9.]*([ \t].*)?$')
+first_line_re = re.compile(b'^#!.*python[0-9.]*([ \t].*)?$')
 
 class build_scripts(Command, Mixin2to3):
 
@@ -94,7 +95,7 @@
                 match = first_line_re.match(first_line)
                 if match:
                     adjust = True
-                    post_interp = match.group(1) or ''
+                    post_interp = match.group(1) or b''
 
             if adjust:
                 logger.info("copying and adjusting %s -> %s", script,
@@ -108,7 +109,7 @@
                            "python%s%s" % (sysconfig.get_config_var("VERSION"),
                                            sysconfig.get_config_var("EXE")))
                     executable = fsencode(executable)
-                    shebang = "#!" + executable + post_interp + "\n"
+                    shebang = b"#!" + executable + post_interp + b"\n"
                     # Python parser starts to read a script using UTF-8 until
                     # it gets a #coding:xxx cookie. The shebang has to be the
                     # first line of a file, the #coding:xxx cookie cannot be
@@ -130,12 +131,9 @@
                             "The shebang (%r) is not decodable "
                             "from the script encoding (%s)" % (
                                 shebang, encoding))
-                    outf = open(outfile, "wb")
-                    try:
+                    with open(outfile, "wb") as outf:
                         outf.write(shebang)
                         outf.writelines(f.readlines())
-                    finally:
-                        outf.close()
                 if f:
                     f.close()
             else:
@@ -148,8 +146,8 @@
                 if self.dry_run:
                     logger.info("changing mode of %s", file)
                 else:
-                    oldmode = os.stat(file).st_mode & 07777
-                    newmode = (oldmode | 0555) & 07777
+                    oldmode = os.stat(file).st_mode & 0o7777
+                    newmode = (oldmode | 0o555) & 0o7777
                     if newmode != oldmode:
                         logger.info("changing mode of %s from %o to %o",
                                  file, oldmode, newmode)
diff --git a/distutils2/command/cmd.py b/distutils2/command/cmd.py
--- a/distutils2/command/cmd.py
+++ b/distutils2/command/cmd.py
@@ -8,7 +8,7 @@
 from distutils2._backport.shutil import copyfile, move, make_archive
 
 
-class Command(object):
+class Command:
     """Abstract base class for defining command classes, the "worker bees"
     of the Packaging.  A useful analogy for command classes is to think of
     them as subroutines with local variables called "options".  The options
@@ -216,7 +216,7 @@
         if val is None:
             setattr(self, option, default)
             return default
-        elif not isinstance(val, basestring):
+        elif not isinstance(val, str):
             raise PackagingOptionError("'%s' must be a %s (got `%s`)" %
                                        (option, what, val))
         return val
@@ -236,14 +236,14 @@
         val = getattr(self, option)
         if val is None:
             return
-        elif isinstance(val, basestring):
+        elif isinstance(val, str):
             setattr(self, option, re.split(r',\s*|\s+', val))
         else:
             if isinstance(val, list):
                 # checks if all elements are str
                 ok = True
                 for element in val:
-                    if not isinstance(element, basestring):
+                    if not isinstance(element, str):
                         ok = False
                         break
             else:
@@ -351,7 +351,7 @@
     def execute(self, func, args, msg=None, level=1):
         util.execute(func, args, msg, dry_run=self.dry_run)
 
-    def mkpath(self, name, mode=00777, dry_run=None, verbose=0):
+    def mkpath(self, name, mode=0o777, dry_run=None, verbose=0):
         if dry_run is None:
             dry_run = self.dry_run
         name = os.path.normpath(name)
@@ -421,7 +421,7 @@
             skip_msg = "skipping %s (inputs unchanged)" % outfile
 
         # Allow 'infiles' to be a single string
-        if isinstance(infiles, basestring):
+        if isinstance(infiles, str):
             infiles = (infiles,)
         elif not isinstance(infiles, (list, tuple)):
             raise TypeError(
diff --git a/distutils2/command/config.py b/distutils2/command/config.py
--- a/distutils2/command/config.py
+++ b/distutils2/command/config.py
@@ -67,17 +67,17 @@
     def finalize_options(self):
         if self.include_dirs is None:
             self.include_dirs = self.distribution.include_dirs or []
-        elif isinstance(self.include_dirs, basestring):
+        elif isinstance(self.include_dirs, str):
             self.include_dirs = self.include_dirs.split(os.pathsep)
 
         if self.libraries is None:
             self.libraries = []
-        elif isinstance(self.libraries, basestring):
+        elif isinstance(self.libraries, str):
             self.libraries = [self.libraries]
 
         if self.library_dirs is None:
             self.library_dirs = []
-        elif isinstance(self.library_dirs, basestring):
+        elif isinstance(self.library_dirs, str):
             self.library_dirs = self.library_dirs.split(os.pathsep)
 
     def run(self):
@@ -110,8 +110,7 @@
 
     def _gen_temp_sourcefile(self, body, headers, lang):
         filename = "_configtest" + LANG_EXT[lang]
-        file = open(filename, "w")
-        try:
+        with open(filename, "w") as file:
             if headers:
                 for header in headers:
                     file.write("#include <%s>\n" % header)
@@ -119,8 +118,6 @@
             file.write(body)
             if body[-1] != "\n":
                 file.write("\n")
-        finally:
-            file.close()
         return filename
 
     def _preprocess(self, body, headers, include_dirs, lang):
@@ -206,11 +203,10 @@
         self._check_compiler()
         src, out = self._preprocess(body, headers, include_dirs, lang)
 
-        if isinstance(pattern, basestring):
+        if isinstance(pattern, str):
             pattern = re.compile(pattern)
 
-        file = open(out)
-        try:
+        with open(out) as file:
             match = False
             while True:
                 line = file.readline()
@@ -219,8 +215,6 @@
                 if pattern.search(line):
                     match = True
                     break
-        finally:
-            file.close()
 
         self._clean()
         return match
@@ -351,8 +345,5 @@
         logger.info(filename)
     else:
         logger.info(head)
-    file = open(filename)
-    try:
+    with open(filename) as file:
         logger.info(file.read())
-    finally:
-        file.close()
diff --git a/distutils2/command/install_data.py b/distutils2/command/install_data.py
--- a/distutils2/command/install_data.py
+++ b/distutils2/command/install_data.py
@@ -48,7 +48,7 @@
             self.mkpath(dir_dest)
             try:
                 out = self.copy_file(_file[0], dir_dest)[0]
-            except Error, e:
+            except Error as e:
                 logger.warning('%s: %s', self.get_command_name(), e)
                 out = destination
 
diff --git a/distutils2/command/install_dist.py b/distutils2/command/install_dist.py
--- a/distutils2/command/install_dist.py
+++ b/distutils2/command/install_dist.py
@@ -13,12 +13,6 @@
 from distutils2.util import convert_path, change_root, get_platform
 from distutils2.errors import PackagingOptionError
 
-import site
-if sys.version_info[:2] >= (2, 6):
-    HAS_USER_SITE = True
-else:
-    HAS_USER_SITE = False
-
 
 class install_dist(Command):
 
@@ -30,6 +24,9 @@
          "installation prefix"),
         ('exec-prefix=', None,
          "(Unix only) prefix for platform-specific files"),
+        ('user', None,
+         "install in user site-packages directory [%s]" %
+         get_path('purelib', '%s_user' % os.name)),
         ('home=', None,
          "(Unix only) home directory to install under"),
 
@@ -100,15 +97,7 @@
         ]
 
     boolean_options = ['compile', 'force', 'skip-build', 'no-distinfo',
-                       'requested', 'no-record']
-
-    if HAS_USER_SITE:
-        user_options.append(
-            ('user', None,
-             "install in user site-packages directory [%s]" %
-             get_path('purelib', '%s_user' % os.name)))
-
-        boolean_options.append('user')
+                       'requested', 'no-record', 'user']
 
     negative_opt = {'no-compile': 'compile', 'no-requested': 'requested'}
 
@@ -118,8 +107,7 @@
         self.prefix = None
         self.exec_prefix = None
         self.home = None
-        if HAS_USER_SITE:
-            self.user = False
+        self.user = False
 
         # These select only the installation base; it's up to the user to
         # specify the installation scheme (currently, that means supplying
@@ -138,9 +126,8 @@
         self.install_lib = None         # set to either purelib or platlib
         self.install_scripts = None
         self.install_data = None
-        if HAS_USER_SITE:
-            self.install_userbase = get_config_var('userbase')
-            self.install_usersite = get_path('purelib', '%s_user' % os.name)
+        self.install_userbase = get_config_var('userbase')
+        self.install_usersite = get_path('purelib', '%s_user' % os.name)
 
         self.compile = None
         self.optimize = None
@@ -221,9 +208,8 @@
             raise PackagingOptionError(
                 "must supply either home or prefix/exec-prefix -- not both")
 
-        if HAS_USER_SITE and self.user and (
-                self.prefix or self.exec_prefix or self.home or
-                self.install_base or self.install_platbase):
+        if self.user and (self.prefix or self.exec_prefix or self.home or
+                          self.install_base or self.install_platbase):
             raise PackagingOptionError(
                 "can't combine user with prefix/exec_prefix/home or "
                 "install_base/install_platbase")
@@ -276,11 +262,9 @@
             'exec_prefix': exec_prefix,
             'srcdir': srcdir,
             'projectbase': projectbase,
-            }
-
-        if HAS_USER_SITE:
-            self.config_vars['userbase'] = self.install_userbase
-            self.config_vars['usersite'] = self.install_usersite
+            'userbase': self.install_userbase,
+            'usersite': self.install_usersite,
+        }
 
         self.expand_basedirs()
 
@@ -298,7 +282,7 @@
         self.dump_dirs("post-expand_dirs()")
 
         # Create directories under USERBASE
-        if HAS_USER_SITE and self.user:
+        if self.user:
             self.create_user_dirs()
 
         # Pick the actual directory to install all modules to: either
@@ -313,10 +297,8 @@
 
         # Convert directories from Unix /-separated syntax to the local
         # convention.
-        self.convert_paths('lib', 'purelib', 'platlib',
-                           'scripts', 'data', 'headers')
-        if HAS_USER_SITE:
-            self.convert_paths('userbase', 'usersite')
+        self.convert_paths('lib', 'purelib', 'platlib', 'scripts',
+                           'data', 'headers', 'userbase', 'usersite')
 
         # Well, we're not actually fully completely finalized yet: we still
         # have to deal with 'extra_path', which is the hack for allowing
@@ -357,7 +339,7 @@
                     "installation scheme is incomplete")
             return
 
-        if HAS_USER_SITE and self.user:
+        if self.user:
             if self.install_userbase is None:
                 raise PackagingPlatformError(
                     "user base directory is not specified")
@@ -385,7 +367,7 @@
 
     def finalize_other(self):
         """Finalize options for non-posix platforms"""
-        if HAS_USER_SITE and self.user:
+        if self.user:
             if self.install_userbase is None:
                 raise PackagingPlatformError(
                     "user base directory is not specified")
@@ -466,7 +448,7 @@
             self.extra_path = self.distribution.extra_path
 
         if self.extra_path is not None:
-            if isinstance(self.extra_path, basestring):
+            if isinstance(self.extra_path, str):
                 self.extra_path = self.extra_path.split(',')
 
             if len(self.extra_path) == 1:
@@ -501,7 +483,7 @@
         home = convert_path(os.path.expanduser("~"))
         for name, path in self.config_vars.items():
             if path.startswith(home) and not os.path.isdir(path):
-                os.makedirs(path, 0700)
+                os.makedirs(path, 0o700)
 
     # -- Command execution methods -------------------------------------
 
diff --git a/distutils2/command/install_distinfo.py b/distutils2/command/install_distinfo.py
--- a/distutils2/command/install_distinfo.py
+++ b/distutils2/command/install_distinfo.py
@@ -4,12 +4,8 @@
 
 import os
 import csv
-import codecs
+import hashlib
 from shutil import rmtree
-try:
-    import hashlib
-except ImportError:
-    from distutils2._backport import hashlib
 
 from distutils2 import logger
 from distutils2.command.cmd import Command
@@ -90,11 +86,8 @@
         installer_path = os.path.join(self.distinfo_dir, 'INSTALLER')
         logger.info('creating %s', installer_path)
         if not self.dry_run:
-            f = open(installer_path, 'w')
-            try:
+            with open(installer_path, 'w') as f:
                 f.write(self.installer)
-            finally:
-                f.close()
         self.outfiles.append(installer_path)
 
         if self.requested:
@@ -111,15 +104,12 @@
                                               'RESOURCES')
                 logger.info('creating %s', resources_path)
                 if not self.dry_run:
-                    f = open(resources_path, 'wb')
-                    try:
+                    with open(resources_path, 'wb') as f:
                         writer = csv.writer(f, delimiter=',',
                                             lineterminator='\n',
                                             quotechar='"')
                         for row in install_data.get_resources_out():
                             writer.writerow(row)
-                    finally:
-                        f.close()
 
                 self.outfiles.append(resources_path)
 
@@ -127,8 +117,7 @@
             record_path = os.path.join(self.distinfo_dir, 'RECORD')
             logger.info('creating %s', record_path)
             if not self.dry_run:
-                f = codecs.open(record_path, 'w', encoding='utf-8')
-                try:
+                with open(record_path, 'w', encoding='utf-8') as f:
                     writer = csv.writer(f, delimiter=',',
                                         lineterminator='\n',
                                         quotechar='"')
@@ -141,19 +130,14 @@
                             writer.writerow((fpath, '', ''))
                         else:
                             size = os.path.getsize(fpath)
-                            fp = open(fpath, 'rb')
-                            try:
+                            with open(fpath, 'rb') as fp:
                                 hash = hashlib.md5()
                                 hash.update(fp.read())
-                            finally:
-                                fp.close()
                             md5sum = hash.hexdigest()
                             writer.writerow((fpath, md5sum, size))
 
                     # add the RECORD file itself
                     writer.writerow((record_path, '', ''))
-                finally:
-                    f.close()
             self.outfiles.append(record_path)
 
     def get_outputs(self):
diff --git a/distutils2/command/install_lib.py b/distutils2/command/install_lib.py
--- a/distutils2/command/install_lib.py
+++ b/distutils2/command/install_lib.py
@@ -114,7 +114,7 @@
         return outfiles
 
     def byte_compile(self, files):
-        if getattr(sys, 'dont_write_bytecode', False):
+        if sys.dont_write_bytecode:
             # XXX do we want this?  because a Python runs without bytecode
             # doesn't mean that the *dists should not contain bytecode
             #--or does it?
diff --git a/distutils2/command/install_scripts.py b/distutils2/command/install_scripts.py
--- a/distutils2/command/install_scripts.py
+++ b/distutils2/command/install_scripts.py
@@ -48,7 +48,7 @@
                 if self.dry_run:
                     logger.info("changing mode of %s", file)
                 else:
-                    mode = (os.stat(file).st_mode | 0555) & 07777
+                    mode = (os.stat(file).st_mode | 0o555) & 0o7777
                     logger.info("changing mode of %s to %o", file, mode)
                     os.chmod(file, mode)
 
diff --git a/distutils2/command/register.py b/distutils2/command/register.py
--- a/distutils2/command/register.py
+++ b/distutils2/command/register.py
@@ -3,8 +3,9 @@
 # Contributed by Richard Jones
 
 import getpass
-import urllib2
-import urlparse
+import urllib.error
+import urllib.parse
+import urllib.request
 
 from distutils2 import logger
 from distutils2.util import (read_pypirc, generate_pypirc, DEFAULT_REPOSITORY,
@@ -80,7 +81,7 @@
     def classifiers(self):
         ''' Fetch the list of classifiers from the server.
         '''
-        response = urllib2.urlopen(self.repository+'?:action=list_classifiers')
+        response = urllib.request.urlopen(self.repository+'?:action=list_classifiers')
         logger.info(response.read())
 
     def verify_metadata(self):
@@ -143,22 +144,22 @@
  4. quit
 Your selection [default 1]: ''')
 
-            choice = raw_input()
+            choice = input()
             if not choice:
                 choice = '1'
             elif choice not in choices:
-                print 'Please choose one of the four options!'
+                print('Please choose one of the four options!')
 
         if choice == '1':
             # get the username and password
             while not username:
-                username = raw_input('Username: ')
+                username = input('Username: ')
             while not password:
                 password = getpass.getpass('Password: ')
 
             # set up the authentication
-            auth = urllib2.HTTPPasswordMgr()
-            host = urlparse.urlparse(self.repository)[1]
+            auth = urllib.request.HTTPPasswordMgr()
+            host = urllib.parse.urlparse(self.repository)[1]
             auth.add_password(self.realm, host, username, password)
             # send the info to the server and report the result
             code, result = self.post_to_server(self.build_post_data('submit'),
@@ -178,7 +179,7 @@
                         get_pypirc_path())
                     choice = 'X'
                     while choice.lower() not in 'yn':
-                        choice = raw_input('Save your login (y/N)?')
+                        choice = input('Save your login (y/N)?')
                         if not choice:
                             choice = 'n'
                     if choice.lower() == 'y':
@@ -189,7 +190,7 @@
             data['name'] = data['password'] = data['email'] = ''
             data['confirm'] = None
             while not data['name']:
-                data['name'] = raw_input('Username: ')
+                data['name'] = input('Username: ')
             while data['password'] != data['confirm']:
                 while not data['password']:
                     data['password'] = getpass.getpass('Password: ')
@@ -198,9 +199,9 @@
                 if data['password'] != data['confirm']:
                     data['password'] = ''
                     data['confirm'] = None
-                    print "Password and confirm don't match!"
+                    print("Password and confirm don't match!")
             while not data['email']:
-                data['email'] = raw_input('   EMail: ')
+                data['email'] = input('   EMail: ')
             code, result = self.post_to_server(data)
             if code != 200:
                 logger.info('server response (%s): %s', code, result)
@@ -211,7 +212,7 @@
             data = {':action': 'password_reset'}
             data['email'] = ''
             while not data['email']:
-                data['email'] = raw_input('Your email address: ')
+                data['email'] = input('Your email address: ')
             code, result = self.post_to_server(data)
             logger.info('server response (%s): %s', code, result)
 
@@ -236,20 +237,20 @@
             'Content-type': content_type,
             'Content-length': str(len(body))
         }
-        req = urllib2.Request(self.repository, body, headers)
+        req = urllib.request.Request(self.repository, body, headers)
 
         # handle HTTP and include the Basic Auth handler
-        opener = urllib2.build_opener(
-            urllib2.HTTPBasicAuthHandler(password_mgr=auth)
+        opener = urllib.request.build_opener(
+            urllib.request.HTTPBasicAuthHandler(password_mgr=auth)
         )
         data = ''
         try:
             result = opener.open(req)
-        except urllib2.HTTPError, e:
+        except urllib.error.HTTPError as e:
             if self.show_response:
                 data = e.fp.read()
             result = e.code, e.msg
-        except urllib2.URLError, e:
+        except urllib.error.URLError as e:
             result = 500, str(e)
         else:
             if self.show_response:
diff --git a/distutils2/command/sdist.py b/distutils2/command/sdist.py
--- a/distutils2/command/sdist.py
+++ b/distutils2/command/sdist.py
@@ -3,7 +3,7 @@
 import os
 import re
 import sys
-from StringIO import StringIO
+from io import StringIO
 
 from distutils2 import logger
 from distutils2.util import resolve_name
@@ -134,7 +134,7 @@
         if self.manifest_builders is None:
             self.manifest_builders = []
         else:
-            if isinstance(self.manifest_builders, basestring):
+            if isinstance(self.manifest_builders, str):
                 self.manifest_builders = self.manifest_builders.split(',')
             builders = []
             for builder in self.manifest_builders:
@@ -143,7 +143,7 @@
                     continue
                 try:
                     builder = resolve_name(builder)
-                except ImportError, e:
+                except ImportError as e:
                     raise PackagingModuleError(e)
 
                 builders.append(builder)
@@ -337,7 +337,7 @@
         """
         return self.archive_files
 
-    def create_tree(self, base_dir, files, mode=0777, verbose=1,
+    def create_tree(self, base_dir, files, mode=0o777, verbose=1,
                     dry_run=False):
         need_dir = set()
         for file in files:
diff --git a/distutils2/command/upload.py b/distutils2/command/upload.py
--- a/distutils2/command/upload.py
+++ b/distutils2/command/upload.py
@@ -4,14 +4,11 @@
 import socket
 import logging
 import platform
-import urlparse
+import urllib.parse
 from base64 import standard_b64encode
-try:
-    from hashlib import md5
-except ImportError:
-    from distutils2._backport.hashlib import md5
-from urllib2 import HTTPError
-from urllib2 import urlopen, Request
+from hashlib import md5
+from urllib.error import HTTPError
+from urllib.request import urlopen, Request
 
 from distutils2 import logger
 from distutils2.errors import PackagingOptionError
@@ -87,7 +84,7 @@
     def upload_file(self, command, pyversion, filename):
         # Makes sure the repository URL is compliant
         scheme, netloc, url, params, query, fragments = \
-            urlparse.urlparse(self.repository)
+            urllib.parse.urlparse(self.repository)
         if params or query or fragments:
             raise AssertionError("Incompatible url %s" % self.repository)
 
@@ -104,11 +101,8 @@
 
         # Fill in the data - send all the metadata in case we need to
         # register a new release
-        f = open(filename, 'rb')
-        try:
+        with open(filename, 'rb') as f:
             content = f.read()
-        finally:
-            f.close()
 
         data = self.distribution.metadata.todict()
 
@@ -124,11 +118,8 @@
             data['comment'] = 'built for %s' % platform.platform(terse=True)
 
         if self.sign:
-            fp = open(filename + '.asc')
-            try:
+            with open(filename + '.asc') as fp:
                 sig = fp.read()
-            finally:
-                fp.close()
             data['gpg_signature'] = [
                 (os.path.basename(filename) + ".asc", sig)]
 
@@ -136,7 +127,7 @@
         # The exact encoding of the authentication string is debated.
         # Anyway PyPI only accepts ascii for both username or password.
         user_pass = (self.username + ":" + self.password).encode('ascii')
-        auth = "Basic " + standard_b64encode(user_pass)
+        auth = b"Basic " + standard_b64encode(user_pass)
 
         # Build up the MIME payload for the POST data
         files = []
@@ -160,10 +151,10 @@
             result = urlopen(request)
             status = result.code
             reason = result.msg
-        except socket.error, e:
+        except socket.error as e:
             logger.error(e)
             return
-        except HTTPError, e:
+        except HTTPError as e:
             status = e.code
             reason = e.msg
 
diff --git a/distutils2/command/upload_docs.py b/distutils2/command/upload_docs.py
--- a/distutils2/command/upload_docs.py
+++ b/distutils2/command/upload_docs.py
@@ -5,9 +5,9 @@
 import socket
 import zipfile
 import logging
-import httplib
-import urlparse
-from StringIO import StringIO
+import http.client
+import urllib.parse
+from io import BytesIO
 
 from distutils2 import logger
 from distutils2.util import (read_pypirc, DEFAULT_REPOSITORY, DEFAULT_REALM,
@@ -18,7 +18,7 @@
 
 def zip_dir(directory):
     """Compresses recursively contents of directory into a BytesIO object"""
-    destination = StringIO()
+    destination = BytesIO()
     zip_file = zipfile.ZipFile(destination, "w")
     try:
         for root, dirs, files in os.walk(directory):
@@ -91,16 +91,16 @@
 
         credentials = self.username + ':' + self.password
         # FIXME should use explicit encoding
-        auth = "Basic " + base64.encodestring(credentials.encode()).strip()
+        auth = b"Basic " + base64.encodebytes(credentials.encode()).strip()
 
         logger.info("Submitting documentation to %s", self.repository)
 
-        scheme, netloc, url, params, query, fragments = urlparse.urlparse(
+        scheme, netloc, url, params, query, fragments = urllib.parse.urlparse(
             self.repository)
         if scheme == "http":
-            conn = httplib.HTTPConnection(netloc)
+            conn = http.client.HTTPConnection(netloc)
         elif scheme == "https":
-            conn = httplib.HTTPSConnection(netloc)
+            conn = http.client.HTTPSConnection(netloc)
         else:
             raise AssertionError("unsupported scheme %r" % scheme)
 
@@ -113,7 +113,7 @@
             conn.endheaders()
             conn.send(body)
 
-        except socket.error, e:
+        except socket.error as e:
             logger.error(e)
             return
 
diff --git a/distutils2/compat.py b/distutils2/compat.py
--- a/distutils2/compat.py
+++ b/distutils2/compat.py
@@ -4,10 +4,7 @@
 Python 3.2, for internal use only.  Whole modules are in _backport.
 """
 
-import os
-import re
 import sys
-import codecs
 from distutils2 import logger
 
 
@@ -62,131 +59,12 @@
 # The rest of this file does not exist in packaging
 # functions are sorted alphabetically and are not included in __all__
 
-try:
-    any
-except NameError:
-    def any(seq):
-        for elem in seq:
-            if elem:
-                return True
-        return False
-
-
-_cookie_re = re.compile("coding[:=]\s*([-\w.]+)")
-
-
-def _get_normal_name(orig_enc):
-    """Imitates get_normal_name in tokenizer.c."""
-    # Only care about the first 12 characters.
-    enc = orig_enc[:12].lower().replace("_", "-")
-    if enc == "utf-8" or enc.startswith("utf-8-"):
-        return "utf-8"
-    if enc in ("latin-1", "iso-8859-1", "iso-latin-1") or \
-       enc.startswith(("latin-1-", "iso-8859-1-", "iso-latin-1-")):
-        return "iso-8859-1"
-    return orig_enc
-
-
-def detect_encoding(readline):
-    """
-    The detect_encoding() function is used to detect the encoding that should
-    be used to decode a Python source file.  It requires one argment, readline,
-    in the same way as the tokenize() generator.
-
-    It will call readline a maximum of twice, and return the encoding used
-    (as a string) and a list of any lines (left as bytes) it has read in.
-
-    It detects the encoding from the presence of a utf-8 bom or an encoding
-    cookie as specified in pep-0263.  If both a bom and a cookie are present,
-    but disagree, a SyntaxError will be raised.  If the encoding cookie is an
-    invalid charset, raise a SyntaxError.  Note that if a utf-8 bom is found,
-    'utf-8-sig' is returned.
-
-    If no encoding is specified, then the default of 'utf-8' will be returned.
-    """
-    bom_found = False
-    encoding = None
-    default = 'utf-8'
-
-    def read_or_stop():
-        try:
-            return readline()
-        except StopIteration:
-            return ''
-
-    def find_cookie(line):
-        try:
-            line_string = line.decode('ascii')
-        except UnicodeDecodeError:
-            return None
-
-        matches = _cookie_re.findall(line_string)
-        if not matches:
-            return None
-        encoding = _get_normal_name(matches[0])
-        try:
-            codec = codecs.lookup(encoding)
-        except LookupError:
-            # This behaviour mimics the Python interpreter
-            raise SyntaxError("unknown encoding: " + encoding)
-
-        if bom_found:
-            if codec.name != 'utf-8':
-                # This behaviour mimics the Python interpreter
-                raise SyntaxError('encoding problem: utf-8')
-            encoding += '-sig'
-        return encoding
-
-    first = read_or_stop()
-    if first.startswith(codecs.BOM_UTF8):
-        bom_found = True
-        first = first[3:]
-        default = 'utf-8-sig'
-    if not first:
-        return default, []
-
-    encoding = find_cookie(first)
-    if encoding:
-        return encoding, [first]
-
-    second = read_or_stop()
-    if not second:
-        return default, [first]
-
-    encoding = find_cookie(second)
-    if encoding:
-        return encoding, [first, second]
-
-    return default, [first, second]
-
 
 def fsencode(filename):
-    if isinstance(filename, str):
+    if isinstance(filename, bytes):
         return filename
-    elif isinstance(filename, unicode):
+    elif isinstance(filename, str):
         return filename.encode(sys.getfilesystemencoding())
     else:
         raise TypeError("expect bytes or str, not %s" %
                         type(filename).__name__)
-
-
-try:
-    from functools import wraps
-except ImportError:
-    def wraps(func=None):
-        """No-op replacement for functools.wraps"""
-        def wrapped(func):
-            return func
-        return wrapped
-
-try:
-    from platform import python_implementation
-except ImportError:
-    def python_implementation():
-        if 'PyPy' in sys.version:
-            return 'PyPy'
-        if os.name == 'java':
-            return 'Jython'
-        if sys.version.startswith('IronPython'):
-            return 'IronPython'
-        return 'CPython'
diff --git a/distutils2/compiler/__init__.py b/distutils2/compiler/__init__.py
--- a/distutils2/compiler/__init__.py
+++ b/distutils2/compiler/__init__.py
@@ -142,7 +142,7 @@
     compilers = []
 
     for name, cls in _COMPILERS.items():
-        if isinstance(cls, basestring):
+        if isinstance(cls, str):
             cls = resolve_name(cls)
             _COMPILERS[name] = cls
 
@@ -179,7 +179,7 @@
             msg = msg + " with '%s' compiler" % compiler
         raise PackagingPlatformError(msg)
 
-    if isinstance(cls, basestring):
+    if isinstance(cls, str):
         cls = resolve_name(cls)
         _COMPILERS[compiler] = cls
 
diff --git a/distutils2/compiler/bcppcompiler.py b/distutils2/compiler/bcppcompiler.py
--- a/distutils2/compiler/bcppcompiler.py
+++ b/distutils2/compiler/bcppcompiler.py
@@ -104,7 +104,7 @@
                 # This needs to be compiled to a .res file -- do it now.
                 try:
                     self.spawn(["brcc32", "-fo", obj, src])
-                except PackagingExecError, msg:
+                except PackagingExecError as msg:
                     raise CompileError(msg)
                 continue # the 'for' loop
 
@@ -128,7 +128,7 @@
                 self.spawn([self.cc] + compile_opts + pp_opts +
                            [input_opt, output_opt] +
                            extra_postargs + [src])
-            except PackagingExecError, msg:
+            except PackagingExecError as msg:
                 raise CompileError(msg)
 
         return objects
@@ -146,7 +146,7 @@
                 pass                    # XXX what goes here?
             try:
                 self.spawn([self.lib] + lib_args)
-            except PackagingExecError, msg:
+            except PackagingExecError as msg:
                 raise LibError(msg)
         else:
             logger.debug("skipping %s (up-to-date)", output_filename)
@@ -268,7 +268,7 @@
             self.mkpath(os.path.dirname(output_filename))
             try:
                 self.spawn([self.linker] + ld_args)
-            except PackagingExecError, msg:
+            except PackagingExecError as msg:
                 raise LinkError(msg)
 
         else:
@@ -351,5 +351,5 @@
                 self.mkpath(os.path.dirname(output_file))
             try:
                 self.spawn(pp_args)
-            except PackagingExecError, exc:
-                raise CompileError(exc)
+            except PackagingExecError as msg:
+                raise CompileError(msg)
diff --git a/distutils2/compiler/ccompiler.py b/distutils2/compiler/ccompiler.py
--- a/distutils2/compiler/ccompiler.py
+++ b/distutils2/compiler/ccompiler.py
@@ -12,7 +12,7 @@
 from distutils2.compiler import gen_preprocess_options
 
 
-class CCompiler(object):
+class CCompiler:
     """Abstract base class to define the interface that must be implemented
     by real compiler classes.  Also has some utility methods used by
     several compiler classes.
@@ -148,7 +148,7 @@
             self.set_executable(key, value)
 
     def set_executable(self, key, value):
-        if isinstance(value, basestring):
+        if isinstance(value, str):
             setattr(self, key, split_quoted(value))
         else:
             setattr(self, key, value)
@@ -170,8 +170,8 @@
             if not (isinstance(defn, tuple) and
                     (len(defn) == 1 or
                      (len(defn) == 2 and
-                      (isinstance(defn[1], basestring) or defn[1] is None))) and
-                    isinstance(defn[0], basestring)):
+                      (isinstance(defn[1], str) or defn[1] is None))) and
+                    isinstance(defn[0], str)):
                 raise TypeError(("invalid macro definition '%s': " % defn) + \
                       "must be tuple (string,), (string, string), or " + \
                       "(string, None)")
@@ -311,7 +311,7 @@
         """Process arguments and decide which source files to compile."""
         if outdir is None:
             outdir = self.output_dir
-        elif not isinstance(outdir, basestring):
+        elif not isinstance(outdir, str):
             raise TypeError("'output_dir' must be a string or None")
 
         if macros is None:
@@ -371,7 +371,7 @@
         """
         if output_dir is None:
             output_dir = self.output_dir
-        elif not isinstance(output_dir, basestring):
+        elif not isinstance(output_dir, str):
             raise TypeError("'output_dir' must be a string or None")
 
         if macros is None:
@@ -403,7 +403,7 @@
 
         if output_dir is None:
             output_dir = self.output_dir
-        elif not isinstance(output_dir, basestring):
+        elif not isinstance(output_dir, str):
             raise TypeError("'output_dir' must be a string or None")
 
         return objects, output_dir
@@ -727,8 +727,7 @@
         if library_dirs is None:
             library_dirs = []
         fd, fname = tempfile.mkstemp(".c", funcname, text=True)
-        f = os.fdopen(fd, "w")
-        try:
+        with os.fdopen(fd, "w") as f:
             for incl in includes:
                 f.write("""#include "%s"\n""" % incl)
             f.write("""\
@@ -736,8 +735,6 @@
     %s();
 }
 """ % funcname)
-        finally:
-            f.close()
         try:
             objects = self.compile([fname], include_dirs=include_dirs)
         except CompileError:
@@ -854,7 +851,7 @@
             return
         return move(src, dst)
 
-    def mkpath(self, name, mode=0777):
+    def mkpath(self, name, mode=0o777):
         name = os.path.normpath(name)
         if os.path.isdir(name) or name == '':
             return
diff --git a/distutils2/compiler/cygwinccompiler.py b/distutils2/compiler/cygwinccompiler.py
--- a/distutils2/compiler/cygwinccompiler.py
+++ b/distutils2/compiler/cygwinccompiler.py
@@ -156,13 +156,13 @@
             # gcc needs '.res' and '.rc' compiled to object files !!!
             try:
                 self.spawn(["windres", "-i", src, "-o", obj])
-            except PackagingExecError, msg:
+            except PackagingExecError as msg:
                 raise CompileError(msg)
         else: # for other files use the C-compiler
             try:
                 self.spawn(self.compiler_so + cc_args + [src, '-o', obj] +
                            extra_postargs)
-            except PackagingExecError, msg:
+            except PackagingExecError as msg:
                 raise CompileError(msg)
 
     def link(self, target_desc, objects, output_filename, output_dir=None,
@@ -344,14 +344,11 @@
     # let's see if __GNUC__ is mentioned in python.h
     fn = sysconfig.get_config_h_filename()
     try:
-        config_h = open(fn)
-        try:
+        with open(fn) as config_h:
             if "__GNUC__" in config_h.read():
                 return CONFIG_H_OK, "'%s' mentions '__GNUC__'" % fn
             else:
                 return CONFIG_H_NOTOK, "'%s' does not mention '__GNUC__'" % fn
-        finally:
-            config_h.close()
-    except IOError, exc:
+    except IOError as exc:
         return (CONFIG_H_UNCERTAIN,
                 "couldn't read '%s': %s" % (fn, exc.strerror))
diff --git a/distutils2/compiler/extension.py b/distutils2/compiler/extension.py
--- a/distutils2/compiler/extension.py
+++ b/distutils2/compiler/extension.py
@@ -13,7 +13,7 @@
 # order to do anything.
 
 
-class Extension(object):
+class Extension:
     """Just a collection of attributes that describes an extension
     module and everything needed to build it (hopefully in a portable
     way, but there are hooks that let you be as unportable as you need).
@@ -86,14 +86,14 @@
                  extra_compile_args=None, extra_link_args=None,
                  export_symbols=None, swig_opts=None, depends=None,
                  language=None, optional=None, **kw):
-        if not isinstance(name, basestring):
+        if not isinstance(name, str):
             raise AssertionError("'name' must be a string")
 
         if not isinstance(sources, list):
             raise AssertionError("'sources' must be a list of strings")
 
         for v in sources:
-            if not isinstance(v, basestring):
+            if not isinstance(v, str):
                 raise AssertionError("'sources' must be a list of strings")
 
         self.name = name
diff --git a/distutils2/compiler/msvc9compiler.py b/distutils2/compiler/msvc9compiler.py
--- a/distutils2/compiler/msvc9compiler.py
+++ b/distutils2/compiler/msvc9compiler.py
@@ -46,7 +46,7 @@
 }
 
 
-class Reg(object):
+class Reg:
     """Helper class to read values from the registry
     """
 
@@ -108,7 +108,7 @@
         return s
     convert_mbcs = staticmethod(convert_mbcs)
 
-class MacroExpander(object):
+class MacroExpander:
 
     def __init__(self, version):
         self.macros = {}
@@ -477,7 +477,7 @@
                 try:
                     self.spawn([self.rc] + pp_opts +
                                [output_opt] + [input_opt])
-                except PackagingExecError, msg:
+                except PackagingExecError as msg:
                     raise CompileError(msg)
                 continue
             elif ext in self._mc_extensions:
@@ -504,7 +504,7 @@
                     self.spawn([self.rc] +
                                ["/fo" + obj] + [rc_file])
 
-                except PackagingExecError, msg:
+                except PackagingExecError as msg:
                     raise CompileError(msg)
                 continue
             else:
@@ -517,7 +517,7 @@
                 self.spawn([self.cc] + compile_opts + pp_opts +
                            [input_opt, output_opt] +
                            extra_postargs)
-            except PackagingExecError, msg:
+            except PackagingExecError as msg:
                 raise CompileError(msg)
 
         return objects
@@ -542,7 +542,7 @@
                 pass # XXX what goes here?
             try:
                 self.spawn([self.lib] + lib_args)
-            except PackagingExecError, msg:
+            except PackagingExecError as msg:
                 raise LibError(msg)
         else:
             logger.debug("skipping %s (up-to-date)", output_filename)
@@ -620,7 +620,7 @@
             self.mkpath(os.path.dirname(output_filename))
             try:
                 self.spawn([self.linker] + ld_args)
-            except PackagingExecError, msg:
+            except PackagingExecError as msg:
                 raise LinkError(msg)
 
             # embed the manifest
@@ -637,7 +637,7 @@
             try:
                 self.spawn(['mt.exe', '-nologo', '-manifest',
                             temp_manifest, out_arg])
-            except PackagingExecError, msg:
+            except PackagingExecError as msg:
                 raise LinkError(msg)
         else:
             logger.debug("skipping %s (up-to-date)", output_filename)
diff --git a/distutils2/compiler/msvccompiler.py b/distutils2/compiler/msvccompiler.py
--- a/distutils2/compiler/msvccompiler.py
+++ b/distutils2/compiler/msvccompiler.py
@@ -105,7 +105,7 @@
     return s
 
 
-class MacroExpander(object):
+class MacroExpander:
 
     def __init__(self, version):
         self.macros = {}
@@ -386,7 +386,7 @@
                 try:
                     self.spawn([self.rc] + pp_opts +
                                [output_opt] + [input_opt])
-                except PackagingExecError, msg:
+                except PackagingExecError as msg:
                     raise CompileError(msg)
                 continue
             elif ext in self._mc_extensions:
@@ -415,7 +415,7 @@
                     self.spawn([self.rc] +
                                 ["/fo" + obj] + [rc_file])
 
-                except PackagingExecError, msg:
+                except PackagingExecError as msg:
                     raise CompileError(msg)
                 continue
             else:
@@ -429,7 +429,7 @@
                 self.spawn([self.cc] + compile_opts + pp_opts +
                            [input_opt, output_opt] +
                            extra_postargs)
-            except PackagingExecError, msg:
+            except PackagingExecError as msg:
                 raise CompileError(msg)
 
         return objects
@@ -448,7 +448,7 @@
                 pass                    # XXX what goes here?
             try:
                 self.spawn([self.lib] + lib_args)
-            except PackagingExecError, msg:
+            except PackagingExecError as msg:
                 raise LibError(msg)
 
         else:
@@ -515,7 +515,7 @@
             self.mkpath(os.path.dirname(output_filename))
             try:
                 self.spawn([self.linker] + ld_args)
-            except PackagingExecError, msg:
+            except PackagingExecError as msg:
                 raise LinkError(msg)
 
         else:
diff --git a/distutils2/compiler/unixccompiler.py b/distutils2/compiler/unixccompiler.py
--- a/distutils2/compiler/unixccompiler.py
+++ b/distutils2/compiler/unixccompiler.py
@@ -165,7 +165,7 @@
                 self.mkpath(os.path.dirname(output_file))
             try:
                 self.spawn(pp_args)
-            except PackagingExecError, msg:
+            except PackagingExecError as msg:
                 raise CompileError(msg)
 
     def _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts):
@@ -175,7 +175,7 @@
         try:
             self.spawn(compiler_so + cc_args + [src, '-o', obj] +
                        extra_postargs)
-        except PackagingExecError, msg:
+        except PackagingExecError as msg:
             raise CompileError(msg)
 
     def create_static_lib(self, objects, output_libname,
@@ -199,7 +199,7 @@
             if self.ranlib:
                 try:
                     self.spawn(self.ranlib + [output_filename])
-                except PackagingExecError, msg:
+                except PackagingExecError as msg:
                     raise LibError(msg)
         else:
             logger.debug("skipping %s (up-to-date)", output_filename)
@@ -253,7 +253,7 @@
                     linker = _darwin_compiler_fixup(linker, ld_args)
 
                 self.spawn(linker + ld_args)
-            except PackagingExecError, msg:
+            except PackagingExecError as msg:
                 raise LinkError(msg)
         else:
             logger.debug("skipping %s (up-to-date)", output_filename)
diff --git a/distutils2/config.py b/distutils2/config.py
--- a/distutils2/config.py
+++ b/distutils2/config.py
@@ -2,11 +2,10 @@
 
 import os
 import sys
-import codecs
 import logging
 
 from shlex import split
-from ConfigParser import RawConfigParser
+from configparser import RawConfigParser
 from distutils2 import logger
 from distutils2.errors import PackagingOptionError
 from distutils2.compiler.extension import Extension
@@ -74,7 +73,7 @@
     return destinations
 
 
-class Config(object):
+class Config:
     """Class used to work with configuration files"""
     def __init__(self, dist):
         self.dist = dist
@@ -155,7 +154,7 @@
                     for line in setup_hooks:
                         try:
                             hook = resolve_name(line)
-                        except ImportError, e:
+                        except ImportError as e:
                             logger.warning('cannot find setup hook: %s',
                                            e.args[0])
                         else:
@@ -190,11 +189,8 @@
                     value = []
                     for filename in filenames:
                         # will raise if file not found
-                        description_file = open(filename)
-                        try:
+                        with open(filename) as description_file:
                             value.append(description_file.read().strip())
-                        finally:
-                            description_file.close()
                         # add filename as a required file
                         if filename not in metadata.requires_files:
                             metadata.requires_files.append(filename)
@@ -214,7 +210,7 @@
             self.dist.packages = []
 
             packages = files.get('packages', [])
-            if isinstance(packages, basestring):
+            if isinstance(packages, str):
                 packages = [packages]
 
             for package in packages:
@@ -224,10 +220,10 @@
                 self.dist.packages.append(package)
 
             self.dist.py_modules = files.get('modules', [])
-            if isinstance(self.dist.py_modules, basestring):
+            if isinstance(self.dist.py_modules, str):
                 self.dist.py_modules = [self.dist.py_modules]
             self.dist.scripts = files.get('scripts', [])
-            if isinstance(self.dist.scripts, basestring):
+            if isinstance(self.dist.scripts, str):
                 self.dist.scripts = [self.dist.scripts]
 
             self.dist.package_data = {}
@@ -310,11 +306,8 @@
 
         for filename in filenames:
             logger.debug("  reading %s", filename)
-            f = codecs.open(filename, 'r', encoding='utf-8')
-            try:
+            with open(filename, 'r', encoding='utf-8') as f:
                 parser.readfp(f)
-            finally:
-                f.close()
 
             if os.path.split(filename)[-1] == 'setup.cfg':
                 self._read_setup_cfg(parser, filename)
@@ -338,7 +331,7 @@
 
                     if opt == 'sub_commands':
                         val = split_multiline(val)
-                        if isinstance(val, basestring):
+                        if isinstance(val, str):
                             val = [val]
 
                     # Hooks use a suffix system to prevent being overriden
@@ -371,19 +364,19 @@
                         setattr(self.dist, opt, strtobool(val))
                     else:
                         setattr(self.dist, opt, val)
-                except ValueError, msg:
+                except ValueError as msg:
                     raise PackagingOptionError(msg)
 
     def _load_compilers(self, compilers):
         compilers = split_multiline(compilers)
-        if isinstance(compilers, basestring):
+        if isinstance(compilers, str):
             compilers = [compilers]
         for compiler in compilers:
             set_compiler(compiler.strip())
 
     def _load_commands(self, commands):
         commands = split_multiline(commands)
-        if isinstance(commands, basestring):
+        if isinstance(commands, str):
             commands = [commands]
         for command in commands:
             set_command(command.strip())
diff --git a/distutils2/create.py b/distutils2/create.py
--- a/distutils2/create.py
+++ b/distutils2/create.py
@@ -23,25 +23,17 @@
 import imp
 import sys
 import glob
-import codecs
 import shutil
+from hashlib import md5
 from textwrap import dedent
-from ConfigParser import RawConfigParser
+from tokenize import detect_encoding
+from configparser import RawConfigParser
 
 # importing this with an underscore as it should be replaced by the
 # dict form or another structures for all purposes
 from distutils2._trove import all_classifiers as _CLASSIFIERS_LIST
-from distutils2.compat import detect_encoding
 from distutils2.version import is_valid_version
 from distutils2._backport import sysconfig
-try:
-    any
-except NameError:
-    from distutils2.compat import any
-try:
-    from hashlib import md5
-except ImportError:
-    from distutils2._backport.hashlib import md5
 
 
 _FILENAME = 'setup.cfg'
@@ -120,26 +112,20 @@
     This function load the setup file in all cases (even if it have already
     been loaded before, because we are monkey patching its setup function with
     a particular one"""
-    f = open("setup.py", "rb")
-    try:
+    with open("setup.py", "rb") as f:
         encoding, lines = detect_encoding(f.readline)
-    finally:
-        f.close()
-    f = open("setup.py")
-    try:
+    with open("setup.py", encoding=encoding) as f:
         imp.load_module("setup", f, "setup.py", (".py", "r", imp.PY_SOURCE))
-    finally:
-        f.close()
 
 
 def ask_yn(question, default=None, helptext=None):
     question += ' (y/n)'
     while True:
         answer = ask(question, default, helptext, required=True)
-        if answer and answer[0].lower() in 'yn':
+        if answer and answer[0].lower() in ('y', 'n'):
             return answer[0].lower()
 
-        print '\nERROR: You must select "Y" or "N".\n'
+        print('\nERROR: You must select "Y" or "N".\n')
 
 
 # XXX use util.ask
@@ -148,11 +134,11 @@
 
 def ask(question, default=None, helptext=None, required=True,
         lengthy=False, multiline=False):
-    prompt = u'%s: ' % (question,)
+    prompt = '%s: ' % (question,)
     if default:
-        prompt = u'%s [%s]: ' % (question, default)
+        prompt = '%s [%s]: ' % (question, default)
         if default and len(question) + len(default) > 70:
-            prompt = u'%s\n    [%s]: ' % (question, default)
+            prompt = '%s\n    [%s]: ' % (question, default)
     if lengthy or multiline:
         prompt += '\n   > '
 
@@ -167,19 +153,19 @@
 
         line = sys.stdin.readline().strip()
         if line == '?':
-            print '=' * 70
-            print helptext
-            print '=' * 70
+            print('=' * 70)
+            print(helptext)
+            print('=' * 70)
             continue
         if default and not line:
             return default
         if not line and required:
-            print '*' * 70
-            print 'This value cannot be empty.'
-            print '==========================='
+            print('*' * 70)
+            print('This value cannot be empty.')
+            print('===========================')
             if helptext:
-                print helptext
-            print '*' * 70
+                print(helptext)
+            print('*' * 70)
             continue
         return line
 
@@ -216,7 +202,7 @@
 LICENCES = _build_licences(_CLASSIFIERS_LIST)
 
 
-class MainProgram(object):
+class MainProgram:
     """Make a project setup configuration file (setup.cfg)."""
 
     def __init__(self):
@@ -286,34 +272,32 @@
     def _write_cfg(self):
         if os.path.exists(_FILENAME):
             if os.path.exists('%s.old' % _FILENAME):
-                print ('ERROR: %(name)s.old backup exists, please check that '
-                       'current %(name)s is correct and remove %(name)s.old' %
-                       {'name': _FILENAME})
+                print("ERROR: %(name)s.old backup exists, please check that "
+                      "current %(name)s is correct and remove %(name)s.old" %
+                      {'name': _FILENAME})
                 return
             shutil.move(_FILENAME, '%s.old' % _FILENAME)
 
-        fp = codecs.open(_FILENAME, 'w', encoding='utf-8')
-        try:
-            fp.write(u'[metadata]\n')
+        with open(_FILENAME, 'w', encoding='utf-8') as fp:
+            fp.write('[metadata]\n')
             # TODO use metadata module instead of hard-coding field-specific
             # behavior here
 
             # simple string entries
             for name in ('name', 'version', 'summary', 'download_url'):
-                fp.write(u'%s = %s\n' % (name, self.data.get(name, 'UNKNOWN')))
+                fp.write('%s = %s\n' % (name, self.data.get(name, 'UNKNOWN')))
 
             # optional string entries
             if 'keywords' in self.data and self.data['keywords']:
-                fp.write(u'keywords = %s\n' % ' '.join(self.data['keywords']))
+                fp.write('keywords = %s\n' % ' '.join(self.data['keywords']))
             for name in ('home_page', 'author', 'author_email',
                          'maintainer', 'maintainer_email', 'description-file'):
                 if name in self.data and self.data[name]:
-                    fp.write(u'%s = %s\n' % (name.decode('utf-8'),
-                                             self.data[name].decode('utf-8')))
+                    fp.write('%s = %s\n' % (name, self.data[name]))
             if 'description' in self.data:
                 fp.write(
-                    u'description = %s\n'
-                    % u'\n       |'.join(self.data['description'].split('\n')))
+                    'description = %s\n'
+                    % '\n       |'.join(self.data['description'].split('\n')))
 
             # multiple use string entries
             for name in ('platform', 'supported-platform', 'classifier',
@@ -321,25 +305,23 @@
                          'requires-external'):
                 if not(name in self.data and self.data[name]):
                     continue
-                fp.write(u'%s = ' % name)
-                fp.write(u''.join('    %s\n' % val
+                fp.write('%s = ' % name)
+                fp.write(''.join('    %s\n' % val
                                  for val in self.data[name]).lstrip())
-            fp.write(u'\n[files]\n')
+            fp.write('\n[files]\n')
             for name in ('packages', 'modules', 'scripts',
                          'package_data', 'extra_files'):
                 if not(name in self.data and self.data[name]):
                     continue
-                fp.write(u'%s = %s\n'
-                         % (name, u'\n    '.join(self.data[name]).strip()))
-            fp.write(u'\nresources =\n')
+                fp.write('%s = %s\n'
+                         % (name, '\n    '.join(self.data[name]).strip()))
+            fp.write('\nresources =\n')
             for src, dest in self.data['resources']:
-                fp.write(u'    %s = %s\n' % (src, dest))
-            fp.write(u'\n')
-        finally:
-            fp.close()
+                fp.write('    %s = %s\n' % (src, dest))
+            fp.write('\n')
 
-        os.chmod(_FILENAME, 0644)
-        print 'Wrote %r.' % _FILENAME
+        os.chmod(_FILENAME, 0o644)
+        print('Wrote "%s".' % _FILENAME)
 
     def convert_py_to_cfg(self):
         """Generate a setup.cfg from an existing setup.py.
@@ -368,9 +350,8 @@
                       ('description', 'summary'),
                       ('long_description', 'description'),
                       ('url', 'home_page'),
-                      ('platforms', 'platform'))
-            if sys.version >= '2.5':
-                labels += (
+                      ('platforms', 'platform'),
+                      # backport only for 2.5+
                       ('provides', 'provides-dist'),
                       ('obsoletes', 'obsoletes-dist'),
                       ('requires', 'requires-dist'))
@@ -386,7 +367,7 @@
             # 2.1 data_files -> resources
             if dist.data_files:
                 if (len(dist.data_files) < 2 or
-                    isinstance(dist.data_files[1], basestring)):
+                    isinstance(dist.data_files[1], str)):
                     dist.data_files = [('', dist.data_files)]
                 # add tokens in the destination paths
                 vars = {'distribution.name': data['name']}
@@ -421,11 +402,8 @@
                                  self.data['description']).lower().encode())
                 ref = ref.digest()
                 for readme in glob.glob('README*'):
-                    fp = codecs.open(readme, encoding='utf-8')
-                    try:
+                    with open(readme, encoding='utf-8') as fp:
                         contents = fp.read()
-                    finally:
-                        fp.close()
                     contents = re.sub('\s', '', contents.lower()).encode()
                     val = md5(contents).digest()
                     if val == ref:
@@ -637,8 +615,8 @@
                         break
 
             if len(found_list) == 0:
-                print ('ERROR: Could not find a matching license for "%s"' %
-                       license)
+                print('ERROR: Could not find a matching license for "%s"' %
+                      license)
                 continue
 
             question = 'Matching licenses:\n\n'
@@ -659,8 +637,8 @@
             try:
                 index = found_list[int(choice) - 1]
             except ValueError:
-                print ('ERROR: Invalid selection, type a number from the list '
-                       'above.')
+                print("ERROR: Invalid selection, type a number from the list "
+                      "above.")
 
             classifiers.add(_CLASSIFIERS_LIST[index])
 
@@ -683,8 +661,8 @@
                     classifiers.add(key)
                     return
                 except (IndexError, ValueError):
-                    print ('ERROR: Invalid selection, type a single digit '
-                           'number.')
+                    print("ERROR: Invalid selection, type a single digit "
+                          "number.")
 
 
 def main():
diff --git a/distutils2/database.py b/distutils2/database.py
--- a/distutils2/database.py
+++ b/distutils2/database.py
@@ -5,11 +5,8 @@
 import csv
 import sys
 import zipimport
-from StringIO import StringIO
-try:
-    from hashlib import md5
-except ImportError:
-    from distutils2._backport.hashlib import md5
+from io import StringIO
+from hashlib import md5
 
 from distutils2 import logger
 from distutils2.errors import PackagingError
@@ -122,7 +119,7 @@
             _cache_generated_egg = True
 
 
-class Distribution(object):
+class Distribution:
     """Created with the *path* of the ``.dist-info`` directory provided to the
     constructor. It reads the metadata contained in ``METADATA`` when it is
     instantiated."""
@@ -162,8 +159,7 @@
 
     def _get_records(self, local=False):
         results = []
-        record = self.get_distinfo_file('RECORD')
-        try:
+        with self.get_distinfo_file('RECORD') as record:
             record_reader = csv.reader(record, delimiter=',',
                                        lineterminator='\n')
             for row in record_reader:
@@ -173,20 +169,15 @@
                     path = path.replace('/', os.sep)
                     path = os.path.join(sys.prefix, path)
                 results.append((path, checksum, size))
-        finally:
-            record.close()
         return results
 
     def get_resource_path(self, relative_path):
-        resources_file = self.get_distinfo_file('RESOURCES')
-        try:
+        with self.get_distinfo_file('RESOURCES') as resources_file:
             resources_reader = csv.reader(resources_file, delimiter=',',
                                           lineterminator='\n')
             for relative, destination in resources_reader:
                 if relative == relative_path:
                     return destination
-        finally:
-            resources_file.close()
         raise KeyError(
             'no resource file with relative path %r is installed' %
             relative_path)
@@ -284,7 +275,7 @@
     __hash__ = object.__hash__
 
 
-class EggInfoDistribution(object):
+class EggInfoDistribution:
     """Created with the *path* of the ``.egg-info`` directory or file provided
     to the constructor. It reads the metadata contained in the file itself, or
     if the given path happens to be a directory, the metadata is read from the
@@ -318,7 +309,7 @@
         def yield_lines(strs):
             """Yield non-empty/non-comment lines of a ``basestring``
             or sequence"""
-            if isinstance(strs, basestring):
+            if isinstance(strs, str):
                 for s in strs.splitlines():
                     s = s.strip()
                     # skip blank lines/comments
@@ -337,11 +328,8 @@
                 self.metadata = Metadata(path=meta_path)
                 try:
                     req_path = os.path.join(path, 'EGG-INFO', 'requires.txt')
-                    fp = open(req_path, 'r')
-                    try:
+                    with open(req_path, 'r') as fp:
                         requires = fp.read()
-                    finally:
-                        fp.close()
                 except IOError:
                     requires = None
             else:
@@ -361,11 +349,8 @@
             if os.path.isdir(path):
                 path = os.path.join(path, 'PKG-INFO')
                 try:
-                    fp = open(os.path.join(path, 'requires.txt'), 'r')
-                    try:
+                    with open(os.path.join(path, 'requires.txt'), 'r') as fp:
                         requires = fp.read()
-                    finally:
-                        fp.close()
                 except IOError:
                     requires = None
             self.metadata = Metadata(path=path)
@@ -427,11 +412,8 @@
     def list_installed_files(self, local=False):
 
         def _md5(path):
-            f = open(path, 'rb')
-            try:
+            with open(path, 'rb') as f:
                 content = f.read()
-            finally:
-                f.close()
             return md5(content).hexdigest()
 
         def _size(path):
diff --git a/distutils2/depgraph.py b/distutils2/depgraph.py
--- a/distutils2/depgraph.py
+++ b/distutils2/depgraph.py
@@ -7,7 +7,7 @@
 
 import sys
 
-from StringIO import StringIO
+from io import StringIO
 from distutils2.errors import PackagingError
 from distutils2.version import VersionPredicate, IrrationalVersionError
 
@@ -107,26 +107,26 @@
     """
     disconnected = []
 
-    f.write(u"digraph dependencies {\n")
+    f.write("digraph dependencies {\n")
     for dist, adjs in graph.adjacency_list.items():
         if len(adjs) == 0 and not skip_disconnected:
             disconnected.append(dist)
         for other, label in adjs:
             if not label is None:
-                f.write(u'"%s" -> "%s" [label="%s"]\n' %
+                f.write('"%s" -> "%s" [label="%s"]\n' %
                                             (dist.name, other.name, label))
             else:
-                f.write(u'"%s" -> "%s"\n' % (dist.name, other.name))
+                f.write('"%s" -> "%s"\n' % (dist.name, other.name))
     if not skip_disconnected and len(disconnected) > 0:
-        f.write(u'subgraph disconnected {\n')
-        f.write(u'label = "Disconnected"\n')
-        f.write(u'bgcolor = red\n')
+        f.write('subgraph disconnected {\n')
+        f.write('label = "Disconnected"\n')
+        f.write('bgcolor = red\n')
 
         for dist in disconnected:
-            f.write(u'"%s"' % dist.name)
-            f.write(u'\n')
-        f.write(u'}\n')
-    f.write(u'}\n')
+            f.write('"%s"' % dist.name)
+            f.write('\n')
+        f.write('}\n')
+    f.write('}\n')
 
 
 def generate_graph(dists):
@@ -234,22 +234,22 @@
             graph = generate_graph(dists)
         finally:
             sys.stderr = old
-    except Exception, e:
+    except Exception as e:
         tempout.seek(0)
         tempout = tempout.read()
-        print 'Could not generate the graph'
-        print tempout
-        print e
+        print('Could not generate the graph')
+        print(tempout)
+        print(e)
         sys.exit(1)
 
     for dist, reqs in graph.missing.items():
         if len(reqs) > 0:
-            print 'Warning: Missing dependencies for %r:' % dist.name, \
-                  ', '.join(reqs)
+            print("Warning: Missing dependencies for %r:" % dist.name,
+                  ", ".join(reqs))
     # XXX replace with argparse
     if len(sys.argv) == 1:
-        print 'Dependency graph:'
-        print '   ', repr(graph).replace('\n', '\n    ')
+        print('Dependency graph:')
+        print('   ', repr(graph).replace('\n', '\n    '))
         sys.exit(0)
     elif len(sys.argv) > 1 and sys.argv[1] in ('-d', '--dot'):
         if len(sys.argv) > 2:
@@ -257,18 +257,15 @@
         else:
             filename = 'depgraph.dot'
 
-        f = open(filename, 'w')
-        try:
+        with open(filename, 'w') as f:
             graph_to_dot(graph, f, True)
-        finally:
-            f.close()
         tempout.seek(0)
         tempout = tempout.read()
-        print tempout
-        print 'Dot file written at %r' % filename
+        print(tempout)
+        print('Dot file written at %r' % filename)
         sys.exit(0)
     else:
-        print 'Supported option: -d [filename]'
+        print('Supported option: -d [filename]')
         sys.exit(1)
 
 
diff --git a/distutils2/dist.py b/distutils2/dist.py
--- a/distutils2/dist.py
+++ b/distutils2/dist.py
@@ -32,7 +32,7 @@
     return USAGE % {'script': script}
 
 
-class Distribution(object):
+class Distribution:
     """Class used to represent a project and work with it.
 
     Most of the work hiding behind 'pysetup run' is really done within a
@@ -355,7 +355,7 @@
         # it takes.
         try:
             cmd_class = get_command_class(command)
-        except PackagingModuleError, msg:
+        except PackagingModuleError as msg:
             raise PackagingArgError(msg)
 
         # XXX We want to push this in distutils2.command
@@ -460,14 +460,14 @@
                 options = self.global_options
             parser.set_option_table(options)
             parser.print_help(self.common_usage + "\nGlobal options:")
-            print
+            print()
 
         if display_options:
             parser.set_option_table(self.display_options)
             parser.print_help(
                 "Information display options (just display " +
                 "information, ignore any commands)")
-            print
+            print()
 
         for command in self.commands:
             if isinstance(command, type) and issubclass(command, Command):
@@ -480,9 +480,9 @@
             else:
                 parser.set_option_table(cls.user_options)
             parser.print_help("Options for %r command:" % cls.__name__)
-            print
+            print()
 
-        print gen_usage(self.script_name)
+        print(gen_usage(self.script_name))
 
     def handle_display_options(self, option_order):
         """If there were any non-global "display-only" options
@@ -494,8 +494,8 @@
         # we ignore "foo bar").
         if self.help_commands:
             self.print_commands()
-            print
-            print gen_usage(self.script_name)
+            print()
+            print(gen_usage(self.script_name))
             return True
 
         # If user supplied any of the "display metadata" options, then
@@ -511,12 +511,12 @@
                 opt = opt.replace('-', '_')
                 value = self.metadata[opt]
                 if opt in ('keywords', 'platform'):
-                    print ','.join(value)
+                    print(','.join(value))
                 elif opt in ('classifier', 'provides', 'requires',
                              'obsoletes'):
-                    print '\n'.join(value)
+                    print('\n'.join(value))
                 else:
-                    print value
+                    print(value)
                 any_display_options = True
 
         return any_display_options
@@ -525,14 +525,14 @@
         """Print a subset of the list of all commands -- used by
         'print_commands()'.
         """
-        print header + ":"
+        print(header + ":")
 
         for cmd in commands:
             cls = self.cmdclass.get(cmd) or get_command_class(cmd)
             description = getattr(cls, 'description',
                                   '(no description available)')
 
-            print "  %-*s  %s" % (max_length, cmd, description)
+            print("  %-*s  %s" % (max_length, cmd, description))
 
     def _get_command_groups(self):
         """Helper function to retrieve all the command class names divided
@@ -562,7 +562,7 @@
                                 "Standard commands",
                                 max_length)
         if extra_commands:
-            print
+            print()
             self.print_command_list(extra_commands,
                                     "Extra commands",
                                     max_length)
@@ -622,7 +622,7 @@
                 neg_opt = {}
 
             try:
-                is_string = isinstance(value, basestring)
+                is_string = isinstance(value, str)
                 if option in neg_opt and is_string:
                     setattr(command_obj, neg_opt[option], not strtobool(value))
                 elif option in bool_opts and is_string:
@@ -633,7 +633,7 @@
                     raise PackagingOptionError(
                         "error in %s: command %r has no such option %r" %
                         (source, command_name, option))
-            except ValueError, msg:
+            except ValueError as msg:
                 raise PackagingOptionError(msg)
 
     def get_reinitialized_command(self, command, reinit_subcommands=False):
@@ -725,10 +725,10 @@
             return
 
         for hook in hooks.values():
-            if isinstance(hook, basestring):
+            if isinstance(hook, str):
                 try:
                     hook_obj = resolve_name(hook)
-                except ImportError, e:
+                except ImportError as e:
                     raise PackagingModuleError(e)
             else:
                 hook_obj = hook
diff --git a/distutils2/fancy_getopt.py b/distutils2/fancy_getopt.py
--- a/distutils2/fancy_getopt.py
+++ b/distutils2/fancy_getopt.py
@@ -28,7 +28,7 @@
 neg_alias_re = re.compile("^(%s)=!(%s)$" % (longopt_pat, longopt_pat))
 
 
-class FancyGetopt(object):
+class FancyGetopt:
     """Wrapper around the standard 'getopt()' module that provides some
     handy extra functionality:
       * short and long options are tied together
@@ -151,13 +151,13 @@
                 raise ValueError("invalid option tuple: %r" % option)
 
             # Type- and value-check the option names
-            if not isinstance(longopt, basestring) or len(longopt) < 2:
+            if not isinstance(longopt, str) or len(longopt) < 2:
                 raise PackagingGetoptError(
                       ("invalid long option '%s': "
                        "must be a string of length >= 2") % longopt)
 
             if (not ((short is None) or
-                     (isinstance(short, basestring) and len(short) == 1))):
+                     (isinstance(short, str) and len(short) == 1))):
                 raise PackagingGetoptError(
                       ("invalid short option '%s': "
                        "must be a single character or None") % short)
@@ -237,7 +237,7 @@
 
         try:
             opts, args = getopt.getopt(args, short_opts, self.long_opts)
-        except getopt.error, msg:
+        except getopt.error as msg:
             raise PackagingArgError(msg)
 
         for opt, val in opts:
@@ -377,7 +377,7 @@
     return parser.getopt(args, object)
 
 
-class OptionDummy(object):
+class OptionDummy:
     """Dummy class just used as a place to hold command-line option
     values as instance attributes."""
 
diff --git a/distutils2/install.py b/distutils2/install.py
--- a/distutils2/install.py
+++ b/distutils2/install.py
@@ -51,7 +51,7 @@
         # try to make the paths.
         try:
             os.makedirs(os.path.dirname(new))
-        except OSError, e:
+        except OSError as e:
             if e.errno != errno.EEXIST:
                 raise
         os.rename(old, new)
@@ -88,7 +88,7 @@
         dist.run_command('install_dist')
         name = dist.metadata['Name']
         return database.get_distribution(name) is not None
-    except (IOError, os.error, PackagingError, CCompilerError), msg:
+    except (IOError, os.error, PackagingError, CCompilerError) as msg:
         raise ValueError("Failed to install, " + str(msg))
 
 
@@ -160,7 +160,7 @@
         try:
             func(source_dir)
             return True
-        except ValueError, err:
+        except ValueError as err:
             # failed to install
             logger.info(str(err))
             return False
@@ -187,7 +187,7 @@
         try:
             _install_dist(dist, path)
             installed_dists.append(dist)
-        except Exception, e:
+        except Exception as e:
             logger.info('Failed: %s', e)
 
             # reverting
@@ -396,7 +396,7 @@
     def _move_file(source, target):
         try:
             os.rename(source, target)
-        except OSError, err:
+        except OSError as err:
             return err
         return None
 
@@ -495,12 +495,9 @@
 
     # trying to write a file there
     try:
-        testfile = tempfile.NamedTemporaryFile(suffix=project,
-                                               dir=purelib_path)
-        try:
-            testfile.write('test')
-        finally:
-            testfile.close()
+        with tempfile.NamedTemporaryFile(suffix=project,
+                                         dir=purelib_path) as testfile:
+            testfile.write(b'test')
     except OSError:
         # FIXME this should check the errno, or be removed altogether (race
         # condition: the directory permissions could be changed between here
@@ -525,7 +522,7 @@
         install_from_infos(install_path,
                            info['install'], info['remove'], info['conflict'])
 
-    except InstallationConflict, e:
+    except InstallationConflict as e:
         if logger.isEnabledFor(logging.INFO):
             projects = ('%r %s' % (p.name, p.version) for p in e.args[0])
             logger.info('%r conflicts with %s', project, ','.join(projects))
diff --git a/distutils2/manifest.py b/distutils2/manifest.py
--- a/distutils2/manifest.py
+++ b/distutils2/manifest.py
@@ -22,7 +22,7 @@
 _COMMENTED_LINE = re.compile('#.*?(?=\n)|\n(?=$)', re.M | re.S)
 
 
-class Manifest(object):
+class Manifest:
     """A list of files built by on exploring the filesystem and filtered by
     applying various patterns to what we find there.
     """
@@ -67,7 +67,7 @@
 
         Updates the list accordingly.
         """
-        if isinstance(path_or_file, basestring):
+        if isinstance(path_or_file, str):
             f = open(path_or_file)
         else:
             f = path_or_file
@@ -89,7 +89,7 @@
                 continue
             try:
                 self._process_template_line(line)
-            except PackagingTemplateError, msg:
+            except PackagingTemplateError as msg:
                 logger.warning("%s, %s", path_or_file, msg)
 
     def write(self, path):
@@ -98,11 +98,8 @@
         named by 'self.manifest'.
         """
         if os.path.isfile(path):
-            fp = open(path)
-            try:
+            with open(path) as fp:
                 first_line = fp.readline()
-            finally:
-                fp.close()
 
             if first_line != '# file GENERATED by distutils2, do NOT edit\n':
                 logger.info("not writing to manually maintained "
@@ -122,12 +119,9 @@
         distribution.
         """
         logger.info("reading manifest file %r", path)
-        manifest = open(path)
-        try:
+        with open(path) as manifest:
             for line in manifest.readlines():
                 self.append(line)
-        finally:
-            manifest.close()
 
     def exclude_pattern(self, pattern, anchor=True, prefix=None,
                         is_regex=False):
@@ -356,7 +350,7 @@
     or just returned as-is (assumes it's a regex object).
     """
     if is_regex:
-        if isinstance(pattern, basestring):
+        if isinstance(pattern, str):
             return re.compile(pattern)
         else:
             return pattern
diff --git a/distutils2/markers.py b/distutils2/markers.py
--- a/distutils2/markers.py
+++ b/distutils2/markers.py
@@ -3,10 +3,9 @@
 import os
 import sys
 import platform
-from tokenize import generate_tokens, NAME, OP, STRING, ENDMARKER
-from StringIO import StringIO as BytesIO
 
-from distutils2.compat import python_implementation
+from tokenize import tokenize, NAME, OP, STRING, ENDMARKER, ENCODING
+from io import BytesIO
 
 __all__ = ['interpret']
 
@@ -33,11 +32,10 @@
          'os.name': os.name,
          'platform.version': platform.version(),
          'platform.machine': platform.machine(),
-         'platform.python_implementation': python_implementation(),
-        }
+         'platform.python_implementation': platform.python_implementation()}
 
 
-class _Operation(object):
+class _Operation:
 
     def __init__(self, execution_context=None):
         self.left = None
@@ -98,7 +96,7 @@
         return _operate(self.op, left, right)
 
 
-class _OR(object):
+class _OR:
     def __init__(self, left, right=None):
         self.left = left
         self.right = right
@@ -113,7 +111,7 @@
         return self.left() or self.right()
 
 
-class _AND(object):
+class _AND:
     def __init__(self, left, right=None):
         self.left = left
         self.right = right
@@ -133,10 +131,10 @@
     marker = marker.strip().encode()
     ops = []
     op_starting = True
-    for token in generate_tokens(BytesIO(marker).readline):
+    for token in tokenize(BytesIO(marker).readline):
         # Unpack token
         toktype, tokval, rowcol, line, logical_line = token
-        if toktype not in (NAME, OP, STRING, ENDMARKER):
+        if toktype not in (NAME, OP, STRING, ENDMARKER, ENCODING):
             raise SyntaxError('Type not supported "%s"' % tokval)
 
         if op_starting:
diff --git a/distutils2/metadata.py b/distutils2/metadata.py
--- a/distutils2/metadata.py
+++ b/distutils2/metadata.py
@@ -4,10 +4,9 @@
 """
 
 import re
-import codecs
 import logging
 
-from StringIO import StringIO
+from io import StringIO
 from email import message_from_file
 from distutils2 import logger
 from distutils2.markers import interpret
@@ -185,7 +184,7 @@
 
 _FILESAFE = re.compile('[^A-Za-z0-9.]+')
 
-class Metadata(object):
+class Metadata:
     """The metadata of a release.
 
     Supports versions 1.0, 1.1 and 1.2 (auto-detected). You can
@@ -218,7 +217,7 @@
         self._fields['Metadata-Version'] = _best_version(self._fields)
 
     def _write_field(self, file, name, value):
-        file.write(u'%s: %s\n' % (name, value))
+        file.write('%s: %s\n' % (name, value))
 
     def __getitem__(self, name):
         return self.get(name)
@@ -311,11 +310,8 @@
 
     def read(self, filepath):
         """Read the metadata values from a file path."""
-        fp = codecs.open(filepath, 'r', encoding='utf-8')
-        try:
+        with open(filepath, 'r', encoding='utf-8') as fp:
             self.read_file(fp)
-        finally:
-            fp.close()
 
     def read_file(self, fileob):
         """Read the metadata values from a file object."""
@@ -337,11 +333,8 @@
 
     def write(self, filepath):
         """Write the metadata fields to filepath."""
-        fp = codecs.open(filepath, 'w', encoding='utf-8')
-        try:
+        with open(filepath, 'w', encoding='utf-8') as fp:
             self.write_file(fp)
-        finally:
-            fp.close()
 
     def write_file(self, fileobject):
         """Write the PKG-INFO format data to a file object."""
@@ -404,13 +397,13 @@
 
         if ((name in _ELEMENTSFIELD or name == 'Platform') and
             not isinstance(value, (list, tuple))):
-            if isinstance(value, basestring):
+            if isinstance(value, str):
                 value = [v.strip() for v in value.split(',')]
             else:
                 value = []
         elif (name in _LISTFIELDS and
               not isinstance(value, (list, tuple))):
-            if isinstance(value, basestring):
+            if isinstance(value, str):
                 value = [value]
             else:
                 value = []
@@ -472,7 +465,7 @@
             valid, value = self._platform(self._fields[name])
             if not valid:
                 return []
-            if isinstance(value, basestring):
+            if isinstance(value, str):
                 return value.split(',')
         valid, value = self._platform(self._fields[name])
         if not valid:
@@ -559,7 +552,7 @@
         return data
 
     # Mapping API
-    # TODO could add iter* variants
+    # XXX these methods should return views or sets in 3.x
 
     def keys(self):
         return list(_version2fieldlist(self['Metadata-Version']))
diff --git a/distutils2/pypi/base.py b/distutils2/pypi/base.py
--- a/distutils2/pypi/base.py
+++ b/distutils2/pypi/base.py
@@ -3,7 +3,7 @@
 from distutils2.pypi.dist import ReleasesList
 
 
-class BaseClient(object):
+class BaseClient:
     """Base class containing common methods for the index crawlers/clients"""
 
     def __init__(self, prefer_final, prefer_source):
diff --git a/distutils2/pypi/dist.py b/distutils2/pypi/dist.py
--- a/distutils2/pypi/dist.py
+++ b/distutils2/pypi/dist.py
@@ -8,9 +8,12 @@
 """
 
 import re
+import hashlib
 import tempfile
-import urllib
-import urlparse
+import urllib.request
+import urllib.parse
+import urllib.error
+
 from distutils2.errors import IrrationalVersionError
 from distutils2.version import (suggest_normalized_version, NormalizedVersion,
                                 get_version_predicate)
@@ -18,10 +21,6 @@
 from distutils2.pypi.errors import (HashDoesNotMatch, UnsupportedHashName,
                                     CantParseArchiveName)
 from distutils2._backport.shutil import unpack_archive
-try:
-    import hashlib
-except ImportError:
-    from distutils2._backport import hashlib
 
 
 __all__ = ['ReleaseInfo', 'DistInfo', 'ReleasesList', 'get_infos_from_url']
@@ -31,7 +30,7 @@
 DIST_TYPES = ['bdist', 'sdist']
 
 
-class IndexReference(object):
+class IndexReference:
     """Mixin used to store the index reference"""
     def set_index(self, index=None):
         self._index = index
@@ -297,8 +296,8 @@
         # if we do not have downloaded it yet, do it.
         if self.downloaded_location is None:
             url = self.url['url']
-            archive_name = urlparse.urlparse(url)[2].split('/')[-1]
-            filename, headers = urllib.urlretrieve(url,
+            archive_name = urllib.parse.urlparse(url)[2].split('/')[-1]
+            filename, headers = urllib.request.urlretrieve(url,
                                                    path + "/" + archive_name)
             self.downloaded_location = filename
             self._check_md5(filename)
@@ -327,12 +326,9 @@
         hashname = self.url['hashname']
         expected_hashval = self.url['hashval']
         if None not in (expected_hashval, hashname):
-            f = open(filename, 'rb')
-            try:
+            with open(filename, 'rb') as f:
                 hashval = hashlib.new(hashname)
                 hashval.update(f.read())
-            finally:
-                f.close()
 
             if hashval.hexdigest() != expected_hashval:
                 raise HashDoesNotMatch("got %s instead of %s"
@@ -491,7 +487,7 @@
         url = url.replace("#md5=%s" % md5_hash, "")
 
     # parse the archive name to find dist name and version
-    archive_name = urlparse.urlparse(url)[2].split('/')[-1]
+    archive_name = urllib.parse.urlparse(url)[2].split('/')[-1]
     extension_matched = False
     # remove the extension from the name
     for ext in EXTENSIONS:
diff --git a/distutils2/pypi/simple.py b/distutils2/pypi/simple.py
--- a/distutils2/pypi/simple.py
+++ b/distutils2/pypi/simple.py
@@ -6,17 +6,18 @@
 reference implementation available at http://pypi.python.org/simple/).
 """
 
-import httplib
+import http.client
 import re
 import socket
 import sys
-import urllib2
-import urlparse
+import urllib.request
+import urllib.parse
+import urllib.error
 import os
 
 from fnmatch import translate
+from functools import wraps
 from distutils2 import logger
-from distutils2.compat import wraps
 from distutils2.metadata import Metadata
 from distutils2.version import get_version_predicate
 from distutils2 import __version__ as distutils2_version
@@ -123,7 +124,7 @@
         self.follow_externals = follow_externals
 
         # mirroring attributes.
-        parsed = urlparse.urlparse(index_url)
+        parsed = urllib.parse.urlparse(index_url)
         self.scheme = parsed[0]
         if self.scheme == 'file':
             ender = os.path.sep
@@ -156,20 +157,17 @@
 
         Return a list of names.
         """
-        if u'*' in name:
-            name.replace(u'*', u'.*')
+        if '*' in name:
+            name.replace('*', '.*')
         else:
-            name = u"%s%s%s" % (u'*.?', name, u'*.?')
-        name = name.replace(u'*', u'[^<]*')  # avoid matching end tag
-        pattern = (u'<a[^>]*>(%s)</a>' % name).encode('utf-8')
+            name = "%s%s%s" % ('*.?', name, '*.?')
+        name = name.replace('*', '[^<]*')  # avoid matching end tag
+        pattern = ('<a[^>]*>(%s)</a>' % name).encode('utf-8')
         projectname = re.compile(pattern, re.I)
         matching_projects = []
 
-        index = self._open_url(self.index_url)
-        try:
+        with self._open_url(self.index_url) as index:
             index_content = index.read()
-        finally:
-            index.close()
 
         for match in projectname.finditer(index_content):
             project_name = match.group(1).decode('utf-8')
@@ -231,10 +229,8 @@
         """
         self._mirrors_used.add(self.index_url)
         index_url = self._mirrors.pop()
-        # XXX use urlparse for a real check of missing scheme part
-        if not (index_url.startswith("http://") or
-                index_url.startswith("https://") or
-                index_url.startswith("file://")):
+        # XXX use urllib.parse for a real check of missing scheme part
+        if not index_url.startswith(("http://", "https://", "file://")):
             index_url = "http://%s" % index_url
 
         if not index_url.endswith("/simple"):
@@ -251,10 +247,10 @@
         # if _index_url is contained in the given URL, we are browsing the
         # index, and it's always "browsable".
         # local files are always considered browable resources
-        if self.index_url in url or urlparse.urlparse(url)[0] == "file":
+        if self.index_url in url or urllib.parse.urlparse(url)[0] == "file":
             return True
         elif self.follow_externals:
-            if self._allowed_hosts(urlparse.urlparse(url)[1]):  # 1 is netloc
+            if self._allowed_hosts(urllib.parse.urlparse(url)[1]):  # 1 is netloc
                 return True
             else:
                 return False
@@ -311,8 +307,7 @@
                              the links we find (eg. run recursively this
                              method on it)
         """
-        f = self._open_url(url)
-        try:
+        with self._open_url(url) as f:
             base_url = f.url
             if url not in self._processed_urls:
                 self._processed_urls.append(url)
@@ -325,7 +320,7 @@
                             try:
                                 infos = get_infos_from_url(link, project_name,
                                             is_external=self.index_url not in url)
-                            except CantParseArchiveName, e:
+                            except CantParseArchiveName as e:
                                 logger.warning(
                                     "version has not been parsed: %s", e)
                             else:
@@ -334,8 +329,6 @@
                             if self._is_browsable(link) and follow_links:
                                 self._process_url(link, project_name,
                                     follow_links=False)
-        finally:
-            f.close()
 
     def _get_link_matcher(self, url):
         """Returns the right link matcher function of the given url
@@ -346,7 +339,7 @@
             return self._default_link_matcher
 
     def _get_full_url(self, url, base_url):
-        return urlparse.urljoin(base_url, self._htmldecode(url))
+        return urllib.parse.urljoin(base_url, self._htmldecode(url))
 
     def _simple_link_matcher(self, content, base_url):
         """Yield all links with a rel="download" or rel="homepage".
@@ -402,11 +395,11 @@
         files support.
 
         """
-        scheme, netloc, path, params, query, frag = urlparse.urlparse(url)
+        scheme, netloc, path, params, query, frag = urllib.parse.urlparse(url)
 
         # authentication stuff
         if scheme in ('http', 'https'):
-            auth, host = urllib2.splituser(netloc)
+            auth, host = urllib.parse.splituser(netloc)
         else:
             auth = None
 
@@ -418,27 +411,27 @@
         # add authorization headers if auth is provided
         if auth:
             auth = "Basic " + \
-                urlparse.unquote(auth).encode('base64').strip()
-            new_url = urlparse.urlunparse((
+                urllib.parse.unquote(auth).encode('base64').strip()
+            new_url = urllib.parse.urlunparse((
                 scheme, host, path, params, query, frag))
-            request = urllib2.Request(new_url)
+            request = urllib.request.Request(new_url)
             request.add_header("Authorization", auth)
         else:
-            request = urllib2.Request(url)
+            request = urllib.request.Request(url)
         request.add_header('User-Agent', USER_AGENT)
         try:
-            fp = urllib2.urlopen(request)
-        except (ValueError, httplib.InvalidURL), v:
+            fp = urllib.request.urlopen(request)
+        except (ValueError, http.client.InvalidURL) as v:
             msg = ' '.join([str(arg) for arg in v.args])
             raise PackagingPyPIError('%s %s' % (url, msg))
-        except urllib2.HTTPError, v:
+        except urllib.error.HTTPError as v:
             return v
-        except urllib2.URLError, v:
+        except urllib.error.URLError as v:
             raise DownloadError("Download error for %s: %s" % (url, v.reason))
-        except httplib.BadStatusLine, v:
+        except http.client.BadStatusLine as v:
             raise DownloadError('%s returned a bad status line. '
                 'The server might be down, %s' % (url, v.line))
-        except httplib.HTTPException, v:
+        except http.client.HTTPException as v:
             raise DownloadError("Download error for %s: %s" % (url, v))
         except socket.timeout:
             raise DownloadError("The server timeouted")
@@ -447,9 +440,9 @@
             # Put authentication info back into request URL if same host,
             # so that links found on the page will work
             s2, h2, path2, param2, query2, frag2 = \
-                urlparse.urlparse(fp.url)
+                urllib.parse.urlparse(fp.url)
             if s2 == scheme and h2 == host:
-                fp.url = urlparse.urlunparse(
+                fp.url = urllib.parse.urlunparse(
                     (s2, netloc, path2, param2, query2, frag2))
         return fp
 
diff --git a/distutils2/pypi/wrapper.py b/distutils2/pypi/wrapper.py
--- a/distutils2/pypi/wrapper.py
+++ b/distutils2/pypi/wrapper.py
@@ -25,13 +25,13 @@
         exception = None
         methods = [func]
         for f in wrapper._indexes.values():
-            if f != func.im_self and hasattr(f, func.f_name):
-                methods.append(getattr(f, func.f_name))
+            if f != func.__self__ and hasattr(f, func.__name__):
+                methods.append(getattr(f, func.__name__))
         for method in methods:
             try:
                 response = method(*args, **kwargs)
                 retry = False
-            except Exception, e:
+            except Exception as e:
                 exception = e
             if not retry:
                 break
@@ -42,7 +42,7 @@
     return decorator
 
 
-class ClientWrapper(object):
+class ClientWrapper:
     """Wrapper around simple and xmlrpc clients,
 
     Choose the best implementation to use depending the needs, using the given
diff --git a/distutils2/pypi/xmlrpc.py b/distutils2/pypi/xmlrpc.py
--- a/distutils2/pypi/xmlrpc.py
+++ b/distutils2/pypi/xmlrpc.py
@@ -6,7 +6,7 @@
 implementation at http://wiki.python.org/moin/PyPiXmlRpc).
 """
 
-import xmlrpclib
+import xmlrpc.client
 
 from distutils2 import logger
 from distutils2.errors import IrrationalVersionError
@@ -171,7 +171,7 @@
                 project.add_release(release=ReleaseInfo(p['name'],
                     p['version'], metadata={'summary': p['summary']},
                     index=self._index))
-            except IrrationalVersionError, e:
+            except IrrationalVersionError as e:
                 logger.warning("Irrational version error found: %s", e)
         return [self._projects[p['name'].lower()] for p in projects]
 
@@ -195,6 +195,6 @@
 
         """
         if not hasattr(self, '_server_proxy'):
-            self._server_proxy = xmlrpclib.ServerProxy(self.server_url)
+            self._server_proxy = xmlrpc.client.ServerProxy(self.server_url)
 
         return self._server_proxy
diff --git a/distutils2/run.py b/distutils2/run.py
--- a/distutils2/run.py
+++ b/distutils2/run.py
@@ -74,7 +74,7 @@
     return optdict
 
 
-class action_help(object):
+class action_help:
     """Prints a help message when the standard help flags: -h and --help
     are used on the commandline.
     """
@@ -86,7 +86,7 @@
         def wrapper(*args, **kwargs):
             f_args = args[1]
             if '--help' in f_args or '-h' in f_args:
-                print self.help_msg
+                print(self.help_msg)
                 return
             return f(*args, **kwargs)
         return wrapper
@@ -132,7 +132,7 @@
     else:
         dists = get_distributions(use_egg_info=True)
         graph = generate_graph(dists)
-        print graph.repr_node(dist)
+        print(graph.repr_node(dist))
 
 
 @action_help("""\
@@ -205,13 +205,13 @@
 
     for key in keys:
         if key in metadata:
-            print metadata._convert_name(key) + ':'
+            print(metadata._convert_name(key) + ':')
             value = metadata[key]
             if isinstance(value, list):
                 for v in value:
-                    print '   ', v
+                    print('   ', v)
             else:
-                print '   ', value.replace('\n', '\n    ')
+                print('   ', value.replace('\n', '\n    '))
 
 
 @action_help("""\
@@ -257,14 +257,14 @@
     commands = STANDARD_COMMANDS  # + extra commands
 
     if args == ['--list-commands']:
-        print 'List of available commands:'
+        print('List of available commands:')
         cmds = sorted(commands)
 
         for cmd in cmds:
             cls = dispatcher.cmdclass.get(cmd) or get_command_class(cmd)
             desc = getattr(cls, 'description',
                             '(no description available)')
-            print '  %s: %s' % (cmd, desc)
+            print('  %s: %s' % (cmd, desc))
         return
 
     while args:
@@ -310,7 +310,7 @@
 
     number = 0
     for dist in results:
-        print '%r %s (from %r)' % (dist.name, dist.version, dist.path)
+        print('%r %s (from %r)' % (dist.name, dist.version, dist.path))
         number += 1
 
     if number == 0:
@@ -371,7 +371,7 @@
 ]
 
 
-class Dispatcher(object):
+class Dispatcher:
     """Reads the command-line options
     """
     def __init__(self, args=None):
@@ -445,7 +445,7 @@
         # it takes.
         try:
             cmd_class = get_command_class(command)
-        except PackagingModuleError, msg:
+        except PackagingModuleError as msg:
             raise PackagingArgError(msg)
 
         # XXX We want to push this in distutils2.command
@@ -545,18 +545,18 @@
         # late import because of mutual dependence between these modules
         from distutils2.command.cmd import Command
 
-        print 'Usage: pysetup [options] action [action_options]'
-        print
+        print('Usage: pysetup [options] action [action_options]')
+        print()
         if global_options_:
             self.print_usage(self.parser)
-            print
+            print()
 
         if display_options_:
             parser.set_option_table(display_options)
             parser.print_help(
                 "Information display options (just display " +
                 "information, ignore any commands)")
-            print
+            print()
 
         for command in commands:
             if isinstance(command, type) and issubclass(command, Command):
@@ -570,15 +570,15 @@
                 parser.set_option_table(cls.user_options)
 
             parser.print_help("Options for %r command:" % cls.__name__)
-            print
+            print()
 
     def _show_command_help(self, command):
-        if isinstance(command, basestring):
+        if isinstance(command, str):
             command = get_command_class(command)
 
         desc = getattr(command, 'description', '(no description available)')
-        print 'Description:', desc
-        print
+        print('Description:', desc)
+        print()
 
         if (hasattr(command, 'help_options') and
             isinstance(command.help_options, list)):
@@ -588,7 +588,7 @@
             self.parser.set_option_table(command.user_options)
 
         self.parser.print_help("Options:")
-        print
+        print()
 
     def _get_command_groups(self):
         """Helper function to retrieve all the command class names divided
@@ -615,7 +615,7 @@
 
         self.print_command_list(std_commands, "Standard commands", max_length)
         if extra_commands:
-            print
+            print()
             self.print_command_list(extra_commands, "Extra commands",
                                     max_length)
 
@@ -623,14 +623,14 @@
         """Print a subset of the list of all commands -- used by
         'print_commands()'.
         """
-        print header + ":"
+        print(header + ":")
 
         for cmd in commands:
             cls = self.cmdclass.get(cmd) or get_command_class(cmd)
             description = getattr(cls, 'description',
                                   '(no description available)')
 
-            print "  %-*s  %s" % (max_length, cmd, description)
+            print("  %-*s  %s" % (max_length, cmd, description))
 
     def __call__(self):
         if self.action is None:
@@ -646,17 +646,16 @@
     old_level = logger.level
     old_handlers = list(logger.handlers)
     try:
-        try:
-            dispatcher = Dispatcher(args)
-            if dispatcher.action is None:
-                return
-            return dispatcher()
-        except KeyboardInterrupt:
-            logger.info('interrupted')
-            return 1
-        except (IOError, os.error, PackagingError, CCompilerError), exc:
-            logger.exception(exc)
-            return 1
+        dispatcher = Dispatcher(args)
+        if dispatcher.action is None:
+            return
+        return dispatcher()
+    except KeyboardInterrupt:
+        logger.info('interrupted')
+        return 1
+    except (IOError, os.error, PackagingError, CCompilerError) as exc:
+        logger.exception(exc)
+        return 1
     finally:
         logger.setLevel(old_level)
         logger.handlers[:] = old_handlers
diff --git a/distutils2/tests/__init__.py b/distutils2/tests/__init__.py
--- a/distutils2/tests/__init__.py
+++ b/distutils2/tests/__init__.py
@@ -7,14 +7,14 @@
 
 Utility code is included in distutils2.tests.support.
 
-Always import unittest from this module, it will be the right version
+Always import unittest from this module: it will be unittest from the
 standard library for packaging tests and unittest2 for distutils2 tests.
 """
 
 import os
 import sys
 import unittest2 as unittest
-from StringIO import StringIO
+from io import StringIO
 
 # XXX move helpers to support, add tests for them, remove things that
 # duplicate test.support (or keep them for the backport; needs thinking)
@@ -42,7 +42,7 @@
     """Test failed."""
 
 
-class BasicTestRunner(object):
+class BasicTestRunner:
     def run(self, test):
         result = unittest.TestResult()
         test(result)
@@ -72,13 +72,13 @@
 def run_unittest(classes, verbose_=1):
     """Run tests from unittest.TestCase-derived classes.
 
-    Originally extracted from stdlib test.test_support and modified to
+    Originally extracted from stdlib test.support and modified to
     support unittest2.
     """
     valid_types = (unittest.TestSuite, unittest.TestCase)
     suite = unittest.TestSuite()
     for cls in classes:
-        if isinstance(cls, basestring):
+        if isinstance(cls, str):
             if cls in sys.modules:
                 suite.addTest(unittest.findTestCases(sys.modules[cls]))
             else:
diff --git a/distutils2/tests/__main__.py b/distutils2/tests/__main__.py
--- a/distutils2/tests/__main__.py
+++ b/distutils2/tests/__main__.py
@@ -4,7 +4,7 @@
 
 import os
 import sys
-from test.test_support import reap_children, reap_threads, run_unittest
+from test.support import reap_children, reap_threads, run_unittest
 
 from distutils2.tests import unittest
 
diff --git a/distutils2/tests/pypi_server.py b/distutils2/tests/pypi_server.py
--- a/distutils2/tests/pypi_server.py
+++ b/distutils2/tests/pypi_server.py
@@ -30,17 +30,15 @@
 """
 
 import os
-import Queue
+import queue
 import select
 import threading
-import SocketServer
-from BaseHTTPServer import HTTPServer
-from SimpleHTTPServer import SimpleHTTPRequestHandler
-from SimpleXMLRPCServer import SimpleXMLRPCServer
+import socketserver
+from functools import wraps
+from http.server import HTTPServer, SimpleHTTPRequestHandler
+from xmlrpc.server import SimpleXMLRPCServer
 
 from distutils2.tests import unittest
-from distutils2.compat import wraps
-
 
 
 PYPI_DEFAULT_STATIC_PATH = os.path.join(
@@ -116,7 +114,7 @@
             self.server = HTTPServer(('127.0.0.1', 0), PyPIRequestHandler)
             self.server.RequestHandlerClass.pypi_server = self
 
-            self.request_queue = Queue.Queue()
+            self.request_queue = queue.Queue()
             self._requests = []
             self.default_response_status = 404
             self.default_response_headers = [('Content-type', 'text/plain')]
@@ -153,7 +151,7 @@
     def stop(self):
         """self shutdown is not supported for python < 2.6"""
         self._run = False
-        if self.isAlive():
+        if self.is_alive():
             self.join()
         self.server.server_close()
 
@@ -170,7 +168,7 @@
         while True:
             try:
                 self._requests.append(self.request_queue.get_nowait())
-            except Queue.Empty:
+            except queue.Empty:
                 break
         return self._requests
 
@@ -222,18 +220,12 @@
                         relative_path += "index.html"
 
                     if relative_path.endswith('.tar.gz'):
-                        file = open(fs_path + relative_path, 'rb')
-                        try:
+                        with open(fs_path + relative_path, 'rb') as file:
                             data = file.read()
-                        finally:
-                            file.close()
                         headers = [('Content-type', 'application/x-gtar')]
                     else:
-                        file = open(fs_path + relative_path)
-                        try:
+                        with open(fs_path + relative_path) as file:
                             data = file.read().encode()
-                        finally:
-                            file.close()
                         headers = [('Content-type', 'text/html')]
 
                     headers.append(('Content-Length', len(data)))
@@ -269,7 +261,7 @@
             self.send_header(header, value)
         self.end_headers()
 
-        if isinstance(data, unicode):
+        if isinstance(data, str):
             data = data.encode('utf-8')
 
         self.wfile.write(data)
@@ -278,12 +270,12 @@
 class PyPIXMLRPCServer(SimpleXMLRPCServer):
     def server_bind(self):
         """Override server_bind to store the server name."""
-        SocketServer.TCPServer.server_bind(self)
+        socketserver.TCPServer.server_bind(self)
         host, port = self.socket.getsockname()[:2]
         self.server_port = port
 
 
-class MockDist(object):
+class MockDist:
     """Fake distribution, used in the Mock PyPI Server"""
 
     def __init__(self, name, version="1.0", hidden=False, url="http://url/",
@@ -398,7 +390,7 @@
         }
 
 
-class XMLRPCMockIndex(object):
+class XMLRPCMockIndex:
     """Mock XMLRPC server"""
 
     def __init__(self, dists=[]):
diff --git a/distutils2/tests/pypi_test_server.py b/distutils2/tests/pypi_test_server.py
--- a/distutils2/tests/pypi_test_server.py
+++ b/distutils2/tests/pypi_test_server.py
@@ -40,7 +40,7 @@
         self.addCleanup(self.pypi.stop)
 
 
-class PyPIServer(object):
+class PyPIServer:
     """Shim to access testpypi.python.org, for testing a real server."""
 
     def __init__(self, test_static_path=None,
diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -83,7 +83,7 @@
         self.buffer.append(record)
 
 
-class LoggingCatcher(object):
+class LoggingCatcher:
     """TestCase-compatible mixin to receive logging calls.
 
     Upon setUp, instances of this classes get a BufferingHandler that's
@@ -139,7 +139,7 @@
         return messages
 
 
-class TempdirManager(object):
+class TempdirManager:
     """TestCase-compatible mixin to create temporary directories and files.
 
     Directories and files created in a test_* method will be removed after it
@@ -182,11 +182,8 @@
         """
         if isinstance(path, (list, tuple)):
             path = os.path.join(*path)
-        f = codecs.open(path, 'w', encoding=encoding)
-        try:
+        with open(path, 'w', encoding=encoding) as f:
             f.write(content)
-        finally:
-            f.close()
 
     def create_dist(self, **kw):
         """Create a stub distribution object and files.
@@ -224,7 +221,7 @@
         self.assertFalse(os.path.isfile(path), "%r exists" % path)
 
 
-class EnvironRestorer(object):
+class EnvironRestorer:
     """TestCase-compatible mixin to restore or delete environment variables.
 
     The variables to restore (or delete if they were not originally present)
@@ -251,7 +248,7 @@
         super(EnvironRestorer, self).tearDown()
 
 
-class DummyCommand(object):
+class DummyCommand:
     """Class to store options for retrieval via set_undefined_options().
 
     Useful for mocking one dependency command in the tests for another
@@ -339,6 +336,11 @@
         cmd = build_ext(dist)
         support.fixup_build_ext(cmd)
         cmd.ensure_finalized()
+
+    In addition, this function also fixes cmd.distribution.include_dirs if
+    the running Python is an uninstalled Python 3.3.  (This fix is not done in
+    packaging, which does not need it, nor in distutils2 for Python 2, which
+    has no in-development version that can't be expected to be installed.)
     """
     if os.name == 'nt':
         cmd.debug = sys.executable.endswith('_d.exe')
@@ -354,12 +356,17 @@
             name, equals, value = runshared.partition('=')
             cmd.library_dirs = value.split(os.pathsep)
 
+    # Allow tests to run with an uninstalled Python 3.3
+    if sys.version_info[:2] == (3, 3) and sysconfig.is_python_build():
+        pysrcdir = sysconfig.get_config_var('projectbase')
+        cmd.distribution.include_dirs.append(os.path.join(pysrcdir, 'Include'))
+
 
 try:
-    from test.test_support import skip_unless_symlink
+    from test.support import skip_unless_symlink
 except ImportError:
     skip_unless_symlink = unittest.skip(
-        'requires test.test_support.skip_unless_symlink')
+        'requires test.support.skip_unless_symlink')
 
 
 requires_zlib = unittest.skipUnless(zlib, 'requires zlib')
@@ -368,7 +375,7 @@
 def unlink(filename):
     try:
         os.unlink(filename)
-    except OSError, error:
+    except OSError as error:
         # The filename need not exist.
         if error.errno not in (errno.ENOENT, errno.ENOTDIR):
             raise
@@ -381,7 +388,7 @@
     This will typically be run on the result of the communicate() method
     of a subprocess.Popen object.
     """
-    stderr = re.sub(r"\[\d+ refs\]\r?\n?$", "", stderr).strip()
+    stderr = re.sub(br"\[\d+ refs\]\r?\n?$", b"", stderr).strip()
     return stderr
 
 
diff --git a/distutils2/tests/test_command_build_clib.py b/distutils2/tests/test_command_build_clib.py
--- a/distutils2/tests/test_command_build_clib.py
+++ b/distutils2/tests/test_command_build_clib.py
@@ -68,7 +68,7 @@
         pkg_dir, dist = self.create_dist()
         cmd = build_clib(dist)
 
-        class FakeCompiler(object):
+        class FakeCompiler:
             def compile(*args, **kw):
                 pass
             create_static_lib = compile
diff --git a/distutils2/tests/test_command_build_ext.py b/distutils2/tests/test_command_build_ext.py
--- a/distutils2/tests/test_command_build_ext.py
+++ b/distutils2/tests/test_command_build_ext.py
@@ -3,7 +3,7 @@
 import site
 import shutil
 import textwrap
-from StringIO import StringIO
+from io import StringIO
 from distutils2.dist import Distribution
 from distutils2.errors import (UnknownFileError, CompileError,
                                PackagingPlatformError)
@@ -21,18 +21,13 @@
     def setUp(self):
         super(BuildExtTestCase, self).setUp()
         self.tmp_dir = self.mkdtemp()
-        if sys.version > "2.6":
-            self.old_user_base = site.USER_BASE
-            site.USER_BASE = self.mkdtemp()
+        self.old_user_base = site.USER_BASE
+        site.USER_BASE = self.mkdtemp()
 
     def tearDown(self):
-        if sys.version > "2.6":
-            site.USER_BASE = self.old_user_base
-
+        site.USER_BASE = self.old_user_base
         super(BuildExtTestCase, self).tearDown()
 
-    @unittest.skipIf(sys.version_info[:2] < (2, 6),
-                     "can't compile xxmodule successfully")
     def test_build_ext(self):
         support.copy_xxmodule_c(self.tmp_dir)
         xx_c = os.path.join(self.tmp_dir, 'xxmodule.c')
@@ -94,7 +89,6 @@
         # make sure we get some library dirs under solaris
         self.assertGreater(len(cmd.library_dirs), 0)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_user_site(self):
         dist = Distribution({'name': 'xx'})
         cmd = build_ext(dist)
@@ -362,8 +356,7 @@
 
         deptarget_c = os.path.join(self.tmp_dir, 'deptargetmodule.c')
 
-        fp = open(deptarget_c, 'w')
-        try:
+        with open(deptarget_c, 'w') as fp:
             fp.write(textwrap.dedent('''\
                 #include <AvailabilityMacros.h>
 
@@ -375,8 +368,6 @@
                 #endif
 
             ''' % operator))
-        finally:
-            fp.close()
 
         # get the deployment target that the interpreter was built with
         target = sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET')
diff --git a/distutils2/tests/test_command_build_py.py b/distutils2/tests/test_command_build_py.py
--- a/distutils2/tests/test_command_build_py.py
+++ b/distutils2/tests/test_command_build_py.py
@@ -61,7 +61,7 @@
         self.assertIn("__init__.py", files)
         self.assertIn("README.txt", files)
         # XXX even with -O, distutils writes pyc, not pyo; bug?
-        if getattr(sys , 'dont_write_bytecode', False):
+        if sys.dont_write_bytecode:
             self.assertNotIn("__init__.pyc", files)
         else:
             self.assertIn("__init__.pyc", files)
@@ -99,8 +99,6 @@
             os.chdir(cwd)
             sys.stdout = old_stdout
 
-    @unittest.skipUnless(hasattr(sys, 'dont_write_bytecode'),
-                         'sys.dont_write_bytecode not supported')
     def test_dont_write_bytecode(self):
         # makes sure byte_compile is not used
         pkg_dir, dist = self.create_dist()
diff --git a/distutils2/tests/test_command_build_scripts.py b/distutils2/tests/test_command_build_scripts.py
--- a/distutils2/tests/test_command_build_scripts.py
+++ b/distutils2/tests/test_command_build_scripts.py
@@ -71,11 +71,8 @@
         return expected
 
     def write_script(self, dir, name, text):
-        f = open(os.path.join(dir, name), "w")
-        try:
+        with open(os.path.join(dir, name), "w") as f:
             f.write(text)
-        finally:
-            f.close()
 
     def test_version_int(self):
         source = self.mkdtemp()
diff --git a/distutils2/tests/test_command_config.py b/distutils2/tests/test_command_config.py
--- a/distutils2/tests/test_command_config.py
+++ b/distutils2/tests/test_command_config.py
@@ -13,11 +13,8 @@
 
     def test_dump_file(self):
         this_file = __file__.rstrip('co')
-        f = open(this_file)
-        try:
+        with open(this_file) as f:
             numlines = len(f.readlines())
-        finally:
-            f.close()
 
         dump_file(this_file, 'I am the header')
 
diff --git a/distutils2/tests/test_command_install_dist.py b/distutils2/tests/test_command_install_dist.py
--- a/distutils2/tests/test_command_install_dist.py
+++ b/distutils2/tests/test_command_install_dist.py
@@ -72,7 +72,6 @@
         check_path(cmd.install_scripts, os.path.join(destination, "bin"))
         check_path(cmd.install_data, destination)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_user_site(self):
         # test install with --user
         # preparing the environment for the test
@@ -173,19 +172,18 @@
         cmd.home = 'home'
         self.assertRaises(PackagingOptionError, cmd.finalize_options)
 
-        if sys.version >= '2.6':
-            # can't combine user with with prefix/exec_prefix/home or
-            # install_(plat)base
-            cmd.prefix = None
-            cmd.user = 'user'
-            self.assertRaises(PackagingOptionError, cmd.finalize_options)
+        # can't combine user with with prefix/exec_prefix/home or
+        # install_(plat)base
+        cmd.prefix = None
+        cmd.user = 'user'
+        self.assertRaises(PackagingOptionError, cmd.finalize_options)
 
     def test_old_record(self):
         # test pre-PEP 376 --record option (outside dist-info dir)
         install_dir = self.mkdtemp()
         project_dir, dist = self.create_dist(scripts=['hello'])
         os.chdir(project_dir)
-        self.write_file('hello', "print 'o hai'")
+        self.write_file('hello', "print('o hai')")
 
         cmd = install_dist(dist)
         dist.command_obj['install_dist'] = cmd
@@ -194,11 +192,8 @@
         cmd.ensure_finalized()
         cmd.run()
 
-        f = open(cmd.record)
-        try:
+        with open(cmd.record) as f:
             content = f.read()
-        finally:
-            f.close()
 
         found = [os.path.basename(line) for line in content.splitlines()]
         expected = ['hello', 'METADATA', 'INSTALLER', 'REQUESTED', 'RECORD']
@@ -207,8 +202,6 @@
         # XXX test that fancy_getopt is okay with options named
         # record and no-record but unrelated
 
-    @unittest.skipIf(sys.version_info[:2] < (2, 6),
-                     "can't compile xxmodule successfully")
     def test_old_record_extensions(self):
         # test pre-PEP 376 --record option with ext modules
         install_dir = self.mkdtemp()
@@ -229,11 +222,8 @@
         cmd.ensure_finalized()
         cmd.run()
 
-        f = open(cmd.record)
-        try:
+        with open(cmd.record) as f:
             content = f.read()
-        finally:
-            f.close()
 
         found = [os.path.basename(line) for line in content.splitlines()]
         expected = [_make_ext_name('xx'),
diff --git a/distutils2/tests/test_command_install_distinfo.py b/distutils2/tests/test_command_install_distinfo.py
--- a/distutils2/tests/test_command_install_distinfo.py
+++ b/distutils2/tests/test_command_install_distinfo.py
@@ -2,7 +2,7 @@
 
 import os
 import csv
-import codecs
+import hashlib
 
 from distutils2.command.install_distinfo import install_distinfo
 from distutils2.command.cmd import Command
@@ -10,10 +10,6 @@
 from distutils2.metadata import Metadata
 from distutils2.tests import unittest, support
 from distutils2._backport import sysconfig
-try:
-    import hashlib
-except:
-    from distutils2._backport import hashlib
 
 
 class DummyInstallCmd(Command):
@@ -59,18 +55,10 @@
         dist_info = os.path.join(install_dir, 'foo-1.0.dist-info')
         self.checkLists(os.listdir(dist_info),
                         ['METADATA', 'RECORD', 'REQUESTED', 'INSTALLER'])
-        fp = open(os.path.join(dist_info, 'INSTALLER'))
-        try:
+        with open(os.path.join(dist_info, 'INSTALLER')) as fp:
             self.assertEqual(fp.read(), 'distutils')
-        finally:
-            fp.close()
-
-        fp = open(os.path.join(dist_info, 'REQUESTED'))
-        try:
+        with open(os.path.join(dist_info, 'REQUESTED')) as fp:
             self.assertEqual(fp.read(), '')
-        finally:
-            fp.close()
-
         meta_path = os.path.join(dist_info, 'METADATA')
         self.assertTrue(Metadata(path=meta_path).check())
 
@@ -91,11 +79,8 @@
         cmd.run()
 
         dist_info = os.path.join(install_dir, 'foo-1.0.dist-info')
-        fp = open(os.path.join(dist_info, 'INSTALLER'))
-        try:
+        with open(os.path.join(dist_info, 'INSTALLER')) as fp:
             self.assertEqual(fp.read(), 'bacon-python')
-        finally:
-            fp.close()
 
     def test_requested(self):
         pkg_dir, dist = self.create_dist(name='foo',
@@ -171,21 +156,15 @@
         # platform-dependent (line endings)
         metadata = os.path.join(modules_dest, 'Spamlib-0.1.dist-info',
                                 'METADATA')
-        fp = open(metadata, 'rb')
-        try:
+        with open(metadata, 'rb') as fp:
             content = fp.read()
-        finally:
-            fp.close()
 
         metadata_size = str(len(content))
         metadata_md5 = hashlib.md5(content).hexdigest()
 
         record = os.path.join(modules_dest, 'Spamlib-0.1.dist-info', 'RECORD')
-        fp = codecs.open(record, encoding='utf-8')
-        try:
+        with open(record, encoding='utf-8') as fp:
             content = fp.read()
-        finally:
-            fp.close()
 
         found = []
         for line in content.splitlines():
@@ -240,29 +219,23 @@
 
         expected = []
         for f in install.get_outputs():
-            if (f.endswith('.pyc') or f.endswith('.pyo') or f == os.path.join(
+            if (f.endswith(('.pyc', '.pyo')) or f == os.path.join(
                 install_dir, 'foo-1.0.dist-info', 'RECORD')):
                 expected.append([f, '', ''])
             else:
                 size = os.path.getsize(f)
                 md5 = hashlib.md5()
-                fp = open(f, 'rb')
-                try:
+                with open(f, 'rb') as fp:
                     md5.update(fp.read())
-                finally:
-                    fp.close()
                 hash = md5.hexdigest()
                 expected.append([f, hash, str(size)])
 
         parsed = []
-        f = open(os.path.join(dist_info, 'RECORD'), 'r')
-        try:
+        with open(os.path.join(dist_info, 'RECORD'), 'r') as f:
             reader = csv.reader(f, delimiter=',',
                                    lineterminator=os.linesep,
                                    quotechar='"')
             parsed = list(reader)
-        finally:
-            f.close()
 
         self.maxDiff = None
         self.checkLists(parsed, expected)
diff --git a/distutils2/tests/test_command_install_lib.py b/distutils2/tests/test_command_install_lib.py
--- a/distutils2/tests/test_command_install_lib.py
+++ b/distutils2/tests/test_command_install_lib.py
@@ -33,8 +33,7 @@
         cmd.finalize_options()
         self.assertEqual(cmd.optimize, 2)
 
-    @unittest.skipIf(getattr(sys, 'dont_write_bytecode', False),
-                     'byte-compile disabled')
+    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
     def test_byte_compile(self):
         pkg_dir, dist = self.create_dist()
         cmd = install_lib(dist)
@@ -83,8 +82,6 @@
         # get_input should return 2 elements
         self.assertEqual(len(cmd.get_inputs()), 2)
 
-    @unittest.skipUnless(hasattr(sys, 'dont_write_bytecode'),
-                         'sys.dont_write_bytecode not supported')
     def test_dont_write_bytecode(self):
         # makes sure byte_compile is not used
         pkg_dir, dist = self.create_dist()
diff --git a/distutils2/tests/test_command_install_scripts.py b/distutils2/tests/test_command_install_scripts.py
--- a/distutils2/tests/test_command_install_scripts.py
+++ b/distutils2/tests/test_command_install_scripts.py
@@ -38,11 +38,8 @@
 
         def write_script(name, text):
             expected.append(name)
-            f = open(os.path.join(source, name), "w")
-            try:
+            with open(os.path.join(source, name), "w") as f:
                 f.write(text)
-            finally:
-                f.close()
 
         write_script("script1.py", ("#! /usr/bin/env python2.3\n"
                                     "# bogus script w/ Python sh-bang\n"
diff --git a/distutils2/tests/test_command_register.py b/distutils2/tests/test_command_register.py
--- a/distutils2/tests/test_command_register.py
+++ b/distutils2/tests/test_command_register.py
@@ -1,8 +1,10 @@
-# encoding: utf-8
 """Tests for distutils2.command.register."""
 import os
 import getpass
-import urllib2
+import urllib.request
+import urllib.error
+import urllib.parse
+
 try:
     import docutils
     DOCUTILS_SUPPORT = True
@@ -36,7 +38,7 @@
 """
 
 
-class Inputs(object):
+class Inputs:
     """Fakes user inputs."""
     def __init__(self, *answers):
         self.answers = answers
@@ -49,7 +51,7 @@
             self.index += 1
 
 
-class FakeOpener(object):
+class FakeOpener:
     """Fakes a PyPI server"""
     def __init__(self):
         self.reqs = []
@@ -85,12 +87,12 @@
             return 'password'
 
         getpass.getpass = _getpass
-        self.old_opener = urllib2.build_opener
-        self.conn = urllib2.build_opener = FakeOpener()
+        self.old_opener = urllib.request.build_opener
+        self.conn = urllib.request.build_opener = FakeOpener()
 
     def tearDown(self):
         getpass.getpass = self._old_getpass
-        urllib2.build_opener = self.old_opener
+        urllib.request.build_opener = self.old_opener
         if hasattr(register_module, 'input'):
             del register_module.input
         super(RegisterTestCase, self).tearDown()
@@ -121,7 +123,7 @@
         # Password : 'password'
         # Save your login (y/N)? : 'y'
         inputs = Inputs('1', 'tarek', 'y')
-        register_module.raw_input = inputs
+        register_module.input = inputs
         cmd.ensure_finalized()
         cmd.run()
 
@@ -129,11 +131,8 @@
         self.assertTrue(os.path.exists(self.rc))
 
         # with the content similar to WANTED_PYPIRC
-        fp = open(self.rc)
-        try:
+        with open(self.rc) as fp:
             content = fp.read()
-        finally:
-            fp.close()
         self.assertEqual(content, WANTED_PYPIRC)
 
         # now let's make sure the .pypirc file generated
@@ -153,7 +152,7 @@
         req1 = dict(self.conn.reqs[0].headers)
         req2 = dict(self.conn.reqs[1].headers)
         self.assertEqual(req2['Content-length'], req1['Content-length'])
-        self.assertIn('xxx', self.conn.reqs[1].data)
+        self.assertIn(b'xxx', self.conn.reqs[1].data)
 
     def test_password_not_in_file(self):
 
@@ -171,7 +170,7 @@
         # this test runs choice 2
         cmd = self._get_cmd()
         inputs = Inputs('2', 'tarek', 'tarek at ziade.org')
-        register_module.raw_input = inputs
+        register_module.input = inputs
         # let's run the command
         # FIXME does this send a real request? use a mock server
         cmd.ensure_finalized()
@@ -182,13 +181,13 @@
         req = self.conn.reqs[0]
         headers = dict(req.headers)
         self.assertEqual(headers['Content-length'], '628')
-        self.assertIn('tarek', req.data)
+        self.assertIn(b'tarek', req.data)
 
     def test_password_reset(self):
         # this test runs choice 3
         cmd = self._get_cmd()
         inputs = Inputs('3', 'tarek at ziade.org')
-        register_module.raw_input = inputs
+        register_module.input = inputs
         cmd.ensure_finalized()
         cmd.run()
 
@@ -197,7 +196,7 @@
         req = self.conn.reqs[0]
         headers = dict(req.headers)
         self.assertEqual(headers['Content-length'], '298')
-        self.assertIn('tarek', req.data)
+        self.assertIn(b'tarek', req.data)
 
     @unittest.skipUnless(DOCUTILS_SUPPORT, 'needs docutils')
     def test_strict(self):
@@ -211,7 +210,7 @@
         cmd.ensure_finalized()
         cmd.strict = True
         inputs = Inputs('1', 'tarek', 'y')
-        register_module.raw_input = inputs
+        register_module.input = inputs
         self.assertRaises(PackagingSetupError, cmd.run)
 
         # metadata is OK but long_description is broken
@@ -232,7 +231,7 @@
         cmd.ensure_finalized()
         cmd.strict = True
         inputs = Inputs('1', 'tarek', 'y')
-        register_module.raw_input = inputs
+        register_module.input = inputs
         cmd.ensure_finalized()
         cmd.run()
 
@@ -240,7 +239,7 @@
         cmd = self._get_cmd()
         cmd.ensure_finalized()
         inputs = Inputs('1', 'tarek', 'y')
-        register_module.raw_input = inputs
+        register_module.input = inputs
         cmd.ensure_finalized()
         cmd.run()
 
diff --git a/distutils2/tests/test_command_sdist.py b/distutils2/tests/test_command_sdist.py
--- a/distutils2/tests/test_command_sdist.py
+++ b/distutils2/tests/test_command_sdist.py
@@ -214,11 +214,8 @@
         self.assertEqual(len(content), 10)
 
         # Checking the MANIFEST
-        fp = open(join(self.tmp_dir, 'MANIFEST'))
-        try:
+        with open(join(self.tmp_dir, 'MANIFEST')) as fp:
             manifest = fp.read()
-        finally:
-            fp.close()
         self.assertEqual(manifest, MANIFEST % {'sep': os.sep})
 
     @requires_zlib
@@ -334,12 +331,9 @@
 
         # Should produce four lines. Those lines are one comment, one default
         # (README) and two package files.
-        f = open(cmd.manifest)
-        try:
+        with open(cmd.manifest) as f:
             manifest = [line.strip() for line in f.read().split('\n')
                         if line.strip() != '']
-        finally:
-            f.close()
         self.assertEqual(len(manifest), 3)
 
         # Adding a file
@@ -352,12 +346,9 @@
 
         cmd.run()
 
-        f = open(cmd.manifest)
-        try:
+        with open(cmd.manifest) as f:
             manifest2 = [line.strip() for line in f.read().split('\n')
                          if line.strip() != '']
-        finally:
-            f.close()
 
         # Do we have the new file in MANIFEST?
         self.assertEqual(len(manifest2), 4)
@@ -370,12 +361,9 @@
         cmd.ensure_finalized()
         cmd.run()
 
-        f = open(cmd.manifest)
-        try:
+        with open(cmd.manifest) as f:
             manifest = [line.strip() for line in f.read().split('\n')
                         if line.strip() != '']
-        finally:
-            f.close()
 
         self.assertEqual(manifest[0],
                          '# file GENERATED by distutils2, do NOT edit')
@@ -388,12 +376,9 @@
         self.write_file((self.tmp_dir, cmd.manifest), 'README.manual')
         cmd.run()
 
-        f = open(cmd.manifest)
-        try:
+        with open(cmd.manifest) as f:
             manifest = [line.strip() for line in f.read().split('\n')
                         if line.strip() != '']
-        finally:
-            f.close()
 
         self.assertEqual(manifest, ['README.manual'])
 
@@ -404,11 +389,8 @@
         cmd.ensure_finalized()
         self.write_file((self.tmp_dir, 'yeah'), 'xxx')
         cmd.run()
-        f = open(cmd.manifest)
-        try:
+        with open(cmd.manifest) as f:
             content = f.read()
-        finally:
-            f.close()
 
         self.assertIn('yeah', content)
 
diff --git a/distutils2/tests/test_command_test.py b/distutils2/tests/test_command_test.py
--- a/distutils2/tests/test_command_test.py
+++ b/distutils2/tests/test_command_test.py
@@ -113,7 +113,7 @@
         record = []
         a_module.recorder = lambda *args: record.append("suite")
 
-        class MockTextTestRunner(object):
+        class MockTextTestRunner:
             def __init__(*_, **__):
                 pass
 
@@ -177,7 +177,7 @@
         self.assertEqual(["runner called"], record)
 
     def prepare_mock_ut2(self):
-        class MockUTClass(object):
+        class MockUTClass:
             def __init__(*_, **__):
                 pass
 
@@ -187,7 +187,7 @@
             def run(self, _):
                 pass
 
-        class MockUTModule(object):
+        class MockUTModule:
             TestLoader = MockUTClass
             TextTestRunner = MockUTClass
 
diff --git a/distutils2/tests/test_command_upload.py b/distutils2/tests/test_command_upload.py
--- a/distutils2/tests/test_command_upload.py
+++ b/distutils2/tests/test_command_upload.py
@@ -1,4 +1,3 @@
-# encoding: utf-8
 """Tests for distutils2.command.upload."""
 import os
 
@@ -44,6 +43,7 @@
 """
 
 
+ at unittest.skipIf(threading is None, 'needs threading')
 class UploadTestCase(support.TempdirManager, support.EnvironRestorer,
                      support.LoggingCatcher, PyPIServerTestCase):
 
@@ -112,8 +112,8 @@
         # what did we send?
         handler, request_data = self.pypi.requests[-1]
         headers = handler.headers
-        self.assertIn('d?d?', request_data)
-        self.assertIn('xxx', request_data)
+        self.assertIn('d?d?'.encode('utf-8'), request_data)
+        self.assertIn(b'xxx', request_data)
 
         self.assertEqual(int(headers['content-length']), len(request_data))
         self.assertLess(int(headers['content-length']), 2500)
@@ -148,12 +148,8 @@
             "----------------GHSKFJDLGDS7543FJKLFHRE75642756743254"
             .encode())[1:4]
 
-        self.assertIn('name=":action"', action)
-        self.assertIn('doc_upload', action)
-
-
-UploadTestCase = unittest.skipIf(threading is None, 'needs threading')(
-        UploadTestCase)
+        self.assertIn(b'name=":action"', action)
+        self.assertIn(b'doc_upload', action)
 
 
 def test_suite():
diff --git a/distutils2/tests/test_command_upload_docs.py b/distutils2/tests/test_command_upload_docs.py
--- a/distutils2/tests/test_command_upload_docs.py
+++ b/distutils2/tests/test_command_upload_docs.py
@@ -32,6 +32,7 @@
 """
 
 
+ at unittest.skipIf(threading is None, "Needs threading")
 class UploadDocsTestCase(support.TempdirManager,
                          support.EnvironRestorer,
                          support.LoggingCatcher,
@@ -94,39 +95,39 @@
 
         self.assertEqual(len(self.pypi.requests), 1)
         handler, request_data = self.pypi.requests[-1]
-        self.assertIn("content", request_data)
+        self.assertIn(b"content", request_data)
         self.assertIn("Basic", handler.headers['authorization'])
         self.assertTrue(handler.headers['content-type']
             .startswith('multipart/form-data;'))
 
         action, name, version, content = request_data.split(
-            '----------------GHSKFJDLGDS7543FJKLFHRE75642756743254')[1:5]
+            b'----------------GHSKFJDLGDS7543FJKLFHRE75642756743254')[1:5]
 
         # check that we picked the right chunks
-        self.assertIn('name=":action"', action)
-        self.assertIn('name="name"', name)
-        self.assertIn('name="version"', version)
-        self.assertIn('name="content"', content)
+        self.assertIn(b'name=":action"', action)
+        self.assertIn(b'name="name"', name)
+        self.assertIn(b'name="version"', version)
+        self.assertIn(b'name="content"', content)
 
         # check their contents
-        self.assertIn('doc_upload', action)
-        self.assertIn('distr-name', name)
-        self.assertIn('docs/index.html', content)
-        self.assertIn('Ce mortel ennui', content)
+        self.assertIn(b'doc_upload', action)
+        self.assertIn(b'distr-name', name)
+        self.assertIn(b'docs/index.html', content)
+        self.assertIn(b'Ce mortel ennui', content)
 
     @unittest.skipIf(_ssl is None, 'Needs SSL support')
     def test_https_connection(self):
         self.https_called = False
         self.addCleanup(
-            setattr, upload_docs_mod.httplib, 'HTTPSConnection',
-            upload_docs_mod.httplib.HTTPSConnection)
+            setattr, upload_docs_mod.http.client, 'HTTPSConnection',
+            upload_docs_mod.http.client.HTTPSConnection)
 
         def https_conn_wrapper(*args):
             self.https_called = True
             # the testing server is http
-            return upload_docs_mod.httplib.HTTPConnection(*args)
+            return upload_docs_mod.http.client.HTTPConnection(*args)
 
-        upload_docs_mod.httplib.HTTPSConnection = https_conn_wrapper
+        upload_docs_mod.http.client.HTTPSConnection = https_conn_wrapper
 
         self.prepare_command()
         self.cmd.run()
@@ -174,9 +175,6 @@
         self.assertTrue(record, "should report the response")
         self.assertIn(self.pypi.default_response_data, record)
 
-UploadDocsTestCase = unittest.skipIf(threading is None, "Needs threading")(
-    UploadDocsTestCase)
-
 
 def test_suite():
     return unittest.makeSuite(UploadDocsTestCase)
diff --git a/distutils2/tests/test_compiler.py b/distutils2/tests/test_compiler.py
--- a/distutils2/tests/test_compiler.py
+++ b/distutils2/tests/test_compiler.py
@@ -6,7 +6,7 @@
 from distutils2.tests import unittest, support
 
 
-class FakeCompiler(object):
+class FakeCompiler:
 
     name = 'fake'
     description = 'Fake'
@@ -36,7 +36,7 @@
         os.environ['ARFLAGS'] = '-arflags'
 
         # make sure AR gets caught
-        class compiler(object):
+        class compiler:
             name = 'unix'
 
             def set_executables(self, **kw):
diff --git a/distutils2/tests/test_config.py b/distutils2/tests/test_config.py
--- a/distutils2/tests/test_config.py
+++ b/distutils2/tests/test_config.py
@@ -1,9 +1,8 @@
-# encoding: utf-8
 """Tests for distutils2.config."""
 import os
 import sys
 import logging
-from StringIO import StringIO
+from io import StringIO
 
 from distutils2 import command
 from distutils2.dist import Distribution
@@ -15,7 +14,7 @@
 from distutils2.tests.support import requires_zlib
 
 
-SETUP_CFG = u"""
+SETUP_CFG = """
 [metadata]
 name = RestingParrot
 version = 0.6.4
@@ -161,7 +160,7 @@
 """
 
 
-class DCompiler(object):
+class DCompiler:
     name = 'd'
     description = 'D Compiler'
 
@@ -181,7 +180,7 @@
     config['files']['modules'] += '\n third'
 
 
-class FooBarBazTest(object):
+class FooBarBazTest:
 
     def __init__(self, dist):
         self.distribution = dist
@@ -474,11 +473,8 @@
         cmd.finalize_options()
         cmd.get_file_list()
         cmd.make_distribution()
-        fp = open('MANIFEST')
-        try:
+        with open('MANIFEST') as fp:
             self.assertIn('README\nREADME2\n', fp.read())
-        finally:
-            fp.close()
 
     def test_sub_commands(self):
         self.write_setup()
diff --git a/distutils2/tests/test_create.py b/distutils2/tests/test_create.py
--- a/distutils2/tests/test_create.py
+++ b/distutils2/tests/test_create.py
@@ -1,9 +1,7 @@
-# encoding: utf-8
 """Tests for distutils2.create."""
 import os
 import sys
-import codecs
-from StringIO import StringIO
+from io import StringIO
 from textwrap import dedent
 from distutils2.create import MainProgram, ask_yn, ask, main
 from distutils2._backport import sysconfig
@@ -98,7 +96,7 @@
 
     def test_convert_setup_py_to_cfg(self):
         self.write_file((self.wdir, 'setup.py'),
-                        dedent(u"""
+                        dedent("""
         # coding: utf-8
         from distutils.core import setup
 
@@ -133,18 +131,15 @@
               scripts=['my_script', 'bin/run'],
               )
         """), encoding='utf-8')
-        sys.stdin.write(u'y\n')
+        sys.stdin.write('y\n')
         sys.stdin.seek(0)
         main()
 
         path = os.path.join(self.wdir, 'setup.cfg')
-        fp = codecs.open(path, encoding='utf-8')
-        try:
+        with open(path, encoding='utf-8') as fp:
             contents = fp.read()
-        finally:
-            fp.close()
 
-        self.assertEqual(contents, dedent(u"""\
+        self.assertEqual(contents, dedent("""\
             [metadata]
             name = pyxfoil
             version = 0.2
@@ -184,14 +179,11 @@
 
     def test_convert_setup_py_to_cfg_with_description_in_readme(self):
         self.write_file((self.wdir, 'setup.py'),
-                        dedent(u"""
+                        dedent("""
         # coding: utf-8
         from distutils.core import setup
-        fp = open('README.txt')
-        try:
+        with open('README.txt') as fp:
             long_description = fp.read()
-        finally:
-            fp.close()
 
         setup(name='pyxfoil',
               version='0.2',
@@ -221,13 +213,10 @@
         main()
 
         path = os.path.join(self.wdir, 'setup.cfg')
-        fp = codecs.open(path, encoding='utf-8')
-        try:
+        with open(path, encoding='utf-8') as fp:
             contents = fp.read()
-        finally:
-            fp.close()
 
-        self.assertEqual(contents, dedent(u"""\
+        self.assertEqual(contents, dedent("""\
             [metadata]
             name = pyxfoil
             version = 0.2
diff --git a/distutils2/tests/test_database.py b/distutils2/tests/test_database.py
--- a/distutils2/tests/test_database.py
+++ b/distutils2/tests/test_database.py
@@ -1,12 +1,10 @@
 import os
+import io
 import csv
 import sys
 import shutil
 import tempfile
-try:
-    from hashlib import md5
-except ImportError:
-    from distutils2._backport.hashlib import md5
+from hashlib import md5
 from textwrap import dedent
 
 from distutils2.tests.test_util import GlobTestCaseBase
@@ -28,11 +26,8 @@
 
 
 def get_hexdigest(filename):
-    fp = open(filename, 'rb')
-    try:
-        checksum = md5(fp.read())
-    finally:
-        fp.close()
+    with open(filename, 'rb') as file:
+        checksum = md5(file.read())
     return checksum.hexdigest()
 
 
@@ -43,7 +38,7 @@
     return path, digest, size
 
 
-class FakeDistsMixin(object):
+class FakeDistsMixin:
 
     def setUp(self):
         super(FakeDistsMixin, self).setUp()
@@ -65,11 +60,11 @@
         # shutil gives no control over the mode of directories :(
         # see http://bugs.python.org/issue1666318
         for root, dirs, files in os.walk(self.fake_dists_path):
-            os.chmod(root, 0755)
+            os.chmod(root, 0o755)
             for f in files:
-                os.chmod(os.path.join(root, f), 0644)
+                os.chmod(os.path.join(root, f), 0o644)
             for d in dirs:
-                os.chmod(os.path.join(root, d), 0755)
+                os.chmod(os.path.join(root, d), 0o755)
 
 
 class CommonDistributionTests(FakeDistsMixin):
@@ -138,10 +133,9 @@
         for distinfo_dir in self.dirs:
 
             record_file = os.path.join(distinfo_dir, 'RECORD')
-            fp = open(record_file, 'w')
-            try:
+            with open(record_file, 'w') as file:
                 record_writer = csv.writer(
-                    fp, delimiter=',', quoting=csv.QUOTE_NONE,
+                    file, delimiter=',', quoting=csv.QUOTE_NONE,
                     lineterminator='\n')
 
                 dist_location = distinfo_dir.replace('.dist-info', '')
@@ -152,12 +146,9 @@
                 for file in ('INSTALLER', 'METADATA', 'REQUESTED'):
                     record_writer.writerow(record_pieces((distinfo_dir, file)))
                 record_writer.writerow([record_file])
-            finally:
-                fp.close()
 
-            fp = open(record_file)
-            try:
-                record_reader = csv.reader(fp, lineterminator='\n')
+            with open(record_file) as file:
+                record_reader = csv.reader(file, lineterminator='\n')
                 record_data = {}
                 for row in record_reader:
                     if row == []:
@@ -165,8 +156,6 @@
                     path, md5_, size = (row[:] +
                                         [None for i in range(len(row), 3)])
                     record_data[path] = md5_, size
-            finally:
-                fp.close()
             self.records[distinfo_dir] = record_data
 
     def test_instantiation(self):
@@ -210,14 +199,11 @@
         ]
 
         for distfile in distinfo_files:
-            value = dist.get_distinfo_file(distfile)
-            try:
-                self.assertIsInstance(value, file)
+            with dist.get_distinfo_file(distfile) as value:
+                self.assertIsInstance(value, io.TextIOWrapper)
                 # Is it the correct file?
                 self.assertEqual(value.name,
                                  os.path.join(distinfo_dir, distfile))
-            finally:
-                value.close()
 
         # Test an absolute path that is part of another distributions dist-info
         other_distinfo_file = os.path.join(
@@ -640,8 +626,7 @@
         metadata_path = os.path.join(dist_info, 'METADATA')
         resources_path = os.path.join(dist_info, 'RESOURCES')
 
-        fp = open(metadata_path, 'w')
-        try:
+        with open(metadata_path, 'w') as fp:
             fp.write(dedent("""\
                 Metadata-Version: 1.2
                 Name: test
@@ -649,25 +634,18 @@
                 Summary: test
                 Author: me
                 """))
-        finally:
-            fp.close()
+
         test_path = 'test.cfg'
 
         fd, test_resource_path = tempfile.mkstemp()
         os.close(fd)
         self.addCleanup(os.remove, test_resource_path)
 
-        fp = open(test_resource_path, 'w')
-        try:
+        with open(test_resource_path, 'w') as fp:
             fp.write('Config')
-        finally:
-            fp.close()
 
-        fp = open(resources_path, 'w')
-        try:
+        with open(resources_path, 'w') as fp:
             fp.write('%s,%s' % (test_path, test_resource_path))
-        finally:
-            fp.close()
 
         # Add fake site-packages to sys.path to retrieve fake dist
         self.addCleanup(sys.path.remove, temp_site_packages)
@@ -682,11 +660,8 @@
                          test_resource_path)
         self.assertRaises(KeyError, get_file_path, dist_name, 'i-dont-exist')
 
-        fp = get_file(dist_name, test_path)
-        try:
+        with get_file(dist_name, test_path) as fp:
             self.assertEqual(fp.read(), 'Config')
-        finally:
-            fp.close()
         self.assertRaises(KeyError, get_file, dist_name, 'i-dont-exist')
 
 
diff --git a/distutils2/tests/test_depgraph.py b/distutils2/tests/test_depgraph.py
--- a/distutils2/tests/test_depgraph.py
+++ b/distutils2/tests/test_depgraph.py
@@ -2,7 +2,7 @@
 import os
 import re
 import sys
-from StringIO import StringIO
+from io import StringIO
 
 from distutils2 import depgraph
 from distutils2.database import get_distribution, enable_cache, disable_cache
diff --git a/distutils2/tests/test_dist.py b/distutils2/tests/test_dist.py
--- a/distutils2/tests/test_dist.py
+++ b/distutils2/tests/test_dist.py
@@ -1,7 +1,6 @@
 """Tests for distutils2.dist."""
 import os
 import sys
-import codecs
 import logging
 import textwrap
 
@@ -52,12 +51,9 @@
     def test_debug_mode(self):
         tmpdir = self.mkdtemp()
         setupcfg = os.path.join(tmpdir, 'setup.cfg')
-        f = open(setupcfg, "w")
-        try:
+        with open(setupcfg, "w") as f:
             f.write("[global]\n")
             f.write("command_packages = foo.bar, splat")
-        finally:
-            f.close()
 
         files = [setupcfg]
         sys.argv.append("build")
@@ -128,11 +124,8 @@
 
         temp_dir = self.mkdtemp()
         user_filename = os.path.join(temp_dir, user_filename)
-        f = open(user_filename, 'w')
-        try:
+        with open(user_filename, 'w') as f:
             f.write('.')
-        finally:
-            f.close()
 
         dist = Distribution()
 
@@ -148,11 +141,8 @@
         else:
             user_filename = os.path.join(temp_home, "pydistutils.cfg")
 
-        f = open(user_filename, 'w')
-        try:
+        with open(user_filename, 'w') as f:
             f.write('[distutils2]\n')
-        finally:
-            f.close()
 
         def _expander(path):
             return temp_home
diff --git a/distutils2/tests/test_install.py b/distutils2/tests/test_install.py
--- a/distutils2/tests/test_install.py
+++ b/distutils2/tests/test_install.py
@@ -18,7 +18,7 @@
     use_xmlrpc_server = fake_dec
 
 
-class InstalledDist(object):
+class InstalledDist:
     """Distribution object, represent distributions currently installed on the
     system"""
     def __init__(self, name, version, deps):
@@ -33,7 +33,7 @@
         return '<InstalledDist %r>' % self.metadata['Name']
 
 
-class ToInstallDist(object):
+class ToInstallDist:
     """Distribution that will be installed"""
 
     def __init__(self, files=False):
@@ -63,7 +63,7 @@
         return self.list_installed_files()
 
 
-class MagicMock(object):
+class MagicMock:
     def __init__(self, return_value=None, raise_exception=False):
         self.called = False
         self._times_called = 0
diff --git a/distutils2/tests/test_manifest.py b/distutils2/tests/test_manifest.py
--- a/distutils2/tests/test_manifest.py
+++ b/distutils2/tests/test_manifest.py
@@ -1,7 +1,7 @@
 """Tests for distutils2.manifest."""
 import os
 import logging
-from StringIO import StringIO
+from io import StringIO
 from distutils2.manifest import Manifest
 
 from distutils2.tests import unittest, support
@@ -37,11 +37,8 @@
     def test_manifest_reader(self):
         tmpdir = self.mkdtemp()
         MANIFEST = os.path.join(tmpdir, 'MANIFEST.in')
-        f = open(MANIFEST, 'w')
-        try:
+        with open(MANIFEST, 'w') as f:
             f.write(_MANIFEST)
-        finally:
-            f.close()
 
         manifest = Manifest()
         manifest.read_template(MANIFEST)
@@ -54,11 +51,8 @@
             self.assertIn('no files found matching', warning)
 
         # manifest also accepts file-like objects
-        f = open(MANIFEST)
-        try:
+        with open(MANIFEST) as f:
             manifest.read_template(f)
-        finally:
-            f.close()
 
         # the manifest should have been read and 3 warnings issued
         # (we didn't provide the files)
diff --git a/distutils2/tests/test_markers.py b/distutils2/tests/test_markers.py
--- a/distutils2/tests/test_markers.py
+++ b/distutils2/tests/test_markers.py
@@ -2,7 +2,6 @@
 import os
 import sys
 import platform
-from distutils2.compat import python_implementation
 from distutils2.markers import interpret
 
 from distutils2.tests import unittest
@@ -18,7 +17,7 @@
         os_name = os.name
         platform_version = platform.version()
         platform_machine = platform.machine()
-        platform_python_implementation = python_implementation()
+        platform_python_implementation = platform.python_implementation()
 
         self.assertTrue(interpret("sys.platform == '%s'" % sys_platform))
         self.assertTrue(interpret(
diff --git a/distutils2/tests/test_metadata.py b/distutils2/tests/test_metadata.py
--- a/distutils2/tests/test_metadata.py
+++ b/distutils2/tests/test_metadata.py
@@ -1,11 +1,9 @@
-# encoding: utf-8
 """Tests for distutils2.metadata."""
 import os
 import sys
-import codecs
 import logging
 from textwrap import dedent
-from StringIO import StringIO
+from io import StringIO
 
 from distutils2.errors import (MetadataConflictError, MetadataMissingError,
                                MetadataUnrecognizedVersionError)
@@ -37,12 +35,8 @@
 
     def test_instantiation(self):
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
-        f = codecs.open(PKG_INFO, 'r', encoding='utf-8')
-        try:
+        with open(PKG_INFO, 'r', encoding='utf-8') as f:
             contents = f.read()
-        finally:
-            f.close()
-
         fp = StringIO(contents)
 
         m = Metadata()
@@ -70,11 +64,8 @@
     def test_metadata_markers(self):
         # see if we can be platform-aware
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
-        f = codecs.open(PKG_INFO, 'r', encoding='utf-8')
-        try:
+        with open(PKG_INFO, 'r', encoding='utf-8') as f:
             content = f.read() % sys.platform
-        finally:
-            f.close()
         metadata = Metadata(platform_dependent=True)
 
         metadata.read_file(StringIO(content))
@@ -92,11 +83,8 @@
 
     def test_mapping_api(self):
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
-        f = codecs.open(PKG_INFO, 'r', encoding='utf-8')
-        try:
+        with open(PKG_INFO, 'r', encoding='utf-8') as f:
             content = f.read() % sys.platform
-        finally:
-            f.close()
         metadata = Metadata(fileobj=StringIO(content))
         self.assertIn('Version', metadata.keys())
         self.assertIn('0.5', metadata.values())
@@ -111,6 +99,8 @@
         metadata.update({'version': '1--2'})
         self.assertEqual(len(self.get_logs()), 1)
 
+        # XXX caveat: the keys method and friends are not 3.x-style views
+        # should be changed or documented
         self.assertEqual(list(metadata), metadata.keys())
 
     def test_read_metadata(self):
@@ -143,21 +133,18 @@
         tmp_dir = self.mkdtemp()
         my_file = os.path.join(tmp_dir, 'f')
 
-        metadata = Metadata(mapping={'author': u'Mister Caf?',
-                                     'name': u'my.project',
-                                     'author': u'Caf? Junior',
-                                     'summary': u'Caf? torr?fi?',
-                                     'description': u'H?h?h?',
-                                     'keywords': [u'caf?', u'coffee']})
+        metadata = Metadata(mapping={'author': 'Mister Caf?',
+                                     'name': 'my.project',
+                                     'author': 'Caf? Junior',
+                                     'summary': 'Caf? torr?fi?',
+                                     'description': 'H?h?h?',
+                                     'keywords': ['caf?', 'coffee']})
         metadata.write(my_file)
 
         # the file should use UTF-8
         metadata2 = Metadata()
-        fp = codecs.open(my_file, encoding='utf-8')
-        try:
+        with open(my_file, encoding='utf-8') as fp:
             metadata2.read_file(fp)
-        finally:
-            fp.close()
 
         # XXX when keywords are not defined, metadata will have
         # 'Keywords': [] but metadata2 will have 'Keywords': ['']
@@ -165,19 +152,16 @@
         self.assertEqual(metadata.items(), metadata2.items())
 
         # ASCII also works, it's a subset of UTF-8
-        metadata = Metadata(mapping={'author': u'Mister Cafe',
-                                     'name': u'my.project',
-                                     'author': u'Cafe Junior',
-                                     'summary': u'Cafe torrefie',
-                                     'description': u'Hehehe'})
+        metadata = Metadata(mapping={'author': 'Mister Cafe',
+                                     'name': 'my.project',
+                                     'author': 'Cafe Junior',
+                                     'summary': 'Cafe torrefie',
+                                     'description': 'Hehehe'})
         metadata.write(my_file)
 
         metadata2 = Metadata()
-        fp = codecs.open(my_file, encoding='utf-8')
-        try:
+        with open(my_file, encoding='utf-8') as fp:
             metadata2.read_file(fp)
-        finally:
-            fp.close()
 
     def test_metadata_read_write(self):
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
@@ -324,21 +308,15 @@
 
     def test_description(self):
         PKG_INFO = os.path.join(os.path.dirname(__file__), 'PKG-INFO')
-        f = codecs.open(PKG_INFO, 'r', encoding='utf-8')
-        try:
+        with open(PKG_INFO, 'r', encoding='utf-8') as f:
             content = f.read() % sys.platform
-        finally:
-            f.close()
         metadata = Metadata()
         metadata.read_file(StringIO(content))
 
         # see if we can read the description now
         DESC = os.path.join(os.path.dirname(__file__), 'LONG_DESC.txt')
-        f = open(DESC)
-        try:
+        with open(DESC) as f:
             wanted = f.read()
-        finally:
-            f.close()
         self.assertEqual(wanted, metadata['Description'])
 
         # save the file somewhere and make sure we can read it back
diff --git a/distutils2/tests/test_mixin2to3.py b/distutils2/tests/test_mixin2to3.py
--- a/distutils2/tests/test_mixin2to3.py
+++ b/distutils2/tests/test_mixin2to3.py
@@ -9,30 +9,22 @@
                         support.LoggingCatcher,
                         unittest.TestCase):
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_convert_code_only(self):
         # used to check if code gets converted properly.
         code = "print 'test'"
 
-        fp = self.mktempfile()
-        try:
+        with self.mktempfile() as fp:
             fp.write(code)
-        finally:
-            fp.close()
 
         mixin2to3 = Mixin2to3()
         mixin2to3._run_2to3([fp.name])
         expected = "print('test')"
 
-        fp = open(fp.name)
-        try:
+        with open(fp.name) as fp:
             converted = fp.read()
-        finally:
-            fp.close()
 
         self.assertEqual(expected, converted)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_doctests_only(self):
         # used to check if doctests gets converted properly.
         doctest = textwrap.dedent('''\
@@ -44,11 +36,8 @@
             It works.
             """''')
 
-        fp = self.mktempfile()
-        try:
+        with self.mktempfile() as fp:
             fp.write(doctest)
-        finally:
-            fp.close()
 
         mixin2to3 = Mixin2to3()
         mixin2to3._run_2to3([fp.name])
@@ -61,24 +50,17 @@
             It works.
             """\n''')
 
-        fp = open(fp.name)
-        try:
+        with open(fp.name) as fp:
             converted = fp.read()
-        finally:
-            fp.close()
 
         self.assertEqual(expected, converted)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_additional_fixers(self):
         # used to check if use_2to3_fixers works
         code = 'type(x) is not T'
 
-        fp = self.mktempfile()
-        try:
+        with self.mktempfile() as fp:
             fp.write(code)
-        finally:
-            fp.close()
 
         mixin2to3 = Mixin2to3()
         mixin2to3._run_2to3(files=[fp.name], doctests=[fp.name],
@@ -86,11 +68,8 @@
 
         expected = 'not isinstance(x, T)'
 
-        fp = open(fp.name)
-        try:
+        with open(fp.name) as fp:
             converted = fp.read()
-        finally:
-            fp.close()
 
         self.assertEqual(expected, converted)
 
diff --git a/distutils2/tests/test_msvc9compiler.py b/distutils2/tests/test_msvc9compiler.py
--- a/distutils2/tests/test_msvc9compiler.py
+++ b/distutils2/tests/test_msvc9compiler.py
@@ -118,22 +118,16 @@
         from distutils2.compiler.msvc9compiler import MSVCCompiler
         tempdir = self.mkdtemp()
         manifest = os.path.join(tempdir, 'manifest')
-        f = open(manifest, 'w')
-        try:
+        with open(manifest, 'w') as f:
             f.write(_MANIFEST)
-        finally:
-            f.close()
 
         compiler = MSVCCompiler()
         compiler._remove_visual_c_ref(manifest)
 
         # see what we got
-        f = open(manifest)
-        try:
+        with open(manifest) as f:
             # removing trailing spaces
             content = '\n'.join(line.rstrip() for line in f.readlines())
-        finally:
-            f.close()
 
         # makes sure the manifest was properly cleaned
         self.assertEqual(content, _CLEANED_MANIFEST)
diff --git a/distutils2/tests/test_pypi_server.py b/distutils2/tests/test_pypi_server.py
--- a/distutils2/tests/test_pypi_server.py
+++ b/distutils2/tests/test_pypi_server.py
@@ -1,5 +1,7 @@
 """Tests for distutils2.command.bdist."""
-import urllib2
+import urllib.request
+import urllib.parse
+import urllib.error
 
 try:
     import threading
@@ -13,6 +15,7 @@
 from distutils2.tests import unittest
 
 
+ at unittest.skipIf(threading is None, "Needs threading")
 class PyPIServerTest(unittest.TestCase):
 
     def test_records_requests(self):
@@ -24,12 +27,13 @@
             server.start()
             self.assertEqual(len(server.requests), 0)
 
-            data = 'Rock Around The Bunker'
+            data = b'Rock Around The Bunker'
 
             headers = {"X-test-header": "Mister Iceberg"}
 
-            request = urllib2.Request(server.full_address, data, headers)
-            urllib2.urlopen(request)
+            request = urllib.request.Request(
+                server.full_address, data, headers)
+            urllib.request.urlopen(request)
             self.assertEqual(len(server.requests), 1)
             handler, request_data = server.requests[-1]
             self.assertIn(data, request_data)
@@ -48,14 +52,11 @@
             server is the same than the one made by a simple file read.
             """
             url = server.full_address + url_path
-            request = urllib2.Request(url)
-            response = urllib2.urlopen(request)
-            file = open(PYPI_DEFAULT_STATIC_PATH + "/test_pypi_server"
-                      + url_path)
-            try:
+            request = urllib.request.Request(url)
+            response = urllib.request.urlopen(request)
+            with open(PYPI_DEFAULT_STATIC_PATH + "/test_pypi_server"
+                      + url_path) as file:
                 return response.read().decode() == file.read()
-            finally:
-                file.close()
 
         server = PyPIServer(static_uri_paths=["simple", "external"],
             static_filesystem_paths=["test_pypi_server"])
@@ -63,10 +64,10 @@
         try:
             # the file does not exists on the disc, so it might not be served
             url = server.full_address + "/simple/unexisting_page"
-            request = urllib2.Request(url)
+            request = urllib.request.Request(url)
             try:
-                urllib2.urlopen(request)
-            except urllib2.HTTPError, e:
+                urllib.request.urlopen(request)
+            except urllib.error.HTTPError as e:
                 self.assertEqual(e.code, 404)
 
             # now try serving a content that do exists
@@ -80,10 +81,6 @@
             server.stop()
 
 
-PyPIServerTest = unittest.skipIf(threading is None, "Needs threading")(
-        PyPIServerTest)
-
-
 def test_suite():
     return unittest.makeSuite(PyPIServerTest)
 
diff --git a/distutils2/tests/test_pypi_simple.py b/distutils2/tests/test_pypi_simple.py
--- a/distutils2/tests/test_pypi_simple.py
+++ b/distutils2/tests/test_pypi_simple.py
@@ -2,8 +2,10 @@
 import re
 import os
 import sys
-import httplib
-import urllib2
+import http.client
+import urllib.error
+import urllib.parse
+import urllib.request
 
 from distutils2.pypi.simple import Crawler
 
@@ -12,7 +14,7 @@
                                       fake_dec)
 
 try:
-    import thread as _thread
+    import _thread
     from distutils2.tests.pypi_server import (use_pypi_server, PyPIServer,
                                               PYPI_DEFAULT_STATIC_PATH)
 except ImportError:
@@ -43,11 +45,11 @@
         url = 'http://127.0.0.1:0/nonesuch/test_simple'
         try:
             v = crawler._open_url(url)
-        except Exception, v:
+        except Exception as v:
             self.assertIn(url, str(v))
         else:
             v.close()
-            self.assertIsInstance(v, urllib2.HTTPError)
+            self.assertIsInstance(v, urllib.error.HTTPError)
 
         # issue 16
         # easy_install inquant.contentmirror.plone breaks because of a typo
@@ -57,35 +59,34 @@
                'inquant.contentmirror.plone/trunk')
         try:
             v = crawler._open_url(url)
-        except Exception, v:
+        except Exception as v:
             self.assertIn(url, str(v))
         else:
             v.close()
-            self.assertIsInstance(v, urllib2.HTTPError)
+            self.assertIsInstance(v, urllib.error.HTTPError)
 
         def _urlopen(*args):
-            raise httplib.BadStatusLine('line')
+            raise http.client.BadStatusLine('line')
 
-        old_urlopen = urllib2.urlopen
-        urllib2.urlopen = _urlopen
+        old_urlopen = urllib.request.urlopen
+        urllib.request.urlopen = _urlopen
         url = 'http://example.org'
         try:
-            try:
-                v = crawler._open_url(url)
-            except Exception, v:
-                self.assertIn('line', str(v))
-            else:
-                v.close()
-                # TODO use self.assertRaises
-                raise AssertionError('Should have raise here!')
+            v = crawler._open_url(url)
+        except Exception as v:
+            self.assertIn('line', str(v))
+        else:
+            v.close()
+            # TODO use self.assertRaises
+            raise AssertionError('Should have raise here!')
         finally:
-            urllib2.urlopen = old_urlopen
+            urllib.request.urlopen = old_urlopen
 
         # issue 20
         url = 'http://http://svn.pythonpaste.org/Paste/wphp/trunk'
         try:
             crawler._open_url(url)
-        except Exception, v:
+        except Exception as v:
             self.assertIn('nonnumeric port', str(v))
 
         # issue #160
@@ -274,22 +275,22 @@
         # Test that the simple link matcher yield the good links.
         generator = crawler._simple_link_matcher(content, crawler.index_url)
         self.assertEqual(('%stest/foobar-1.tar.gz#md5=abcdef' %
-                          crawler.index_url, True), generator.next())
-        self.assertEqual(('http://dl-link1', True), generator.next())
+                          crawler.index_url, True), next(generator))
+        self.assertEqual(('http://dl-link1', True), next(generator))
         self.assertEqual(('%stest' % crawler.index_url, False),
-                         generator.next())
-        self.assertRaises(StopIteration, generator.next)
+                         next(generator))
+        self.assertRaises(StopIteration, generator.__next__)
 
         # Follow the external links is possible (eg. homepages)
         crawler.follow_externals = True
         generator = crawler._simple_link_matcher(content, crawler.index_url)
         self.assertEqual(('%stest/foobar-1.tar.gz#md5=abcdef' %
-                          crawler.index_url, True), generator.next())
-        self.assertEqual(('http://dl-link1', True), generator.next())
-        self.assertEqual(('http://dl-link2', False), generator.next())
+                          crawler.index_url, True), next(generator))
+        self.assertEqual(('http://dl-link1', True), next(generator))
+        self.assertEqual(('http://dl-link2', False), next(generator))
         self.assertEqual(('%stest' % crawler.index_url, False),
-                         generator.next())
-        self.assertRaises(StopIteration, generator.next)
+                         next(generator))
+        self.assertRaises(StopIteration, generator.__next__)
 
     def test_browse_local_files(self):
         # Test that we can browse local files"""
diff --git a/distutils2/tests/test_pypi_xmlrpc.py b/distutils2/tests/test_pypi_xmlrpc.py
--- a/distutils2/tests/test_pypi_xmlrpc.py
+++ b/distutils2/tests/test_pypi_xmlrpc.py
@@ -13,6 +13,7 @@
     use_xmlrpc_server = fake_dec
 
 
+ at unittest.skipIf(threading is None, "Needs threading")
 class TestXMLRPCClient(unittest.TestCase):
     def _get_client(self, server, *args, **kwargs):
         return Client(server.full_address, *args, **kwargs)
@@ -91,10 +92,6 @@
         self.assertEqual(['FooFoo'], release.metadata['obsoletes_dist'])
 
 
-TestXMLRPCClient = unittest.skipIf(threading is None, "Needs threading")(
-        TestXMLRPCClient)
-
-
 def test_suite():
     suite = unittest.TestSuite()
     suite.addTest(unittest.makeSuite(TestXMLRPCClient))
diff --git a/distutils2/tests/test_run.py b/distutils2/tests/test_run.py
--- a/distutils2/tests/test_run.py
+++ b/distutils2/tests/test_run.py
@@ -2,7 +2,7 @@
 
 import os
 import sys
-from StringIO import StringIO
+from io import StringIO
 
 from distutils2 import install
 from distutils2.tests import unittest, support
@@ -71,12 +71,11 @@
         else:
             pythonpath = d2parent
 
-        status, out, err = assert_python_ok(
-            '-c', 'from distutils2.run import main; main()', '--help',
-            PYTHONPATH=pythonpath)
+        status, out, err = assert_python_ok('-m', 'distutils2.run', '--help',
+                                            PYTHONPATH=pythonpath)
         self.assertEqual(status, 0)
-        self.assertGreater(out, '')
-        self.assertEqual(err, '')
+        self.assertGreater(out, b'')
+        self.assertEqual(err, b'')
 
 
 def test_suite():
diff --git a/distutils2/tests/test_uninstall.py b/distutils2/tests/test_uninstall.py
--- a/distutils2/tests/test_uninstall.py
+++ b/distutils2/tests/test_uninstall.py
@@ -1,7 +1,7 @@
 """Tests for the uninstall command."""
 import os
 import sys
-from StringIO import StringIO
+from io import StringIO
 import stat
 import distutils2.util
 
diff --git a/distutils2/tests/test_util.py b/distutils2/tests/test_util.py
--- a/distutils2/tests/test_util.py
+++ b/distutils2/tests/test_util.py
@@ -5,7 +5,7 @@
 import logging
 import tempfile
 import subprocess
-from StringIO import StringIO
+from io import StringIO
 
 from distutils2.tests import support, unittest
 from distutils2.tests.test_config import SETUP_CFG
@@ -55,24 +55,24 @@
 """
 
 EXPECTED_MULTIPART_OUTPUT = [
-    '---x',
-    'Content-Disposition: form-data; name="username"',
-    '',
-    'wok',
-    '---x',
-    'Content-Disposition: form-data; name="password"',
-    '',
-    'secret',
-    '---x',
-    'Content-Disposition: form-data; name="picture"; filename="wok.png"',
-    '',
-    'PNG89',
-    '---x--',
-    '',
+    b'---x',
+    b'Content-Disposition: form-data; name="username"',
+    b'',
+    b'wok',
+    b'---x',
+    b'Content-Disposition: form-data; name="password"',
+    b'',
+    b'secret',
+    b'---x',
+    b'Content-Disposition: form-data; name="picture"; filename="wok.png"',
+    b'',
+    b'PNG89',
+    b'---x--',
+    b'',
 ]
 
 
-class FakePopen(object):
+class FakePopen:
     test_class = None
 
     def __init__(self, args, bufsize=0, executable=None,
@@ -82,7 +82,7 @@
                  startupinfo=None, creationflags=0,
                  restore_signals=True, start_new_session=False,
                  pass_fds=()):
-        if isinstance(args, basestring):
+        if isinstance(args, str):
             args = args.split()
         self.cmd = args[0]
         exes = self.test_class._exes
@@ -319,8 +319,6 @@
         res = get_compiler_versions()
         self.assertEqual(res[2], None)
 
-    @unittest.skipUnless(hasattr(sys, 'dont_write_bytecode'),
-                         'sys.dont_write_bytecode not supported')
     def test_dont_write_bytecode(self):
         # makes sure byte_compile raise a PackagingError
         # if sys.dont_write_bytecode is True
@@ -377,7 +375,7 @@
                                         'pkg1.pkg3.pkg6']))
 
     def test_resolve_name(self):
-        self.assertIs(str, resolve_name('__builtin__.str'))
+        self.assertIs(str, resolve_name('builtins.str'))
         self.assertEqual(
             UtilTestCase.__name__,
             resolve_name("distutils2.tests.test_util.UtilTestCase").__name__)
@@ -407,7 +405,6 @@
         finally:
             sys.path.remove(tmp_dir)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_run_2to3_on_code(self):
         content = "print 'test'"
         converted_content = "print('test')"
@@ -421,7 +418,6 @@
         file_handle.close()
         self.assertEqual(new_content, converted_content)
 
-    @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_run_2to3_on_doctests(self):
         # to check if text files containing doctests only get converted.
         content = ">>> print 'test'\ntest\n"
@@ -448,24 +444,24 @@
         if os.name == 'posix':
             exe = os.path.join(tmpdir, 'foo.sh')
             self.write_file(exe, '#!/bin/sh\nexit 1')
-            os.chmod(exe, 0777)
+            os.chmod(exe, 0o777)
         else:
             exe = os.path.join(tmpdir, 'foo.bat')
             self.write_file(exe, 'exit 1')
 
-        os.chmod(exe, 0777)
+        os.chmod(exe, 0o777)
         self.assertRaises(PackagingExecError, spawn, [exe])
 
         # now something that works
         if os.name == 'posix':
             exe = os.path.join(tmpdir, 'foo.sh')
             self.write_file(exe, '#!/bin/sh\nexit 0')
-            os.chmod(exe, 0777)
+            os.chmod(exe, 0o777)
         else:
             exe = os.path.join(tmpdir, 'foo.bat')
             self.write_file(exe, 'exit 0')
 
-        os.chmod(exe, 0777)
+        os.chmod(exe, 0o777)
         spawn([exe])  # should work without any error
 
     def test_server_registration(self):
@@ -497,11 +493,8 @@
         self.assertFalse(os.path.exists(rc))
         generate_pypirc('tarek', 'xxx')
         self.assertTrue(os.path.exists(rc))
-        f = open(rc)
-        try:
+        with open(rc) as f:
             content = f.read()
-        finally:
-            f.close()
         self.assertEqual(content, WANTED)
 
     def test_cfg_to_args(self):
@@ -538,10 +531,10 @@
 
     def test_encode_multipart(self):
         fields = [('username', 'wok'), ('password', 'secret')]
-        files = [('picture', 'wok.png', 'PNG89')]
-        content_type, body = encode_multipart(fields, files, '-x')
-        self.assertEqual('multipart/form-data; boundary=-x', content_type)
-        self.assertEqual(EXPECTED_MULTIPART_OUTPUT, body.split('\r\n'))
+        files = [('picture', 'wok.png', b'PNG89')]
+        content_type, body = encode_multipart(fields, files, b'-x')
+        self.assertEqual(b'multipart/form-data; boundary=-x', content_type)
+        self.assertEqual(EXPECTED_MULTIPART_OUTPUT, body.split(b'\r\n'))
 
 
 class GlobTestCaseBase(support.TempdirManager,
@@ -787,21 +780,16 @@
             dir_paths.append(path)
         for f in files:
             path = os.path.join(tempdir, f)
-            _f = open(path, 'w')
-            try:
+            with open(path, 'w') as _f:
                 _f.write(f)
-            finally:
-                _f.close()
             file_paths.append(path)
 
-        record_file = open(record_file_path, 'w')
-        try:
+        with open(record_file_path, 'w') as record_file:
             for fpath in file_paths:
                 record_file.write(fpath + '\n')
             for dpath in dir_paths:
                 record_file.write(dpath + '\n')
-        finally:
-            record_file.close()
+
         return (tempdir, record_file_path)
 
 
diff --git a/distutils2/tests/xxmodule.c b/distutils2/tests/xxmodule.c
--- a/distutils2/tests/xxmodule.c
+++ b/distutils2/tests/xxmodule.c
@@ -10,7 +10,7 @@
    your own types of attributes instead.  Maybe you want to name your
    local variables other than 'self'.  If your object type is needed in
    other files, you'll have to create a file "foobarobject.h"; see
-   intobject.h for an example. */
+   floatobject.h for an example. */
 
 /* Xxo objects */
 
@@ -19,23 +19,23 @@
 static PyObject *ErrorObject;
 
 typedef struct {
-	PyObject_HEAD
-	PyObject	*x_attr;	/* Attributes dictionary */
+    PyObject_HEAD
+    PyObject            *x_attr;        /* Attributes dictionary */
 } XxoObject;
 
 static PyTypeObject Xxo_Type;
 
-#define XxoObject_Check(v)	(Py_TYPE(v) == &Xxo_Type)
+#define XxoObject_Check(v)      (Py_TYPE(v) == &Xxo_Type)
 
 static XxoObject *
 newXxoObject(PyObject *arg)
 {
-	XxoObject *self;
-	self = PyObject_New(XxoObject, &Xxo_Type);
-	if (self == NULL)
-		return NULL;
-	self->x_attr = NULL;
-	return self;
+    XxoObject *self;
+    self = PyObject_New(XxoObject, &Xxo_Type);
+    if (self == NULL)
+        return NULL;
+    self->x_attr = NULL;
+    return self;
 }
 
 /* Xxo methods */
@@ -43,101 +43,101 @@
 static void
 Xxo_dealloc(XxoObject *self)
 {
-	Py_XDECREF(self->x_attr);
-	PyObject_Del(self);
+    Py_XDECREF(self->x_attr);
+    PyObject_Del(self);
 }
 
 static PyObject *
 Xxo_demo(XxoObject *self, PyObject *args)
 {
-	if (!PyArg_ParseTuple(args, ":demo"))
-		return NULL;
-	Py_INCREF(Py_None);
-	return Py_None;
+    if (!PyArg_ParseTuple(args, ":demo"))
+        return NULL;
+    Py_INCREF(Py_None);
+    return Py_None;
 }
 
 static PyMethodDef Xxo_methods[] = {
-	{"demo",	(PyCFunction)Xxo_demo,	METH_VARARGS,
-		PyDoc_STR("demo() -> None")},
-	{NULL,		NULL}		/* sentinel */
+    {"demo",            (PyCFunction)Xxo_demo,  METH_VARARGS,
+        PyDoc_STR("demo() -> None")},
+    {NULL,              NULL}           /* sentinel */
 };
 
 static PyObject *
-Xxo_getattr(XxoObject *self, char *name)
+Xxo_getattro(XxoObject *self, PyObject *name)
 {
-	if (self->x_attr != NULL) {
-		PyObject *v = PyDict_GetItemString(self->x_attr, name);
-		if (v != NULL) {
-			Py_INCREF(v);
-			return v;
-		}
-	}
-	return Py_FindMethod(Xxo_methods, (PyObject *)self, name);
+    if (self->x_attr != NULL) {
+        PyObject *v = PyDict_GetItem(self->x_attr, name);
+        if (v != NULL) {
+            Py_INCREF(v);
+            return v;
+        }
+    }
+    return PyObject_GenericGetAttr((PyObject *)self, name);
 }
 
 static int
 Xxo_setattr(XxoObject *self, char *name, PyObject *v)
 {
-	if (self->x_attr == NULL) {
-		self->x_attr = PyDict_New();
-		if (self->x_attr == NULL)
-			return -1;
-	}
-	if (v == NULL) {
-		int rv = PyDict_DelItemString(self->x_attr, name);
-		if (rv < 0)
-			PyErr_SetString(PyExc_AttributeError,
-			        "delete non-existing Xxo attribute");
-		return rv;
-	}
-	else
-		return PyDict_SetItemString(self->x_attr, name, v);
+    if (self->x_attr == NULL) {
+        self->x_attr = PyDict_New();
+        if (self->x_attr == NULL)
+            return -1;
+    }
+    if (v == NULL) {
+        int rv = PyDict_DelItemString(self->x_attr, name);
+        if (rv < 0)
+            PyErr_SetString(PyExc_AttributeError,
+                "delete non-existing Xxo attribute");
+        return rv;
+    }
+    else
+        return PyDict_SetItemString(self->x_attr, name, v);
 }
 
 static PyTypeObject Xxo_Type = {
-	/* The ob_type field must be initialized in the module init function
-	 * to be portable to Windows without using C++. */
-	PyVarObject_HEAD_INIT(NULL, 0)
-	"xxmodule.Xxo",		/*tp_name*/
-	sizeof(XxoObject),	/*tp_basicsize*/
-	0,			/*tp_itemsize*/
-	/* methods */
-	(destructor)Xxo_dealloc, /*tp_dealloc*/
-	0,			/*tp_print*/
-	(getattrfunc)Xxo_getattr, /*tp_getattr*/
-	(setattrfunc)Xxo_setattr, /*tp_setattr*/
-	0,			/*tp_compare*/
-	0,			/*tp_repr*/
-	0,			/*tp_as_number*/
-	0,			/*tp_as_sequence*/
-	0,			/*tp_as_mapping*/
-	0,			/*tp_hash*/
-        0,                      /*tp_call*/
-        0,                      /*tp_str*/
-        0,                      /*tp_getattro*/
-        0,                      /*tp_setattro*/
-        0,                      /*tp_as_buffer*/
-        Py_TPFLAGS_DEFAULT,     /*tp_flags*/
-        0,                      /*tp_doc*/
-        0,                      /*tp_traverse*/
-        0,                      /*tp_clear*/
-        0,                      /*tp_richcompare*/
-        0,                      /*tp_weaklistoffset*/
-        0,                      /*tp_iter*/
-        0,                      /*tp_iternext*/
-        0,                      /*tp_methods*/
-        0,                      /*tp_members*/
-        0,                      /*tp_getset*/
-        0,                      /*tp_base*/
-        0,                      /*tp_dict*/
-        0,                      /*tp_descr_get*/
-        0,                      /*tp_descr_set*/
-        0,                      /*tp_dictoffset*/
-        0,                      /*tp_init*/
-        0,                      /*tp_alloc*/
-        0,                      /*tp_new*/
-        0,                      /*tp_free*/
-        0,                      /*tp_is_gc*/
+    /* The ob_type field must be initialized in the module init function
+     * to be portable to Windows without using C++. */
+    PyVarObject_HEAD_INIT(NULL, 0)
+    "xxmodule.Xxo",             /*tp_name*/
+    sizeof(XxoObject),          /*tp_basicsize*/
+    0,                          /*tp_itemsize*/
+    /* methods */
+    (destructor)Xxo_dealloc, /*tp_dealloc*/
+    0,                          /*tp_print*/
+    (getattrfunc)0,         /*tp_getattr*/
+    (setattrfunc)Xxo_setattr, /*tp_setattr*/
+    0,                          /*tp_reserved*/
+    0,                          /*tp_repr*/
+    0,                          /*tp_as_number*/
+    0,                          /*tp_as_sequence*/
+    0,                          /*tp_as_mapping*/
+    0,                          /*tp_hash*/
+    0,                      /*tp_call*/
+    0,                      /*tp_str*/
+    (getattrofunc)Xxo_getattro, /*tp_getattro*/
+    0,                      /*tp_setattro*/
+    0,                      /*tp_as_buffer*/
+    Py_TPFLAGS_DEFAULT,     /*tp_flags*/
+    0,                      /*tp_doc*/
+    0,                      /*tp_traverse*/
+    0,                      /*tp_clear*/
+    0,                      /*tp_richcompare*/
+    0,                      /*tp_weaklistoffset*/
+    0,                      /*tp_iter*/
+    0,                      /*tp_iternext*/
+    Xxo_methods,            /*tp_methods*/
+    0,                      /*tp_members*/
+    0,                      /*tp_getset*/
+    0,                      /*tp_base*/
+    0,                      /*tp_dict*/
+    0,                      /*tp_descr_get*/
+    0,                      /*tp_descr_set*/
+    0,                      /*tp_dictoffset*/
+    0,                      /*tp_init*/
+    0,                      /*tp_alloc*/
+    0,                      /*tp_new*/
+    0,                      /*tp_free*/
+    0,                      /*tp_is_gc*/
 };
 /* --------------------------------------------------------------------- */
 
@@ -151,12 +151,12 @@
 static PyObject *
 xx_foo(PyObject *self, PyObject *args)
 {
-	long i, j;
-	long res;
-	if (!PyArg_ParseTuple(args, "ll:foo", &i, &j))
-		return NULL;
-	res = i+j; /* XXX Do something here */
-	return PyInt_FromLong(res);
+    long i, j;
+    long res;
+    if (!PyArg_ParseTuple(args, "ll:foo", &i, &j))
+        return NULL;
+    res = i+j; /* XXX Do something here */
+    return PyLong_FromLong(res);
 }
 
 
@@ -165,14 +165,14 @@
 static PyObject *
 xx_new(PyObject *self, PyObject *args)
 {
-	XxoObject *rv;
+    XxoObject *rv;
 
-	if (!PyArg_ParseTuple(args, ":new"))
-		return NULL;
-	rv = newXxoObject(args);
-	if (rv == NULL)
-		return NULL;
-	return (PyObject *)rv;
+    if (!PyArg_ParseTuple(args, ":new"))
+        return NULL;
+    rv = newXxoObject(args);
+    if (rv == NULL)
+        return NULL;
+    return (PyObject *)rv;
 }
 
 /* Example with subtle bug from extensions manual ("Thin Ice"). */
@@ -180,20 +180,20 @@
 static PyObject *
 xx_bug(PyObject *self, PyObject *args)
 {
-	PyObject *list, *item;
+    PyObject *list, *item;
 
-	if (!PyArg_ParseTuple(args, "O:bug", &list))
-		return NULL;
+    if (!PyArg_ParseTuple(args, "O:bug", &list))
+        return NULL;
 
-	item = PyList_GetItem(list, 0);
-	/* Py_INCREF(item); */
-	PyList_SetItem(list, 1, PyInt_FromLong(0L));
-	PyObject_Print(item, stdout, 0);
-	printf("\n");
-	/* Py_DECREF(item); */
+    item = PyList_GetItem(list, 0);
+    /* Py_INCREF(item); */
+    PyList_SetItem(list, 1, PyLong_FromLong(0L));
+    PyObject_Print(item, stdout, 0);
+    printf("\n");
+    /* Py_DECREF(item); */
 
-	Py_INCREF(Py_None);
-	return Py_None;
+    Py_INCREF(Py_None);
+    return Py_None;
 }
 
 /* Test bad format character */
@@ -201,61 +201,61 @@
 static PyObject *
 xx_roj(PyObject *self, PyObject *args)
 {
-	PyObject *a;
-	long b;
-	if (!PyArg_ParseTuple(args, "O#:roj", &a, &b))
-		return NULL;
-	Py_INCREF(Py_None);
-	return Py_None;
+    PyObject *a;
+    long b;
+    if (!PyArg_ParseTuple(args, "O#:roj", &a, &b))
+        return NULL;
+    Py_INCREF(Py_None);
+    return Py_None;
 }
 
 
 /* ---------- */
 
 static PyTypeObject Str_Type = {
-	/* The ob_type field must be initialized in the module init function
-	 * to be portable to Windows without using C++. */
-	PyVarObject_HEAD_INIT(NULL, 0)
-	"xxmodule.Str",		/*tp_name*/
-	0,			/*tp_basicsize*/
-	0,			/*tp_itemsize*/
-	/* methods */
-	0,			/*tp_dealloc*/
-	0,			/*tp_print*/
-	0,			/*tp_getattr*/
-	0,			/*tp_setattr*/
-	0,			/*tp_compare*/
-	0,			/*tp_repr*/
-	0,			/*tp_as_number*/
-	0,			/*tp_as_sequence*/
-	0,			/*tp_as_mapping*/
-	0,			/*tp_hash*/
-	0,			/*tp_call*/
-	0,			/*tp_str*/
-	0,			/*tp_getattro*/
-	0,			/*tp_setattro*/
-	0,			/*tp_as_buffer*/
-	Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
-	0,			/*tp_doc*/
-	0,			/*tp_traverse*/
-	0,			/*tp_clear*/
-	0,			/*tp_richcompare*/
-	0,			/*tp_weaklistoffset*/
-	0,			/*tp_iter*/
-	0,			/*tp_iternext*/
-	0,			/*tp_methods*/
-	0,			/*tp_members*/
-	0,			/*tp_getset*/
-	0, /* see initxx */	/*tp_base*/
-	0,			/*tp_dict*/
-	0,			/*tp_descr_get*/
-	0,			/*tp_descr_set*/
-	0,			/*tp_dictoffset*/
-	0,			/*tp_init*/
-	0,			/*tp_alloc*/
-	0,			/*tp_new*/
-	0,			/*tp_free*/
-	0,			/*tp_is_gc*/
+    /* The ob_type field must be initialized in the module init function
+     * to be portable to Windows without using C++. */
+    PyVarObject_HEAD_INIT(NULL, 0)
+    "xxmodule.Str",             /*tp_name*/
+    0,                          /*tp_basicsize*/
+    0,                          /*tp_itemsize*/
+    /* methods */
+    0,                          /*tp_dealloc*/
+    0,                          /*tp_print*/
+    0,                          /*tp_getattr*/
+    0,                          /*tp_setattr*/
+    0,                          /*tp_reserved*/
+    0,                          /*tp_repr*/
+    0,                          /*tp_as_number*/
+    0,                          /*tp_as_sequence*/
+    0,                          /*tp_as_mapping*/
+    0,                          /*tp_hash*/
+    0,                          /*tp_call*/
+    0,                          /*tp_str*/
+    0,                          /*tp_getattro*/
+    0,                          /*tp_setattro*/
+    0,                          /*tp_as_buffer*/
+    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
+    0,                          /*tp_doc*/
+    0,                          /*tp_traverse*/
+    0,                          /*tp_clear*/
+    0,                          /*tp_richcompare*/
+    0,                          /*tp_weaklistoffset*/
+    0,                          /*tp_iter*/
+    0,                          /*tp_iternext*/
+    0,                          /*tp_methods*/
+    0,                          /*tp_members*/
+    0,                          /*tp_getset*/
+    0, /* see PyInit_xx */      /*tp_base*/
+    0,                          /*tp_dict*/
+    0,                          /*tp_descr_get*/
+    0,                          /*tp_descr_set*/
+    0,                          /*tp_dictoffset*/
+    0,                          /*tp_init*/
+    0,                          /*tp_alloc*/
+    0,                          /*tp_new*/
+    0,                          /*tp_free*/
+    0,                          /*tp_is_gc*/
 };
 
 /* ---------- */
@@ -263,54 +263,54 @@
 static PyObject *
 null_richcompare(PyObject *self, PyObject *other, int op)
 {
-	Py_INCREF(Py_NotImplemented);
-	return Py_NotImplemented;
+    Py_INCREF(Py_NotImplemented);
+    return Py_NotImplemented;
 }
 
 static PyTypeObject Null_Type = {
-	/* The ob_type field must be initialized in the module init function
-	 * to be portable to Windows without using C++. */
-	PyVarObject_HEAD_INIT(NULL, 0)
-	"xxmodule.Null",	/*tp_name*/
-	0,			/*tp_basicsize*/
-	0,			/*tp_itemsize*/
-	/* methods */
-	0,			/*tp_dealloc*/
-	0,			/*tp_print*/
-	0,			/*tp_getattr*/
-	0,			/*tp_setattr*/
-	0,			/*tp_compare*/
-	0,			/*tp_repr*/
-	0,			/*tp_as_number*/
-	0,			/*tp_as_sequence*/
-	0,			/*tp_as_mapping*/
-	0,			/*tp_hash*/
-	0,			/*tp_call*/
-	0,			/*tp_str*/
-	0,			/*tp_getattro*/
-	0,			/*tp_setattro*/
-	0,			/*tp_as_buffer*/
-	Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
-	0,			/*tp_doc*/
-	0,			/*tp_traverse*/
-	0,			/*tp_clear*/
-	null_richcompare,	/*tp_richcompare*/
-	0,			/*tp_weaklistoffset*/
-	0,			/*tp_iter*/
-	0,			/*tp_iternext*/
-	0,			/*tp_methods*/
-	0,			/*tp_members*/
-	0,			/*tp_getset*/
-	0, /* see initxx */	/*tp_base*/
-	0,			/*tp_dict*/
-	0,			/*tp_descr_get*/
-	0,			/*tp_descr_set*/
-	0,			/*tp_dictoffset*/
-	0,			/*tp_init*/
-	0,			/*tp_alloc*/
-	0, /* see initxx */	/*tp_new*/
-	0,			/*tp_free*/
-	0,			/*tp_is_gc*/
+    /* The ob_type field must be initialized in the module init function
+     * to be portable to Windows without using C++. */
+    PyVarObject_HEAD_INIT(NULL, 0)
+    "xxmodule.Null",            /*tp_name*/
+    0,                          /*tp_basicsize*/
+    0,                          /*tp_itemsize*/
+    /* methods */
+    0,                          /*tp_dealloc*/
+    0,                          /*tp_print*/
+    0,                          /*tp_getattr*/
+    0,                          /*tp_setattr*/
+    0,                          /*tp_reserved*/
+    0,                          /*tp_repr*/
+    0,                          /*tp_as_number*/
+    0,                          /*tp_as_sequence*/
+    0,                          /*tp_as_mapping*/
+    0,                          /*tp_hash*/
+    0,                          /*tp_call*/
+    0,                          /*tp_str*/
+    0,                          /*tp_getattro*/
+    0,                          /*tp_setattro*/
+    0,                          /*tp_as_buffer*/
+    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
+    0,                          /*tp_doc*/
+    0,                          /*tp_traverse*/
+    0,                          /*tp_clear*/
+    null_richcompare,           /*tp_richcompare*/
+    0,                          /*tp_weaklistoffset*/
+    0,                          /*tp_iter*/
+    0,                          /*tp_iternext*/
+    0,                          /*tp_methods*/
+    0,                          /*tp_members*/
+    0,                          /*tp_getset*/
+    0, /* see PyInit_xx */      /*tp_base*/
+    0,                          /*tp_dict*/
+    0,                          /*tp_descr_get*/
+    0,                          /*tp_descr_set*/
+    0,                          /*tp_dictoffset*/
+    0,                          /*tp_init*/
+    0,                          /*tp_alloc*/
+    0, /* see PyInit_xx */      /*tp_new*/
+    0,                          /*tp_free*/
+    0,                          /*tp_is_gc*/
 };
 
 
@@ -320,60 +320,77 @@
 /* List of functions defined in the module */
 
 static PyMethodDef xx_methods[] = {
-	{"roj",		xx_roj,		METH_VARARGS,
-		PyDoc_STR("roj(a,b) -> None")},
-	{"foo",		xx_foo,		METH_VARARGS,
-	 	xx_foo_doc},
-	{"new",		xx_new,		METH_VARARGS,
-		PyDoc_STR("new() -> new Xx object")},
-	{"bug",		xx_bug,		METH_VARARGS,
-		PyDoc_STR("bug(o) -> None")},
-	{NULL,		NULL}		/* sentinel */
+    {"roj",             xx_roj,         METH_VARARGS,
+        PyDoc_STR("roj(a,b) -> None")},
+    {"foo",             xx_foo,         METH_VARARGS,
+        xx_foo_doc},
+    {"new",             xx_new,         METH_VARARGS,
+        PyDoc_STR("new() -> new Xx object")},
+    {"bug",             xx_bug,         METH_VARARGS,
+        PyDoc_STR("bug(o) -> None")},
+    {NULL,              NULL}           /* sentinel */
 };
 
 PyDoc_STRVAR(module_doc,
 "This is a template module just for instruction.");
 
-/* Initialization function for the module (*must* be called initxx) */
+/* Initialization function for the module (*must* be called PyInit_xx) */
+
+
+static struct PyModuleDef xxmodule = {
+    PyModuleDef_HEAD_INIT,
+    "xx",
+    module_doc,
+    -1,
+    xx_methods,
+    NULL,
+    NULL,
+    NULL,
+    NULL
+};
 
 PyMODINIT_FUNC
-initxx(void)
+PyInit_xx(void)
 {
-	PyObject *m;
+    PyObject *m = NULL;
 
-	/* Due to cross platform compiler issues the slots must be filled
-	 * here. It's required for portability to Windows without requiring
-	 * C++. */
-	Null_Type.tp_base = &PyBaseObject_Type;
-	Null_Type.tp_new = PyType_GenericNew;
-	Str_Type.tp_base = &PyUnicode_Type;
+    /* Due to cross platform compiler issues the slots must be filled
+     * here. It's required for portability to Windows without requiring
+     * C++. */
+    Null_Type.tp_base = &PyBaseObject_Type;
+    Null_Type.tp_new = PyType_GenericNew;
+    Str_Type.tp_base = &PyUnicode_Type;
 
-	/* Finalize the type object including setting type of the new type
-	 * object; doing it here is required for portability, too. */
-	if (PyType_Ready(&Xxo_Type) < 0)
-		return;
+    /* Finalize the type object including setting type of the new type
+     * object; doing it here is required for portability, too. */
+    if (PyType_Ready(&Xxo_Type) < 0)
+        goto fail;
 
-	/* Create the module and add the functions */
-	m = Py_InitModule3("xx", xx_methods, module_doc);
-	if (m == NULL)
-		return;
+    /* Create the module and add the functions */
+    m = PyModule_Create(&xxmodule);
+    if (m == NULL)
+        goto fail;
 
-	/* Add some symbolic constants to the module */
-	if (ErrorObject == NULL) {
-		ErrorObject = PyErr_NewException("xx.error", NULL, NULL);
-		if (ErrorObject == NULL)
-			return;
-	}
-	Py_INCREF(ErrorObject);
-	PyModule_AddObject(m, "error", ErrorObject);
+    /* Add some symbolic constants to the module */
+    if (ErrorObject == NULL) {
+        ErrorObject = PyErr_NewException("xx.error", NULL, NULL);
+        if (ErrorObject == NULL)
+            goto fail;
+    }
+    Py_INCREF(ErrorObject);
+    PyModule_AddObject(m, "error", ErrorObject);
 
-	/* Add Str */
-	if (PyType_Ready(&Str_Type) < 0)
-		return;
-	PyModule_AddObject(m, "Str", (PyObject *)&Str_Type);
+    /* Add Str */
+    if (PyType_Ready(&Str_Type) < 0)
+        goto fail;
+    PyModule_AddObject(m, "Str", (PyObject *)&Str_Type);
 
-	/* Add Null */
-	if (PyType_Ready(&Null_Type) < 0)
-		return;
-	PyModule_AddObject(m, "Null", (PyObject *)&Null_Type);
+    /* Add Null */
+    if (PyType_Ready(&Null_Type) < 0)
+        goto fail;
+    PyModule_AddObject(m, "Null", (PyObject *)&Null_Type);
+    return m;
+ fail:
+    Py_XDECREF(m);
+    return NULL;
 }
diff --git a/distutils2/util.py b/distutils2/util.py
--- a/distutils2/util.py
+++ b/distutils2/util.py
@@ -5,22 +5,15 @@
 import csv
 import sys
 import errno
-import codecs
 import shutil
 import string
+import hashlib
 import posixpath
 import subprocess
+from glob import iglob as std_iglob
 from fnmatch import fnmatchcase
 from inspect import getsource
-from ConfigParser import RawConfigParser
-try:
-    from glob import iglob as std_iglob
-except ImportError:
-    from glob import glob as std_iglob
-try:
-    import hashlib
-except ImportError:
-    from distutils2._backport import hashlib
+from configparser import RawConfigParser
 
 from distutils2 import logger
 from distutils2.errors import (PackagingPlatformError, PackagingFileError,
@@ -333,7 +326,7 @@
     """
     # nothing is done if sys.dont_write_bytecode is True
     # FIXME this should not raise an error
-    if getattr(sys, 'dont_write_bytecode', False):
+    if sys.dont_write_bytecode:
         raise PackagingByteCompileError('byte-compiling is disabled.')
 
     # First, if the caller didn't force us into direct or indirect mode,
@@ -362,9 +355,9 @@
             if script_fd is not None:
                 script = os.fdopen(script_fd, "w", encoding='utf-8')
             else:
-                script = codecs.open(script_name, "w", encoding='utf-8')
+                script = open(script_name, "w", encoding='utf-8')
 
-            try:
+            with script:
                 script.write("""\
 from distutils2.util import byte_compile
 files = [
@@ -391,8 +384,6 @@
              verbose=%r, dry_run=False,
              direct=True)
 """ % (optimize, force, prefix, base_dir, verbose))
-            finally:
-                script.close()
 
         cmd = [sys.executable, script_name]
         if optimize == 1:
@@ -549,12 +540,10 @@
     idiom in all cases, only with Command.execute, which runs depending on
     the dry_run argument and also logs its arguments).
     """
-    f = open(filename, "w")
-    try:
+    with open(filename, "w") as f:
         for line in contents:
             f.write(line + "\n")
-    finally:
-        f.close()
+
 
 def _is_package(path):
     return os.path.isdir(path) and os.path.isfile(
@@ -657,7 +646,7 @@
     for part in parts[1:]:
         try:
             ret = getattr(ret, part)
-        except AttributeError, exc:
+        except AttributeError as exc:
             raise ImportError(exc)
 
     return ret
@@ -773,13 +762,10 @@
 def generate_pypirc(username, password):
     """Create a default .pypirc file."""
     rc = get_pypirc_path()
-    f = open(rc, 'w')
+    with open(rc, 'w') as f:
+        f.write(DEFAULT_PYPIRC % (username, password))
     try:
-        f.write(DEFAULT_PYPIRC % (username, password))
-    finally:
-        f.close()
-    try:
-        os.chmod(rc, 0600)
+        os.chmod(rc, 0o600)
     except OSError:
         # should do something better here
         pass
@@ -863,7 +849,7 @@
     r.refactor(files, write=True, doctests_only=doctests_only)
 
 
-class Mixin2to3(object):
+class Mixin2to3:
     """ Wrapper class for commands that run 2to3.
     To configure 2to3, setup scripts may either change
     the class variables, or inherit from this class
@@ -986,11 +972,8 @@
     if not os.path.exists(path):
         raise PackagingFileError("file '%s' does not exist" %
                                  os.path.abspath(path))
-    f = codecs.open(path, encoding='utf-8')
-    try:
+    with open(path, encoding='utf-8') as f:
         config.readfp(f)
-    finally:
-        f.close()
 
     kwargs = {}
     for arg in D1_D2_SETUP_ARGS:
@@ -1012,11 +995,8 @@
                     filenames = split_multiline(filenames)
                     in_cfg_value = []
                     for filename in filenames:
-                        fp = open(filename)
-                        try:
+                        with open(filename) as fp:
                             in_cfg_value.append(fp.read())
-                        finally:
-                            fp.close()
                     in_cfg_value = '\n\n'.join(in_cfg_value)
             else:
                 continue
@@ -1053,11 +1033,8 @@
     if os.path.exists("setup.py"):
         raise PackagingFileError("a setup.py file already exists")
 
-    fp = codecs.open("setup.py", "w", encoding='utf-8')
-    try:
+    with open("setup.py", "w", encoding='utf-8') as fp:
         fp.write(_SETUP_TMPL % {'func': getsource(cfg_to_args)})
-    finally:
-        fp.close()
 
 
 # Taken from the pip project
@@ -1065,19 +1042,18 @@
 def ask(message, options):
     """Prompt the user with *message*; *options* contains allowed responses."""
     while True:
-        response = raw_input(message)
+        response = input(message)
         response = response.strip().lower()
         if response not in options:
-            print 'invalid response:', repr(response)
-            print 'choose one of', ', '.join(repr(o) for o in options)
+            print('invalid response:', repr(response))
+            print('choose one of', ', '.join(repr(o) for o in options))
         else:
             return response
 
 
 def _parse_record_file(record_file):
     distinfo, extra_metadata, installed = ({}, [], [])
-    rfile = open(record_file, 'r')
-    try:
+    with open(record_file, 'r') as rfile:
         for path in rfile:
             path = path.strip()
             if path.endswith('egg-info') and os.path.isfile(path):
@@ -1098,8 +1074,6 @@
                 continue
             else:
                 installed.append(path)
-    finally:
-        rfile.close()
 
     distinfo['egginfo'] = egginfo
     distinfo['metadata'] = metadata
@@ -1115,8 +1089,7 @@
 
 
 def _write_record_file(record_path, installed_files):
-    f = codecs.open(record_path, 'w', encoding='utf-8')
-    try:
+    with open(record_path, 'w', encoding='utf-8') as f:
         writer = csv.writer(f, delimiter=',', lineterminator=os.linesep,
                             quotechar='"')
 
@@ -1126,19 +1099,14 @@
                 writer.writerow((fpath, '', ''))
             else:
                 hash = hashlib.md5()
-                fp = open(fpath, 'rb')
-                try:
+                with open(fpath, 'rb') as fp:
                     hash.update(fp.read())
-                finally:
-                    fp.close()
                 md5sum = hash.hexdigest()
                 size = os.path.getsize(fpath)
                 writer.writerow((fpath, md5sum, size))
 
         # add the RECORD file itself
         writer.writerow((record_path, '', ''))
-    finally:
-        f.close()
     return record_path
 
 
@@ -1173,11 +1141,8 @@
 
     installer_path = distinfo['installer_path']
     logger.info('creating %s', installer_path)
-    f = open(installer_path, 'w')
-    try:
+    with open(installer_path, 'w') as f:
         f.write(installer)
-    finally:
-        f.close()
 
     if requested:
         requested_path = distinfo['requested_path']
@@ -1218,27 +1183,20 @@
 
 
 def _has_text(setup_py, installer):
-    installer_pattern = re.compile('import %s|from %s' %
-                                   (installer, installer))
-    setup = codecs.open(setup_py, 'r', encoding='utf-8')
-    try:
+    installer_pattern = re.compile('import {0}|from {0}'.format(installer))
+    with open(setup_py, 'r', encoding='utf-8') as setup:
         for line in setup:
             if re.search(installer_pattern, line):
                 logger.debug("Found %s text in setup.py.", installer)
                 return True
-    finally:
-        setup.close()
     logger.debug("No %s text found in setup.py.", installer)
     return False
 
 
 def _has_required_metadata(setup_cfg):
     config = RawConfigParser()
-    f = codecs.open(setup_cfg, 'r', encoding='utf8')
-    try:
+    with open(setup_cfg, 'r', encoding='utf8') as f:
         config.readfp(f)
-    finally:
-        f.close()
     return (config.has_section('metadata') and
             'name' in config.options('metadata') and
             'version' in config.options('metadata'))
@@ -1341,7 +1299,7 @@
               "cannot copy tree '%s': not a directory" % src)
     try:
         names = os.listdir(src)
-    except os.error, e:
+    except os.error as e:
         errstr = e[1]
         if dry_run:
             names = []
@@ -1387,9 +1345,9 @@
 # I don't use os.makedirs because a) it's new to Python 1.5.2, and
 # b) it blows up if the directory already exists (I want to silently
 # succeed in that case).
-def _mkpath(name, mode=0777, verbose=True, dry_run=False):
+def _mkpath(name, mode=0o777, verbose=True, dry_run=False):
     # Detect a common bug -- name is None
-    if not isinstance(name, basestring):
+    if not isinstance(name, str):
         raise PackagingInternalError(
               "mkpath: 'name' must be a string (got %r)" % (name,))
 
@@ -1428,7 +1386,7 @@
         if not dry_run:
             try:
                 os.mkdir(head, mode)
-            except OSError, exc:
+            except OSError as exc:
                 if not (exc.errno == errno.EEXIST and os.path.isdir(head)):
                     raise PackagingFileError(
                           "could not create '%s': %s" % (head, exc.args[-1]))
@@ -1445,15 +1403,15 @@
     form fields, *files* is a sequence of (name: str, filename: str, value:
     bytes) elements for data to be uploaded as files.
 
-    Returns (content_type: bytes, body: bytes) ready for httplib.HTTP.
+    Returns (content_type: bytes, body: bytes) ready for http.client.HTTP.
     """
     # Taken from
     # http://code.activestate.com/recipes/146306-http-client-to-post-using-multipartform-data/
 
     if boundary is None:
-        boundary = '--------------GHSKFJDLGDS7543FJKLFHRE75642756743254'
-    elif not isinstance(boundary, str):
-        raise TypeError('boundary must be str, not %r' % type(boundary))
+        boundary = b'--------------GHSKFJDLGDS7543FJKLFHRE75642756743254'
+    elif not isinstance(boundary, bytes):
+        raise TypeError('boundary must be bytes, not %r' % type(boundary))
 
     l = []
     for key, values in fields:
@@ -1463,21 +1421,23 @@
 
         for value in values:
             l.extend((
-                '--' + boundary,
-                # XXX should encode to match packaging but it causes bugs
-                ('Content-Disposition: form-data; name="%s"' % key), '', value))
+                b'--' + boundary,
+                ('Content-Disposition: form-data; name="%s"' %
+                 key).encode('utf-8'),
+                b'',
+                value.encode('utf-8')))
 
     for key, filename, value in files:
         l.extend((
-            '--' + boundary,
+            b'--' + boundary,
             ('Content-Disposition: form-data; name="%s"; filename="%s"' %
-             (key, filename)),
-            '',
+             (key, filename)).encode('utf-8'),
+            b'',
             value))
 
-    l.append('--' + boundary + '--')
-    l.append('')
+    l.append(b'--' + boundary + b'--')
+    l.append(b'')
 
-    body = '\r\n'.join(l)
-    content_type = 'multipart/form-data; boundary=' + boundary
+    body = b'\r\n'.join(l)
+    content_type = b'multipart/form-data; boundary=' + boundary
     return content_type, body
diff --git a/distutils2/version.py b/distutils2/version.py
--- a/distutils2/version.py
+++ b/distutils2/version.py
@@ -38,7 +38,7 @@
     $''', re.VERBOSE)
 
 
-class NormalizedVersion(object):
+class NormalizedVersion:
     """A rational version.
 
     Good:
@@ -342,7 +342,7 @@
     return comp, NormalizedVersion(version)
 
 
-class VersionPredicate(object):
+class VersionPredicate:
     """Defines a predicate: ProjectName (>ver1,ver2, ..)"""
 
     _operators = {"<": lambda x, y: x < y,
@@ -384,7 +384,7 @@
 
     def match(self, version):
         """Check if the provided version matches the predicates."""
-        if isinstance(version, basestring):
+        if isinstance(version, str):
             version = NormalizedVersion(version)
         for operator, predicate in self.predicates:
             if not self._operators[operator](version, predicate):
@@ -444,6 +444,6 @@
     """Return a VersionPredicate object, from a string or an already
     existing object.
     """
-    if isinstance(requirements, basestring):
+    if isinstance(requirements, str):
         requirements = VersionPredicate(requirements)
     return requirements
diff --git a/runtests.py b/runtests.py
--- a/runtests.py
+++ b/runtests.py
@@ -130,10 +130,4 @@
 
 
 if __name__ == "__main__":
-    if sys.version < '2.5':
-        try:
-            from distutils2._backport import hashlib
-        except ImportError:
-            import subprocess
-            subprocess.call([sys.executable, 'setup.py', 'build_ext'])
     sys.exit(test_main())
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1,13 +1,6 @@
-#!/usr/bin/env python
-# -*- encoding: utf-8 -*-
-import os
-import re
-import sys
-import codecs
-from distutils import sysconfig
-from distutils.core import setup, Extension
-from distutils.ccompiler import new_compiler
-from ConfigParser import RawConfigParser
+#!/usr/bin/env python3
+from distutils.core import setup
+from configparser import RawConfigParser
 
 
 def split_multiline(value):
@@ -48,11 +41,8 @@
         }
     config = RawConfigParser()
     config.optionxform = lambda x: x.lower().replace('_', '-')
-    fp = codecs.open(path, encoding='utf-8')
-    try:
+    with open(path, encoding='utf-8') as fp:
         config.readfp(fp)
-    finally:
-        fp.close()
     kwargs = {}
     for section in opts_to_args:
         for optname, argname, xform in opts_to_args[section]:
@@ -67,11 +57,8 @@
         filenames = config.get('metadata', 'description-file')
         for filename in split_multiline(filenames):
             descriptions = []
-            fp = open(filename)
-            try:
+            with open(filename) as fp:
                 descriptions.append(fp.read())
-            finally:
-                fp.close()
         kwargs['long_description'] = '\n\n'.join(descriptions)
     # Handle `package_data`
     if 'package_data' in kwargs:
@@ -83,126 +70,6 @@
         kwargs['package_data'] = package_data
     return kwargs
 
-# (from Python's setup.py, in PyBuildExt.detect_modules())
-def prepare_hashlib_extensions():
-    """Decide which C extensions to build and create the appropriate
-    Extension objects to build them.  Return a list of Extensions.
-    """
-    ssl_libs = None
-    ssl_inc_dir = None
-    ssl_lib_dirs = []
-    ssl_inc_dirs = []
-    if os.name == 'posix':
-        # (from Python's setup.py, in PyBuildExt.detect_modules())
-        # lib_dirs and inc_dirs are used to search for files;
-        # if a file is found in one of those directories, it can
-        # be assumed that no additional -I,-L directives are needed.
-        lib_dirs = []
-        inc_dirs = []
-        if os.path.normpath(sys.prefix) != '/usr':
-            lib_dirs.append(sysconfig.get_config_var('LIBDIR'))
-            inc_dirs.append(sysconfig.get_config_var('INCLUDEDIR'))
-        # Ensure that /usr/local is always used
-        lib_dirs.append('/usr/local/lib')
-        inc_dirs.append('/usr/local/include')
-        # Add the compiler defaults; this compiler object is only used
-        # to locate the OpenSSL files.
-        compiler = new_compiler()
-        lib_dirs.extend(compiler.library_dirs)
-        inc_dirs.extend(compiler.include_dirs)
-        # Now the platform defaults
-        lib_dirs.extend(['/lib64', '/usr/lib64', '/lib', '/usr/lib'])
-        inc_dirs.extend(['/usr/include'])
-        # Find the SSL library directory
-        ssl_libs = ['ssl', 'crypto']
-        ssl_lib = compiler.find_library_file(lib_dirs, 'ssl')
-        if ssl_lib is None:
-            ssl_lib_dirs = ['/usr/local/ssl/lib', '/usr/contrib/ssl/lib']
-            ssl_lib = compiler.find_library_file(ssl_lib_dirs, 'ssl')
-            if ssl_lib is not None:
-                ssl_lib_dirs.append(os.path.dirname(ssl_lib))
-            else:
-                ssl_libs = None
-        # Locate the SSL headers
-        for ssl_inc_dir in inc_dirs + ['/usr/local/ssl/include',
-                                       '/usr/contrib/ssl/include']:
-            ssl_h = os.path.join(ssl_inc_dir, 'openssl', 'ssl.h')
-            if os.path.exists(ssl_h):
-                if ssl_inc_dir not in inc_dirs:
-                    ssl_inc_dirs.append(ssl_inc_dir)
-                break
-    elif os.name == 'nt':
-        # (from Python's PCbuild/build_ssl.py, in find_best_ssl_dir())
-        # Look for SSL 1 level up from here.  That is, the same place the
-        # other externals for Python core live.
-        # note: do not abspath src_dir; the build will fail if any
-        # higher up directory name has spaces in it.
-        src_dir = '..'
-        try:
-            fnames = os.listdir(src_dir)
-        except OSError:
-            fnames = []
-        ssl_dir = None
-        best_parts = []
-        for fname in fnames:
-            fqn = os.path.join(src_dir, fname)
-            if os.path.isdir(fqn) and fname.startswith("openssl-"):
-                # We have a candidate, determine the best
-                parts = re.split("[.-]", fname)[1:]
-                # Ignore all "beta" or any other qualifiers;
-                # eg - openssl-0.9.7-beta1
-                if len(parts) < 4 and parts > best_parts:
-                    best_parts = parts
-                    ssl_dir = fqn
-        if ssl_dir is not None:
-            ssl_libs = ['gdi32', 'user32', 'advapi32',
-                        os.path.join(ssl_dir, 'out32', 'libeay32')]
-            ssl_inc_dir = os.path.join(ssl_dir, 'inc32')
-            ssl_inc_dirs.append(ssl_inc_dir)
-
-    # Find out which version of OpenSSL we have
-    openssl_ver = 0
-    openssl_ver_re = re.compile(
-        '^\s*#\s*define\s+OPENSSL_VERSION_NUMBER\s+(0x[0-9a-fA-F]+)' )
-    if ssl_inc_dir is not None:
-        opensslv_h = os.path.join(ssl_inc_dir, 'openssl', 'opensslv.h')
-        try:
-            incfile = open(opensslv_h, 'r')
-            for line in incfile:
-                m = openssl_ver_re.match(line)
-                if m:
-                    openssl_ver = int(m.group(1), 16)
-        except IOError:
-            e = str(sys.last_value)
-            print("IOError while reading %s: %s" % (opensslv_h, e))
-
-    # Now we can determine which extension modules need to be built.
-    exts = []
-    if ssl_libs is not None and openssl_ver >= 0x907000:
-        # The _hashlib module wraps optimized implementations
-        # of hash functions from the OpenSSL library.
-        exts.append(Extension('distutils2._backport._hashlib',
-                              ['distutils2/_backport/_hashopenssl.c'],
-                              include_dirs=ssl_inc_dirs,
-                              library_dirs=ssl_lib_dirs,
-                              libraries=ssl_libs))
-    else:
-        # no openssl at all, use our own md5 and sha1
-        exts.append(Extension('distutils2._backport._sha',
-                              ['distutils2/_backport/shamodule.c']))
-        exts.append(Extension('distutils2._backport._md5',
-                              sources=['distutils2/_backport/md5module.c',
-                                       'distutils2/_backport/md5.c'],
-                              depends=['distutils2/_backport/md5.h']) )
-    if openssl_ver < 0x908000:
-        # OpenSSL doesn't do these until 0.9.8 so we'll bring our own
-        exts.append(Extension('distutils2._backport._sha256',
-                              ['distutils2/_backport/sha256module.c']))
-        exts.append(Extension('distutils2._backport._sha512',
-                              ['distutils2/_backport/sha512module.c']))
-    return exts
 
 setup_kwargs = cfg_to_args('setup.cfg')
-if sys.version < '2.5':
-    setup_kwargs['ext_modules'] = prepare_hashlib_extensions()
 setup(**setup_kwargs)
diff --git a/tests.sh b/tests.sh
--- a/tests.sh
+++ b/tests.sh
@@ -1,40 +1,29 @@
 #!/bin/sh
-echo -n "Running tests with Python 2.4... "
-python2.4 setup.py build_ext -f -q 2> /dev/null > /dev/null
-python2.4 -Wd runtests.py -q
+echo -n "Running tests with Python 3.1... "
+python3.1 -Wd runtests.py -q
 if [ $? -ne 0 ];then
     echo Failed, re-running
-    python2.4 -Wd runtests.py
+    python3.1 -Wd runtests.py
     exit 1
 else
     echo Success
 fi
 
-echo -n "Running tests with Python 2.5... "
-python2.5 -Wd runtests.py -q
+echo -n "Running tests with Python 3.2... "
+python3.2 -Wd runtests.py -q
 if [ $? -ne 0 ];then
     echo Failed, re-running
-    python2.5 -Wd runtests.py
+    python3.2 -Wd runtests.py
     exit 1
 else
     echo Success
 fi
 
-echo -n "Running tests with Python 2.6... "
-python2.6 -Wd runtests.py -q
+echo -n "Running tests with Python 3.3... "
+python3.3 -Wd runtests.py -q
 if [ $? -ne 0 ];then
     echo Failed, re-running
-    python2.6 -Wd runtests.py
-    exit 1
-else
-    echo Success
-fi
-
-echo -n "Running tests with Python 2.7... "
-python2.7 -Wd -bb -3 runtests.py -q
-if [ $? -ne 0 ];then
-    echo Failed, re-running
-    python2.7 -Wd -bb -3 runtests.py
+    python3.3 -Wd runtests.py
     exit 1
 else
     echo Success

-- 
Repository URL: http://hg.python.org/distutils2

From solipsis at pitrou.net  Sun Sep 25 05:21:03 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sun, 25 Sep 2011 05:21:03 +0200
Subject: [Python-checkins] Daily reference leaks (5e456e1a9e8c): sum=0
Message-ID: <E1R7fH5-0000O0-PR@ap.vmr.nerim.net>

results for 5e456e1a9e8c on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogK01g79', '-x']

From python-checkins at python.org  Sun Sep 25 16:35:01 2011
From: python-checkins at python.org (mark.dickinson)
Date: Sun, 25 Sep 2011 16:35:01 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Return_+-Py=5FHUGE=5FVAL_fo?=
 =?utf8?q?r_tgamma=28+-0=29_instead_of_risking_FP_exceptions_by?=
Message-ID: <E1R7pnJ-0005I7-AV@dinsdale.python.org>

http://hg.python.org/cpython/rev/7482d0e3476a
changeset:   72467:7482d0e3476a
user:        Mark Dickinson <mdickinson at enthought.com>
date:        Sun Sep 25 15:26:43 2011 +0100
summary:
  Return +-Py_HUGE_VAL for tgamma(+-0) instead of risking FP exceptions by computing 1.0 / 0.0.

files:
  Modules/mathmodule.c |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c
--- a/Modules/mathmodule.c
+++ b/Modules/mathmodule.c
@@ -239,7 +239,8 @@
     }
     if (x == 0.0) {
         errno = EDOM;
-        return 1.0/x; /* tgamma(+-0.0) = +-inf, divide-by-zero */
+        /* tgamma(+-0.0) = +-inf, divide-by-zero */
+        return copysign(Py_HUGE_VAL, x);
     }
 
     /* integer arguments */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep 25 16:35:02 2011
From: python-checkins at python.org (mark.dickinson)
Date: Sun, 25 Sep 2011 16:35:02 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=231621=3A_Fix_undefi?=
 =?utf8?q?ned_behaviour_from_signed_overflow_in_datetime_module?=
Message-ID: <E1R7pnK-0005Jk-Gh@dinsdale.python.org>

http://hg.python.org/cpython/rev/3fb9464f9b02
changeset:   72468:3fb9464f9b02
user:        Mark Dickinson <mdickinson at enthought.com>
date:        Sun Sep 25 15:34:32 2011 +0100
summary:
  Issue #1621: Fix undefined behaviour from signed overflow in datetime module hashes, array and list iterations, and get_integer (stringlib/string_format.h)

files:
  Modules/_datetimemodule.c         |   8 ++++----
  Modules/arraymodule.c             |   2 +-
  Objects/listobject.c              |   6 +++---
  Objects/stringlib/string_format.h |  14 ++++++--------
  4 files changed, 14 insertions(+), 16 deletions(-)


diff --git a/Modules/_datetimemodule.c b/Modules/_datetimemodule.c
--- a/Modules/_datetimemodule.c
+++ b/Modules/_datetimemodule.c
@@ -2746,13 +2746,13 @@
 generic_hash(unsigned char *data, int len)
 {
     register unsigned char *p;
-    register Py_hash_t x;
+    register Py_uhash_t x;
 
     p = (unsigned char *) data;
-    x = *p << 7;
+    x = (Py_uhash_t)*p << 7;
     while (--len >= 0)
-        x = (1000003*x) ^ *p++;
-    x ^= len;
+        x = (1000003U*x) ^ (Py_uhash_t)*p++;
+    x ^= (Py_uhash_t)len;
     if (x == -1)
         x = -2;
 
diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c
--- a/Modules/arraymodule.c
+++ b/Modules/arraymodule.c
@@ -2351,7 +2351,7 @@
                 self->ob_item + (cur + 1) * itemsize,
                 lim * itemsize);
         }
-        cur = start + slicelength * step;
+        cur = start + (size_t)slicelength * step;
         if (cur < (size_t)Py_SIZE(self)) {
             memmove(self->ob_item + (cur-slicelength) * itemsize,
                 self->ob_item + cur * itemsize,
diff --git a/Objects/listobject.c b/Objects/listobject.c
--- a/Objects/listobject.c
+++ b/Objects/listobject.c
@@ -2434,7 +2434,7 @@
             src = self->ob_item;
             dest = ((PyListObject *)result)->ob_item;
             for (cur = start, i = 0; i < slicelength;
-                 cur += step, i++) {
+                 cur += (size_t)step, i++) {
                 it = src[cur];
                 Py_INCREF(it);
                 dest[i] = it;
@@ -2525,7 +2525,7 @@
                     self->ob_item + cur + 1,
                     lim * sizeof(PyObject *));
             }
-            cur = start + slicelength*step;
+            cur = start + (size_t)slicelength * step;
             if (cur < (size_t)Py_SIZE(self)) {
                 memmove(self->ob_item + cur - slicelength,
                     self->ob_item + cur,
@@ -2589,7 +2589,7 @@
             selfitems = self->ob_item;
             seqitems = PySequence_Fast_ITEMS(seq);
             for (cur = start, i = 0; i < slicelength;
-                 cur += step, i++) {
+                 cur += (size_t)step, i++) {
                 garbage[i] = selfitems[cur];
                 ins = seqitems[i];
                 Py_INCREF(ins);
diff --git a/Objects/stringlib/string_format.h b/Objects/stringlib/string_format.h
--- a/Objects/stringlib/string_format.h
+++ b/Objects/stringlib/string_format.h
@@ -209,19 +209,17 @@
         if (digitval < 0)
             return -1;
         /*
-           This trick was copied from old Unicode format code.  It's cute,
-           but would really suck on an old machine with a slow divide
-           implementation.  Fortunately, in the normal case we do not
-           expect too many digits.
+           Detect possible overflow before it happens:
+
+              accumulator * 10 + digitval > PY_SSIZE_T_MAX if and only if
+              accumulator > (PY_SSIZE_T_MAX - digitval) / 10.
         */
-        oldaccumulator = accumulator;
-        accumulator *= 10;
-        if ((accumulator+10)/10 != oldaccumulator+1) {
+        if (accumulator > (PY_SSIZE_T_MAX - digitval) / 10) {
             PyErr_Format(PyExc_ValueError,
                          "Too many decimal digits in format string");
             return -1;
         }
-        accumulator += digitval;
+        accumulator = accumulator * 10 + digitval;
     }
     return accumulator;
 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep 25 17:37:06 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Sun, 25 Sep 2011 17:37:06 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Depend_setobjec?=
 =?utf8?q?t=2Eo_and_dictobject=2Eo_on_stringlib/eq=2Eh=2E?=
Message-ID: <E1R7qlO-0001kW-5M@dinsdale.python.org>

http://hg.python.org/cpython/rev/4951ffc178db
changeset:   72469:4951ffc178db
branch:      3.2
parent:      72459:b378864d8ff3
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Sun Sep 25 17:36:11 2011 +0200
summary:
  Depend setobject.o and dictobject.o on stringlib/eq.h.

files:
  Makefile.pre.in |  3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)


diff --git a/Makefile.pre.in b/Makefile.pre.in
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -641,6 +641,9 @@
 Objects/unicodeobject.o: $(srcdir)/Objects/unicodeobject.c \
 				$(BYTESTR_DEPS)
 
+Objects/dictobject.o: $(srcdir)/Objects/stringlib/eq.h
+Objects/setobject.o: $(srcdir)/Objects/stringlib/eq.h
+
 $(OPCODETARGETS_H): $(OPCODETARGETGEN_FILES)
 	$(OPCODETARGETGEN) $(OPCODETARGETS_H)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep 25 17:37:06 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Sun, 25 Sep 2011 17:37:06 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_merged?=
Message-ID: <E1R7qlO-0001kg-Ua@dinsdale.python.org>

http://hg.python.org/cpython/rev/2b47f0146639
changeset:   72470:2b47f0146639
parent:      72468:3fb9464f9b02
parent:      72469:4951ffc178db
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Sun Sep 25 17:36:31 2011 +0200
summary:
  merged

files:
  Makefile.pre.in |  3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)


diff --git a/Makefile.pre.in b/Makefile.pre.in
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -641,6 +641,9 @@
 Objects/unicodeobject.o: $(srcdir)/Objects/unicodeobject.c \
 				$(BYTESTR_DEPS)
 
+Objects/dictobject.o: $(srcdir)/Objects/stringlib/eq.h
+Objects/setobject.o: $(srcdir)/Objects/stringlib/eq.h
+
 $(OPCODETARGETS_H): $(OPCODETARGETGEN_FILES)
 	$(OPCODETARGETGEN) $(OPCODETARGETS_H)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Sep 25 22:58:17 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Sun, 25 Sep 2011 22:58:17 +0200
Subject: [Python-checkins] =?utf8?q?peps=3A_Update_to_current_object_layou?=
	=?utf8?q?t=2E?=
Message-ID: <E1R7vmD-0001F9-3p@dinsdale.python.org>

http://hg.python.org/peps/rev/a97dfa0fa127
changeset:   3944:a97dfa0fa127
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Sun Sep 25 22:58:13 2011 +0200
summary:
  Update to current object layout.

files:
  pep-0393.txt |  191 ++++++++++++++++++++++----------------
  1 files changed, 112 insertions(+), 79 deletions(-)


diff --git a/pep-0393.txt b/pep-0393.txt
--- a/pep-0393.txt
+++ b/pep-0393.txt
@@ -47,52 +47,88 @@
 For many strings (e.g. ASCII), multiple representations may actually
 share memory (e.g. the shortest form may be shared with the UTF-8 form
 if all characters are ASCII). With such sharing, the overhead of
-compatibility representations is reduced.
+compatibility representations is reduced. If representations do share
+data, it is also possible to omit structure fields, reducing the base
+size of string objects.
 
 Specification
 =============
 
-The Unicode object structure is changed to this definition::
+Unicode structures are now defined as a hierarchy of structures,
+namely::
 
   typedef struct {
     PyObject_HEAD
     Py_ssize_t length;
+    Py_hash_t hash;
+    struct {
+        unsigned int interned:2;
+        unsigned int kind:2;
+        unsigned int compact:1;
+        unsigned int ascii:1;
+        unsigned int ready:1;
+    } state;
+    wchar_t *wstr;
+  } PyASCIIObject;
+
+  typedef struct {
+    PyASCIIObject _base;
+    Py_ssize_t utf8_length;
+    char *utf8;
+    Py_ssize_t wstr_length;
+  } PyCompactUnicodeObject;
+
+  typedef struct {
+    PyCompactUnicodeObject _base;
     union {
         void *any;
         Py_UCS1 *latin1;
         Py_UCS2 *ucs2;
         Py_UCS4 *ucs4;
     } data;
-    Py_hash_t hash;
-    int state;
-    Py_ssize_t utf8_length;
-    void *utf8;
-    Py_ssize_t wstr_length;
-    void *wstr;
   } PyUnicodeObject;
 
-These fields have the following interpretations:
+Objects for which both size and maximum character are known at
+creation time are called "compact" unicode objects; character data
+immediately follow the base structure. If the maximum character is
+less than 128, they use the PyASCIIObject structure, and the UTF-8
+data, the UTF-8 length and the wstr length are the same as the length
+and the ASCII data. For non-ASCII strings, the PyCompactObject
+structure is used. Resizing compact objects is not supported.
+
+Objects for which the maximum character is not given at creation time
+are called "legacy" objects, created through
+PyUnicode_FromStringAndSize(NULL, length). They use the
+PyUnicodeObject structure. Initially, their data is only in the wstr
+pointer; when PyUnicode_READY is called, the data pointer (union) is
+allocated. Resizing is possible as long PyUnicode_READY has not been
+called.
+
+The fields have the following interpretations:
 
 - length: number of code points in the string (result of sq_length)
-- data: shortest-form representation of the unicode string.
-  The string is null-terminated (in its respective representation).
-- hash: same as in Python 3.2
-- state:
-
-  * lowest 2 bits (mask 0x03) - interned-state (SSTATE_*) as in 3.2
-  * next 2 bits (mask 0x0C) - form of str:
-
+- interned: interned-state (SSTATE_*) as in 3.2
+- kind: form of string
     + 00 => str is not initialized (data are in wstr)
     + 01 => 1 byte (Latin-1)
     + 10 => 2 byte (UCS-2)
     + 11 => 4 byte (UCS-4);
-
-  * next bit (mask 0x10): 1 if str memory follows PyUnicodeObject  
-
-- utf8_length, utf8: UTF-8 representation (null-terminated)
+- compact: the object uses one of the compact representations
+  (implies ready)
+- ascii: the object uses the PyASCIIObject representation
+  (implies compact and ready)
+- ready: the canonical represenation is ready to be accessed through
+  PyUnicode_DATA and PyUnicode_GET_LENGTH. This is set either if the
+  object is compact, or the data pointer and length have been
+  initialized.
 - wstr_length, wstr: representation in platform's wchar_t
   (null-terminated). If wchar_t is 16-bit, this form may use surrogate
   pairs (in which cast wstr_length differs form length).
+  wstr_length differs from length only if there are surrogate pairs
+  in the representation.
+- utf8_length, utf8: UTF-8 representation (null-terminated).
+- data: shortest-form representation of the unicode string.
+  The string is null-terminated (in its respective representation).
 
 All three representations are optional, although the data form is
 considered the canonical representation which can be absent only
@@ -111,10 +147,6 @@
 BMP-not-Latin-1 characters if sizeof(wchar_t) is 2, and uses some
 non-BMP characters if sizeof(wchar_t) is 4).
 
-If the string is created directly with the canonical representation
-(see below), this representation doesn't take a separate memory block,
-but is allocated right after the PyUnicodeObject struct.
-
 String Creation
 ---------------
 
@@ -140,12 +172,11 @@
 or implicitly). Resizing a Unicode string remains possible until it
 is finalized.
 
-PyUnicode_Ready() converts a string containing only a wstr
+PyUnicode_READY() converts a string containing only a wstr
 representation into the canonical representation. Unless wstr and data
 can share the memory, the wstr representation is discarded after the
-conversion. PyUnicode_FAST_READY() is a wrapper that avoids the 
-function call if the string is already ready. Both APIs return 0
-on success and -1 on failure.
+conversion. The macro returns 0 on success and -1 on failure, which
+happens in particular if the memory allocation fails.
 
 String Access
 -------------
@@ -175,9 +206,6 @@
 converts a string to a char* (such as the ParseTuple functions) will
 use PyUnicode_AsUTF8 to compute a conversion.
 
-PyUnicode_AsUnicode is deprecated; it computes the wstr representation
-on first use.
-
 Stable ABI
 ----------
 
@@ -189,27 +217,37 @@
 about the internals of CPython's data types, include PyUnicodeObject
 instances.  It will need to be slightly updated to track the change.
 
+Deprecations, Removals, and Incompatibilities
+---------------------------------------------
+
+While the Py_UNICODE representation and APIs are deprecated with this
+PEP, no removal of the respective APIs is scheduled. The APIs should
+remain available at least five years after the PEP is accepted; before
+they are removed, existing extension modules should be studied to find
+out whether a sufficient majority of the open-source code on PyPI has
+been ported to the new API. A reasonable motivation for using the
+deprecated API even in new code is for code that shall work both on
+Python 2 and Python 3.
+
+_PyUnicode_AsDefaultEncodedString is removed. It previously returned a
+borrowed reference to an UTF-8-encoded bytes object. Since the unicode
+object cannot anymore cache such a reference, implementing it without
+leaking memory is not possible. No deprecation phase is provided,
+since it was an API for internal use only.
+
+Extension modules using the legacy API may inadvertently call
+PyUnicode_READY, by calling some API that requires that the object is
+ready, and then continue accessing the (now invalid) Py_UNICODE
+pointer. Such code will break with this PEP. The code was already
+flawed in 3.2, as there is was no explicit guarantee that the
+PyUnicode_AS_UNICODE result would stay valid after an API call (due to
+the possiblity of string resizing). Modules that face this issue
+need to re-fetch the Py_UNICODE pointer after API calls; doing
+so will continue to work correctly in earlier Python versions.
+
 Open Issues
 ===========
 
-- When an application uses the legacy API, it may hold onto
-  the Py_UNICODE* representation, and yet start calling Unicode
-  APIs, which would call PyUnicode_Ready, invalidating the 
-  Py_UNICODE* representation; this would be an incompatible change.
-  The following solutions can be considered:
-
-  * accept it as an incompatible change. Applications using the
-    legacy API will have to fill out the Py_UNICODE buffer completely
-    before calling any API on the string under construction.
-  * require explicit PyUnicode_Ready calls in such applications;
-    fail with a fatal error if a non-ready string is ever read.
-    This would also be an incompatible change, but one that is
-    more easily detected during testing.
-  * as a compromise between these approaches, implicit PyUnicode_Ready
-    calls (i.e. those not deliberately following the construction of
-    a PyUnicode object) could produce a warning if they convert an
-    object.
-
 - Which of the APIs created during the development of the PEP should
   be public?
 
@@ -226,11 +264,6 @@
 applications that care about this problem can be rewritten to use the
 data representation.
 
-The question was raised whether the wchar_t representation is
-discouraged, or scheduled for removal. This is not the intent of this
-PEP; applications that use them will see a performance penalty,
-though. Future versions of Python may consider to remove them.
-
 Performance
 -----------
 
@@ -240,31 +273,31 @@
 a reduction in memory usage. For small strings, the effects depend on
 the pointer size of the system, and the size of the Py_UNICODE/wchar_t
 type. The following table demonstrates this for various small ASCII
-string sizes and platforms.
+and Latin-1 string sizes and platforms.
 
-+-------+---------------------------------+----------------+
-|string | Python 3.2                      | This PEP       |
-|size   +----------------+----------------+                |
-|       | 16-bit wchar_t | 32-bit wchar_t |                |
-|       +---------+------+--------+-------+--------+-------+
-|       | 32-bit  |64-bit| 32-bit |64-bit | 32-bit |64-bit |
-+-------+---------+------+--------+-------+--------+-------+
-|1      | 40      | 64   | 40     |  64   | 48     | 88    |
-+-------+---------+------+--------+-------+--------+-------+
-|2      | 40      | 64   | 48     |  72   | 48     | 88    |
-+-------+---------+------+--------+-------+--------+-------+
-|3      | 40      | 64   | 48     |  72   | 48     | 88    |
-+-------+---------+------+--------+-------+--------+-------+
-|4      | 48      | 72   | 56     |  80   | 48     | 88    |
-+-------+---------+------+--------+-------+--------+-------+
-|5      | 48      | 72   | 56     |  80   | 48     | 88    |
-+-------+---------+------+--------+-------+--------+-------+
-|6      | 48      | 72   | 64     |  88   | 48     | 88    |
-+-------+---------+------+--------+-------+--------+-------+
-|7      | 48      | 72   | 64     |  88   | 48     | 88    |
-+-------+---------+------+--------+-------+--------+-------+
-|8      | 56      | 80   | 72     |  96   | 56     | 88    |
-+-------+---------+------+--------+-------+--------+-------+
++-------+---------------------------------+---------------------------------+
+|string | Python 3.2                      | This PEP                        |
+|size   +----------------+----------------+----------------+----------------+
+|       | 16-bit wchar_t | 32-bit wchar_t |   ASCII        |   Latin-1      |
+|       +---------+------+--------+-------+--------+-------+--------+-------+
+|       | 32-bit  |64-bit| 32-bit |64-bit | 32-bit |64-bit | 32-bit |64-bit |
++-------+---------+------+--------+-------+--------+-------+--------+-------+
+|1      | 32      | 64   | 40     |  64   | 32     | 56    | 40     | 80    |
++-------+---------+------+--------+-------+--------+-------+--------+-------+
+|2      | 40      | 64   | 40     |  72   | 32     | 56    | 40     | 80    |
++-------+---------+------+--------+-------+--------+-------+--------+-------+
+|3      | 40      | 64   | 48     |  72   | 32     | 56    | 40     | 80    |
++-------+---------+------+--------+-------+--------+-------+--------+-------+
+|4      | 40      | 72   | 48     |  80   | 32     | 56    | 48     | 80    |
++-------+---------+------+--------+-------+--------+-------+--------+-------+
+|5      | 40      | 72   | 56     |  80   | 32     | 56    | 48     | 80    |
++-------+---------+------+--------+-------+--------+-------+--------+-------+
+|6      | 48      | 72   | 56     |  88   | 32     | 56    | 48     | 80    |
++-------+---------+------+--------+-------+--------+-------+--------+-------+
+|7      | 48      | 72   | 64     |  88   | 32     | 56    | 48     | 80    |
++-------+---------+------+--------+-------+--------+-------+--------+-------+
+|8      | 48      | 80   | 64     |  96   | 40     | 64    | 48     | 88    |
++-------+---------+------+--------+-------+--------+-------+--------+-------+
 
 The runtime effect is significantly affected by the API being
 used. After porting the relevant pieces of code to the new API,

-- 
Repository URL: http://hg.python.org/peps

From solipsis at pitrou.net  Mon Sep 26 05:21:02 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Mon, 26 Sep 2011 05:21:02 +0200
Subject: [Python-checkins] Daily reference leaks (2b47f0146639): sum=0
Message-ID: <E1R81kc-0000Fn-Il@ap.vmr.nerim.net>

results for 2b47f0146639 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogK8G21W', '-x']

From ncoghlan at gmail.com  Mon Sep 26 05:47:40 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 26 Sep 2011 13:47:40 +1000
Subject: [Python-checkins] cpython: Issue #12981: rewrite
 multiprocessing_{sendfd, recvfd} in Python.
In-Reply-To: <E1R7Wa1-00037I-Lc@dinsdale.python.org>
References: <E1R7Wa1-00037I-Lc@dinsdale.python.org>
Message-ID: <CADiSq7f+xgWUkU6_9yKsOSaGeeJEE7qE0TqkLPoPqKbV=5EsZw@mail.gmail.com>

On Sun, Sep 25, 2011 at 4:04 AM, charles-francois.natali
<python-checkins at python.org> wrote:
> +if not(sys.platform == 'win32' or (hasattr(socket, 'CMSG_LEN') and
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? hasattr(socket, 'SCM_RIGHTS'))):
> ? ? raise ImportError('pickling of connections not supported')

I'm pretty sure the functionality checks for CMSG_LEN and SCM_RIGHTS
mean the platform check for Windows is now redundant.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia

From python-checkins at python.org  Mon Sep 26 12:25:54 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Mon, 26 Sep 2011 12:25:54 +0200
Subject: [Python-checkins] =?utf8?q?peps=3A_Address_GvR=27s_comments=2E?=
Message-ID: <E1R88Nm-00084H-9E@dinsdale.python.org>

http://hg.python.org/peps/rev/81a533fd93f2
changeset:   3945:81a533fd93f2
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Mon Sep 26 12:25:49 2011 +0200
summary:
  Address GvR's comments.

files:
  pep-0393.txt |  63 ++++++++++++++++++++++++++++++++++++++-
  1 files changed, 61 insertions(+), 2 deletions(-)


diff --git a/pep-0393.txt b/pep-0393.txt
--- a/pep-0393.txt
+++ b/pep-0393.txt
@@ -19,7 +19,8 @@
 representation in common cases, but give access to full UCS-4 on all
 systems. For compatibility with existing APIs, several representations
 may exist in parallel; over time, this compatibility should be phased
-out.
+out. The distinction between narrow and wide Unicode builds is
+dropped.  An implementation of this PEP is available at [1]_.
 
 Rationale
 =========
@@ -206,10 +207,62 @@
 converts a string to a char* (such as the ParseTuple functions) will
 use PyUnicode_AsUTF8 to compute a conversion.
 
+New API
+-------
+
+This section summarizes the API additions.
+
+Macros to access the internal representation of a Unicode object
+(read-only):
+
+- PyUnicode_IS_COMPACT_ASCII(o), PyUnicode_IS_COMPACT(o),
+  PyUnicode_IS_READY(o)
+- PyUnicode_GET_LENGTH(o)
+- PyUnicode_KIND(o), PyUnicode_CHARACTER_SIZE(o),
+  PyUnicode_MAX_CHAR_VALUE(o)
+- PyUnicode_DATA(o), PyUnicode_1BYTE_DATA(o), PyUnicode_2BYTE_DATA(o),
+  PyUnicode_4BYTE_DATA(o)
+
+Character access macros:
+
+- PyUnicode_READ(kind, data, index), PyUnicode_READ_CHAR(o, index)
+- PyUnicode_WRITE(kind, data, index, value)
+
+Other macros:
+
+- PyUnicode_READY(o)
+- PyUnicode_CONVERT_BYTES(from_type, tp_type, begin, end, to)
+
+String creation functions:
+
+- PyUnicode_New(size, maxchar)
+- PyUnicode_FromKindAndData(kind, data, size)
+- PyUnicode_Substring(o, start, end)
+- PyUnicode_Chr(ch)
+
+Character access utility functions:
+
+- PyUnicode_CopyCharacters(to, to_start, from, from_start, how_many)
+- PyUnicode_FindChar(str, ch, start, end, direction)
+
+Representation conversion:
+
+- PyUnicode_AsUCS4(o, buffer, buflen)
+- PyUnicode_AsUCS4Copy(o)
+- PyUnicode_AsUnicodeAndSize(o, size_out)
+- PyUnicode_AsUTF8(o)
+- PyUnicode_AsUTF8AndSize(o, size_out)
+
+UCS4 utility functions:
+
+- Py_UCS4_{strlen, strcpy, strcat, strncpy, strcmp, strncpy, strcmp,
+  strncmp, strchr, strrchr}
+
 Stable ABI
 ----------
 
-None of the functions in this PEP become part of the stable ABI.
+None of the functions in this PEP become part of the stable ABI
+(PEP 384).
 
 GDB Debugging Hooks
 -------------------
@@ -363,6 +416,12 @@
 PyUnicode_CopyCharacters help in analyzing and creating string
 objects, operating on indices instead of data pointers.
 
+References
+==========
+
+.. [1] PEP 393 branch
+       https://bitbucket.org/t0rsten/pep-393
+
 Copyright
 =========
 

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Mon Sep 26 12:32:36 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Mon, 26 Sep 2011 12:32:36 +0200
Subject: [Python-checkins] =?utf8?q?peps=3A_All_open_issues_are_now_addres?=
	=?utf8?q?sed=2E?=
Message-ID: <E1R88UG-0000Bg-13@dinsdale.python.org>

http://hg.python.org/peps/rev/4473ac0f72dc
changeset:   3946:4473ac0f72dc
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Mon Sep 26 12:32:32 2011 +0200
summary:
  All open issues are now addressed.

files:
  pep-0393.txt |  6 ------
  1 files changed, 0 insertions(+), 6 deletions(-)


diff --git a/pep-0393.txt b/pep-0393.txt
--- a/pep-0393.txt
+++ b/pep-0393.txt
@@ -298,12 +298,6 @@
 need to re-fetch the Py_UNICODE pointer after API calls; doing
 so will continue to work correctly in earlier Python versions.
 
-Open Issues
-===========
-
-- Which of the APIs created during the development of the PEP should
-  be public?
-
 Discussion
 ==========
 

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Mon Sep 26 18:02:24 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 26 Sep 2011 18:02:24 +0200
Subject: [Python-checkins] =?utf8?q?distutils2_=28merge_default_-=3E_pytho?=
 =?utf8?q?n3=29=3A_Merge_devnotes_update?=
Message-ID: <E1R8DdQ-00026g-WB@dinsdale.python.org>

http://hg.python.org/distutils2/rev/51bb31c537cb
changeset:   1196:51bb31c537cb
branch:      python3
parent:      1194:d9ce66d22bb5
parent:      1195:9e4d083c4ad6
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Sep 24 01:19:50 2011 +0200
summary:
  Merge devnotes update

files:
  DEVNOTES.txt |  18 +++++++++++-------
  1 files changed, 11 insertions(+), 7 deletions(-)


diff --git a/DEVNOTES.txt b/DEVNOTES.txt
--- a/DEVNOTES.txt
+++ b/DEVNOTES.txt
@@ -1,12 +1,16 @@
 Notes for developers
 ====================
 
-- Distutils2 runs on Python from 2.4 to 2.7 so make sure you don't use code
-  that doesn't work under one of these Python versions.
+- Distutils2 runs on Python from 3.1 to 3.2 so make sure you don't use code
+  that doesn't work under one of these Python versions.  The version compatible
+  with 2.4-2.7 in the "default" branch.
 
-- For 2.4, you need to run "python2.4 setup.py build" before you can run tests
-  or pysetup.
+- When merging default into python3, don't keep maximum compatibility with
+  Python 2 but use idiomatic 3.x code.  For difficult conversions like encoding
+  handling with I/O, you can have a look or use a diff tool with the same file
+  in distutils or packaging from Python 3.3.
 
-- Always run tests.sh before you push a change. This implies
-  that you have all Python versions installed from 2.4 to 2.7. Be sure to have 
-  docutils installed on all python versions no avoid skipping tests as well.
+- Always run tests.sh before you commit a change.  This implies that you have
+  Python 3.1 and 3.2 installed as well as a checkout of Python 3.3 with its
+  executable available as "python3.3" in your $PATH.  Be sure to also have
+  docutils installed on all Python versions to avoid skipping tests.

-- 
Repository URL: http://hg.python.org/distutils2

From solipsis at pitrou.net  Tue Sep 27 05:22:43 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Tue, 27 Sep 2011 05:22:43 +0200
Subject: [Python-checkins] Daily reference leaks (2b47f0146639): sum=0
Message-ID: <E1R8OFn-0000TD-R5@ap.vmr.nerim.net>

results for 2b47f0146639 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog6GIDyV', '-x']

From python-checkins at python.org  Tue Sep 27 07:28:39 2011
From: python-checkins at python.org (georg.brandl)
Date: Tue, 27 Sep 2011 07:28:39 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_markup=2E?=
Message-ID: <E1R8QDf-0005VD-SM@dinsdale.python.org>

http://hg.python.org/cpython/rev/84280fac98b9
changeset:   72471:84280fac98b9
user:        Georg Brandl <georg at python.org>
date:        Tue Sep 27 07:30:00 2011 +0200
summary:
  Fix markup.

files:
  Doc/library/array.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/library/array.rst b/Doc/library/array.rst
--- a/Doc/library/array.rst
+++ b/Doc/library/array.rst
@@ -52,8 +52,8 @@
 
 (2)
    The ``'q'`` and ``'Q'`` type codes are available only if
-   the platform C compiler used to build Python supports C :ctype:`long long`,
-   or, on Windows, :ctype:`__int64`.
+   the platform C compiler used to build Python supports C :c:type:`long long`,
+   or, on Windows, :c:type:`__int64`.
 
    .. versionadded:: 3.3
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 00:09:42 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Wed, 28 Sep 2011 00:09:42 +0200
Subject: [Python-checkins] =?utf8?q?peps=3A_Add_character_accessor_functio?=
 =?utf8?q?ns=2C_and_put_representation-independent_functions?=
Message-ID: <E1R8fqQ-00071T-96@dinsdale.python.org>

http://hg.python.org/peps/rev/560454f6cd2f
changeset:   3947:560454f6cd2f
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Tue Sep 27 23:46:28 2011 +0200
summary:
  Add character accessor functions, and put representation-independent functions into the stable ABI.

files:
  pep-0393.txt |  10 +++++++---
  1 files changed, 7 insertions(+), 3 deletions(-)


diff --git a/pep-0393.txt b/pep-0393.txt
--- a/pep-0393.txt
+++ b/pep-0393.txt
@@ -242,6 +242,8 @@
 
 Character access utility functions:
 
+- PyUnicode_GetLength(o), PyUnicode_ReadChar(o, index), 
+  PyUnicode_WriteChar(o, index, character)
 - PyUnicode_CopyCharacters(to, to_start, from, from_start, how_many)
 - PyUnicode_FindChar(str, ch, start, end, direction)
 
@@ -261,14 +263,16 @@
 Stable ABI
 ----------
 
-None of the functions in this PEP become part of the stable ABI
-(PEP 384).
+The following functions are added to the stable ABI (PEP 384), as they
+are independent of the actual representation of Unicode objects:
+PyUnicode_New, PyUnicode_Chr, PyUnicode_GetLength, PyUnicode_ReadChar,
+PyUnicode_WriteChar, PyUnicode_Find, PyUnicode_FindChar.
 
 GDB Debugging Hooks
 -------------------
 Tools/gdb/libpython.py contains debugging hooks that embed knowledge
 about the internals of CPython's data types, include PyUnicodeObject
-instances.  It will need to be slightly updated to track the change.
+instances.  It has been updated to track the change.
 
 Deprecations, Removals, and Incompatibilities
 ---------------------------------------------

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Wed Sep 28 00:09:43 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Wed, 28 Sep 2011 00:09:43 +0200
Subject: [Python-checkins] =?utf8?q?peps=3A_Guido_pronounced_that_the_PEP_?=
	=?utf8?q?is_accepted=2E?=
Message-ID: <E1R8fqR-00071j-Py@dinsdale.python.org>

http://hg.python.org/peps/rev/ec90ccb58ec1
changeset:   3949:ec90ccb58ec1
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Wed Sep 28 00:09:37 2011 +0200
summary:
  Guido pronounced that the PEP is accepted.

files:
  pep-0393.txt |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/pep-0393.txt b/pep-0393.txt
--- a/pep-0393.txt
+++ b/pep-0393.txt
@@ -3,7 +3,7 @@
 Version: $Revision$
 Last-Modified: $Date$
 Author: Martin v. L?wis <martin at v.loewis.de>
-Status: Draft
+Status: Accepted
 Type: Standards Track
 Content-Type: text/x-rst
 Created: 24-Jan-2010

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Wed Sep 28 00:09:43 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Wed, 28 Sep 2011 00:09:43 +0200
Subject: [Python-checkins] =?utf8?q?peps=3A_Address_Victor=27s_comments=2E?=
Message-ID: <E1R8fqR-00071e-2f@dinsdale.python.org>

http://hg.python.org/peps/rev/5c69799e1a94
changeset:   3948:5c69799e1a94
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Wed Sep 28 00:08:51 2011 +0200
summary:
  Address Victor's comments.

files:
  pep-0393.txt |  31 +++++++++++++++++++++++--------
  1 files changed, 23 insertions(+), 8 deletions(-)


diff --git a/pep-0393.txt b/pep-0393.txt
--- a/pep-0393.txt
+++ b/pep-0393.txt
@@ -169,7 +169,7 @@
 PyUnicode_FromUnicode remains supported but is deprecated. If the
 Py_UNICODE pointer is non-null, the data representation is set. If the
 pointer is NULL, a properly-sized wstr representation is allocated,
-which can be modified until PyUnicode_Ready() is called (explicitly
+which can be modified until PyUnicode_READY() is called (explicitly
 or implicitly). Resizing a Unicode string remains possible until it
 is finalized.
 
@@ -192,10 +192,9 @@
  - PyUnciode_READ(kind, data, index)
  - PyUnicode_WRITE(kind, data, index, value)
  - PyUnicode_READ_CHAR(unicode, index)
- - PyUnicode_WRITE_CHAR(unicode, index, value)
 
 All these macros assume that the string is in canonical form;
-callers need to ensure this by calling PyUnicode_FAST_READY.
+callers need to ensure this by calling PyUnicode_READY.
 
 A new function PyUnicode_AsUTF8 is provided to access the UTF-8
 representation. It is thus identical to the existing
@@ -231,14 +230,13 @@
 Other macros:
 
 - PyUnicode_READY(o)
-- PyUnicode_CONVERT_BYTES(from_type, tp_type, begin, end, to)
+- PyUnicode_CONVERT_BYTES(from_type, to_type, begin, end, to)
 
 String creation functions:
 
 - PyUnicode_New(size, maxchar)
 - PyUnicode_FromKindAndData(kind, data, size)
 - PyUnicode_Substring(o, start, end)
-- PyUnicode_Chr(ch)
 
 Character access utility functions:
 
@@ -265,8 +263,9 @@
 
 The following functions are added to the stable ABI (PEP 384), as they
 are independent of the actual representation of Unicode objects:
-PyUnicode_New, PyUnicode_Chr, PyUnicode_GetLength, PyUnicode_ReadChar,
-PyUnicode_WriteChar, PyUnicode_Find, PyUnicode_FindChar.
+PyUnicode_New, PyUnicode_Substring, PyUnicode_GetLength,
+PyUnicode_ReadChar, PyUnicode_WriteChar, PyUnicode_Find,
+PyUnicode_FindChar.
 
 GDB Debugging Hooks
 -------------------
@@ -286,6 +285,22 @@
 deprecated API even in new code is for code that shall work both on
 Python 2 and Python 3.
 
+The following macros and functions are deprecated:
+
+- PyUnicode_FromUnicode
+- PyUnicode_GET_SIZE, PyUnicode_GetSize, PyUnicode_GET_DATA_SIZE,
+- PyUnicode_AS_UNICODE, PyUnicode_AsUnicode, PyUnicode_AsUnicodeAndSize
+- PyUnicode_COPY, PyUnicode_FILL, PyUnicode_MATCH
+- PyUnicode_Encode, PyUnicode_EncodeUTF7, PyUnicode_EncodeUTF8,
+  PyUnicode_EncodeUTF16, PyUnicode_EncodeUTF32,
+  PyUnicode_EncodeUnicodeEscape, PyUnicode_EncodeRawUnicodeEscape,
+  PyUnicode_EncodeLatin1, PyUnicode_EncodeASCII,
+  PyUnicode_EncodeCharmap, PyUnicode_TranslateCharmap,
+  PyUnicode_EncodeMBCS, PyUnicode_EncodeDecimal, 
+  PyUnicode_TransformDecimalToASCII
+- Py_UNICODE_{strlen, strcat, strcpy, strcmp, strchr, strrchr}
+- PyUnicode_AsUnicodeCopy
+
 _PyUnicode_AsDefaultEncodedString is removed. It previously returned a
 borrowed reference to an UTF-8-encoded bytes object. Since the unicode
 object cannot anymore cache such a reference, implementing it without
@@ -371,7 +386,7 @@
 
 - the Py_UNICODE type,
 - PyUnicode_AS_UNICODE and PyUnicode_AsUnicode,
-- PyUnicode_GET_LENGTH and PyUnicode_GetSize, and
+- PyUnicode_GET_SIZE and PyUnicode_GetSize, and
 - PyUnicode_FromUnicode.
 
 When iterating over an existing string, or looking at specific

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Wed Sep 28 03:57:24 2011
From: python-checkins at python.org (meador.inge)
Date: Wed, 28 Sep 2011 03:57:24 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzMDEz?=
 =?utf8?q?=3A_ctypes=3A_Fix_a_reference_leak_in_PyCArrayType=5Ffrom=5Fctyp?=
 =?utf8?q?e=2E?=
Message-ID: <E1R8jOm-0000MS-Do@dinsdale.python.org>

http://hg.python.org/cpython/rev/1fb5b0cc6367
changeset:   72472:1fb5b0cc6367
branch:      2.7
parent:      72461:5a1cb8506cea
user:        Meador Inge <meadori at gmail.com>
date:        Tue Sep 27 20:35:28 2011 -0500
summary:
  Issue #13013: ctypes: Fix a reference leak in PyCArrayType_from_ctype.

Thanks to Suman Saha for finding the bug and providing a patch.

files:
  Misc/ACKS                 |  1 +
  Misc/NEWS                 |  3 +++
  Modules/_ctypes/_ctypes.c |  1 +
  3 files changed, 5 insertions(+), 0 deletions(-)


diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -720,6 +720,7 @@
 Mark Russell
 Nick Russo
 S?bastien Sabl?
+Suman Saha
 Hajime Saitou
 George Sakkis
 Rich Salz
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -207,6 +207,9 @@
 Extension Modules
 -----------------
 
+- Issue #13013: ctypes: Fix a reference leak in PyCArrayType_from_ctype.
+  Thanks to Suman Saha for finding the bug and providing a patch.
+
 - Issue #13022: Fix: _multiprocessing.recvfd() doesn't check that
   file descriptor was actually received.
 
diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c
--- a/Modules/_ctypes/_ctypes.c
+++ b/Modules/_ctypes/_ctypes.c
@@ -4678,6 +4678,7 @@
     if (!PyType_Check(itemtype)) {
         PyErr_SetString(PyExc_TypeError,
                         "Expected a type object");
+        Py_DECREF(key);
         return NULL;
     }
 #ifdef MS_WIN64

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 03:57:25 2011
From: python-checkins at python.org (meador.inge)
Date: Wed, 28 Sep 2011 03:57:25 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMDEz?=
 =?utf8?q?=3A_ctypes=3A_Fix_a_reference_leak_in_PyCArrayType=5Ffrom=5Fctyp?=
 =?utf8?q?e=2E?=
Message-ID: <E1R8jOn-0000MW-7r@dinsdale.python.org>

http://hg.python.org/cpython/rev/58a75eeb5c8e
changeset:   72473:58a75eeb5c8e
branch:      3.2
parent:      72469:4951ffc178db
user:        Meador Inge <meadori at gmail.com>
date:        Tue Sep 27 20:45:30 2011 -0500
summary:
  Issue #13013: ctypes: Fix a reference leak in PyCArrayType_from_ctype.

Thanks to Suman Saha for finding the bug and providing a patch.

files:
  Misc/ACKS                 |  1 +
  Misc/NEWS                 |  3 +++
  Modules/_ctypes/_ctypes.c |  1 +
  3 files changed, 5 insertions(+), 0 deletions(-)


diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -774,6 +774,7 @@
 Mark Russell
 Nick Russo
 S?bastien Sabl?
+Suman Saha
 Hajime Saitou
 George Sakkis
 Rich Salz
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -82,6 +82,9 @@
 Extension Modules
 -----------------
 
+- Issue #13013: ctypes: Fix a reference leak in PyCArrayType_from_ctype.
+  Thanks to Suman Saha for finding the bug and providing a patch.
+
 - Issue #13022: Fix: _multiprocessing.recvfd() doesn't check that
   file descriptor was actually received.
 
diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c
--- a/Modules/_ctypes/_ctypes.c
+++ b/Modules/_ctypes/_ctypes.c
@@ -4474,6 +4474,7 @@
     if (!PyType_Check(itemtype)) {
         PyErr_SetString(PyExc_TypeError,
                         "Expected a type object");
+        Py_DECREF(key);
         return NULL;
     }
 #ifdef MS_WIN64

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 03:57:26 2011
From: python-checkins at python.org (meador.inge)
Date: Wed, 28 Sep 2011 03:57:26 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313013=3A_ctypes=3A_Fix_a_reference_leak_in_PyCArray?=
 =?utf8?q?Type=5Ffrom=5Fctype=2E?=
Message-ID: <E1R8jOo-0000Ma-81@dinsdale.python.org>

http://hg.python.org/cpython/rev/1726fa560112
changeset:   72474:1726fa560112
parent:      72471:84280fac98b9
parent:      72473:58a75eeb5c8e
user:        Meador Inge <meadori at gmail.com>
date:        Tue Sep 27 20:52:04 2011 -0500
summary:
  Issue #13013: ctypes: Fix a reference leak in PyCArrayType_from_ctype.

Thanks to Suman Saha for finding the bug and providing a patch.

files:
  Misc/ACKS                 |  1 +
  Misc/NEWS                 |  3 +++
  Modules/_ctypes/_ctypes.c |  1 +
  3 files changed, 5 insertions(+), 0 deletions(-)


diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -828,6 +828,7 @@
 Nick Russo
 Patrick Sabin
 S?bastien Sabl?
+Suman Saha
 Hajime Saitou
 George Sakkis
 Rich Salz
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1295,6 +1295,9 @@
 Extension Modules
 -----------------
 
+- Issue #13013: ctypes: Fix a reference leak in PyCArrayType_from_ctype.
+  Thanks to Suman Saha for finding the bug and providing a patch.
+
 - Issue #13022: Fix: _multiprocessing.recvfd() doesn't check that
   file descriptor was actually received.
 
diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c
--- a/Modules/_ctypes/_ctypes.c
+++ b/Modules/_ctypes/_ctypes.c
@@ -4474,6 +4474,7 @@
     if (!PyType_Check(itemtype)) {
         PyErr_SetString(PyExc_TypeError,
                         "Expected a type object");
+        Py_DECREF(key);
         return NULL;
     }
 #ifdef MS_WIN64

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Wed Sep 28 05:19:10 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Wed, 28 Sep 2011 05:19:10 +0200
Subject: [Python-checkins] Daily reference leaks (84280fac98b9): sum=0
Message-ID: <E1R8kfu-00008b-0Q@ap.vmr.nerim.net>

results for 84280fac98b9 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogFm5gyM', '-x']

From python-checkins at python.org  Wed Sep 28 08:35:59 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Wed, 28 Sep 2011 08:35:59 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Implement_PEP_393=2E?=
Message-ID: <E1R8nkN-0007Z5-Nx@dinsdale.python.org>

http://hg.python.org/cpython/rev/8beaa9a37387
changeset:   72475:8beaa9a37387
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Wed Sep 28 07:41:54 2011 +0200
summary:
  Implement PEP 393.

files:
  Doc/c-api/unicode.rst                  |     9 +
  Include/Python.h                       |     5 +
  Include/complexobject.h                |     5 +-
  Include/floatobject.h                  |     5 +-
  Include/longobject.h                   |     6 +-
  Include/pyerrors.h                     |     6 +
  Include/pyport.h                       |     3 +
  Include/unicodeobject.h                |   783 +-
  Lib/json/decoder.py                    |     3 +-
  Lib/test/json_tests/test_scanstring.py |    11 +-
  Lib/test/test_codeccallbacks.py        |     7 +-
  Lib/test/test_codecs.py                |     4 +
  Lib/test/test_peepholer.py             |     4 -
  Lib/test/test_re.py                    |     7 +
  Lib/test/test_sys.py                   |    38 +-
  Lib/test/test_unicode.py               |    41 +-
  Makefile.pre.in                        |     6 +-
  Misc/NEWS                              |     2 +
  Modules/_codecsmodule.c                |     8 +-
  Modules/_csv.c                         |     2 +-
  Modules/_ctypes/_ctypes.c              |     6 +-
  Modules/_ctypes/callproc.c             |     8 -
  Modules/_ctypes/cfield.c               |    64 +-
  Modules/_cursesmodule.c                |     7 +-
  Modules/_datetimemodule.c              |    13 +-
  Modules/_dbmmodule.c                   |    12 +-
  Modules/_elementtree.c                 |    31 +-
  Modules/_io/_iomodule.h                |     2 +-
  Modules/_io/stringio.c                 |    69 +-
  Modules/_io/textio.c                   |   352 +-
  Modules/_json.c                        |   252 +-
  Modules/_pickle.c                      |     4 +-
  Modules/_sqlite/connection.c           |    19 +-
  Modules/_sre.c                         |   382 +-
  Modules/_testcapimodule.c              |     2 +-
  Modules/_tkinter.c                     |    70 +-
  Modules/arraymodule.c                  |     8 +-
  Modules/md5module.c                    |    10 +-
  Modules/operator.c                     |    27 +-
  Modules/pyexpat.c                      |    11 +-
  Modules/sha1module.c                   |    10 +-
  Modules/sha256module.c                 |    10 +-
  Modules/sha512module.c                 |    10 +-
  Modules/sre.h                          |     4 +-
  Modules/syslogmodule.c                 |    14 +-
  Modules/unicodedata.c                  |    28 +-
  Modules/zipimport.c                    |   141 +-
  Objects/abstract.c                     |     4 +-
  Objects/bytearrayobject.c              |   147 +-
  Objects/bytesobject.c                  |   127 +-
  Objects/codeobject.c                   |    15 +-
  Objects/complexobject.c                |    19 +-
  Objects/dictobject.c                   |    20 +-
  Objects/exceptions.c                   |    26 +-
  Objects/fileobject.c                   |    17 +-
  Objects/floatobject.c                  |    19 +-
  Objects/longobject.c                   |    84 +-
  Objects/moduleobject.c                 |     9 +-
  Objects/object.c                       |    10 +-
  Objects/setobject.c                    |    40 +-
  Objects/stringlib/count.h              |     9 +-
  Objects/stringlib/eq.h                 |    23 +-
  Objects/stringlib/fastsearch.h         |     4 +-
  Objects/stringlib/find.h               |    31 +-
  Objects/stringlib/formatter.h          |  1516 --
  Objects/stringlib/localeutil.h         |    27 +-
  Objects/stringlib/partition.h          |    12 +-
  Objects/stringlib/split.h              |    26 +-
  Objects/stringlib/string_format.h      |  1385 --
  Objects/stringlib/stringdefs.h         |     2 +
  Objects/stringlib/ucs1lib.h            |    35 +
  Objects/stringlib/ucs2lib.h            |    34 +
  Objects/stringlib/ucs4lib.h            |    34 +
  Objects/stringlib/undef.h              |    10 +
  Objects/stringlib/unicode_format.h     |  1416 ++
  Objects/stringlib/unicodedefs.h        |     2 +
  Objects/typeobject.c                   |    18 +-
  Objects/unicodeobject.c                |  6112 ++++++++---
  Objects/uniops.h                       |    91 +
  PC/_subprocess.c                       |    61 +-
  PC/import_nt.c                         |     2 +-
  PC/msvcrtmodule.c                      |     8 +-
  PC/pyconfig.h                          |     4 -
  PC/winreg.c                            |     8 +-
  Parser/tokenizer.c                     |     6 +-
  Python/_warnings.c                     |    16 +-
  Python/ast.c                           |    61 +-
  Python/bltinmodule.c                   |    26 +-
  Python/ceval.c                         |    17 +-
  Python/codecs.c                        |    44 +-
  Python/compile.c                       |    89 +-
  Python/errors.c                        |     4 +-
  Python/formatter_unicode.c             |  1445 ++-
  Python/getargs.c                       |    46 +-
  Python/import.c                        |   347 +-
  Python/marshal.c                       |     4 +-
  Python/peephole.c                      |    18 -
  Python/symtable.c                      |     8 +-
  Python/traceback.c                     |    59 +-
  Tools/gdb/libpython.py                 |    27 +-
  configure                              |    65 +-
  configure.in                           |    46 +-
  pyconfig.h.in                          |     6 -
  103 files changed, 9492 insertions(+), 6770 deletions(-)


diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst
--- a/Doc/c-api/unicode.rst
+++ b/Doc/c-api/unicode.rst
@@ -1072,6 +1072,15 @@
    occurred and an exception has been set.
 
 
+.. c:function:: Py_ssize_t PyUnicode_FindChar(PyObject *str, Py_UCS4 ch, Py_ssize_t start, Py_ssize_t end, int direction)
+
+   Return the first position of the character *ch* in ``str[start:end]`` using
+   the given *direction* (*direction* == 1 means to do a forward search,
+   *direction* == -1 a backward search).  The return value is the index of the
+   first match; a value of ``-1`` indicates that no match was found, and ``-2``
+   indicates that an error occurred and an exception has been set.
+
+
 .. c:function:: Py_ssize_t PyUnicode_Count(PyObject *str, PyObject *substr, Py_ssize_t start, Py_ssize_t end)
 
    Return the number of non-overlapping occurrences of *substr* in
diff --git a/Include/Python.h b/Include/Python.h
--- a/Include/Python.h
+++ b/Include/Python.h
@@ -160,4 +160,9 @@
 #define PyDoc_STR(str) ""
 #endif
 
+#define PY_ARRAY_LENGTH(array) (sizeof(array) / sizeof((array)[0]))
+
+#define PY_MIN(x, y) (((x) > (y)) ? (y) : (x))
+#define PY_MAX(x, y) (((x) > (y)) ? (x) : (y))
+
 #endif /* !Py_PYTHON_H */
diff --git a/Include/complexobject.h b/Include/complexobject.h
--- a/Include/complexobject.h
+++ b/Include/complexobject.h
@@ -64,8 +64,9 @@
    (Advanced String Formatting). */
 #ifndef Py_LIMITED_API
 PyAPI_FUNC(PyObject *) _PyComplex_FormatAdvanced(PyObject *obj,
-                                                 Py_UNICODE *format_spec,
-                                                 Py_ssize_t format_spec_len);
+                                                 PyObject *format_spec,
+                                                 Py_ssize_t start,
+                                                 Py_ssize_t end);
 #endif
 
 #ifdef __cplusplus
diff --git a/Include/floatobject.h b/Include/floatobject.h
--- a/Include/floatobject.h
+++ b/Include/floatobject.h
@@ -113,8 +113,9 @@
 /* Format the object based on the format_spec, as defined in PEP 3101
    (Advanced String Formatting). */
 PyAPI_FUNC(PyObject *) _PyFloat_FormatAdvanced(PyObject *obj,
-					       Py_UNICODE *format_spec,
-					       Py_ssize_t format_spec_len);
+					       PyObject *format_spec,
+					       Py_ssize_t start,
+					       Py_ssize_t end);
 #endif /* Py_LIMITED_API */
 
 #ifdef __cplusplus
diff --git a/Include/longobject.h b/Include/longobject.h
--- a/Include/longobject.h
+++ b/Include/longobject.h
@@ -80,6 +80,7 @@
 PyAPI_FUNC(PyObject *) PyLong_FromString(char *, char **, int);
 #ifndef Py_LIMITED_API
 PyAPI_FUNC(PyObject *) PyLong_FromUnicode(Py_UNICODE*, Py_ssize_t, int);
+PyAPI_FUNC(PyObject *) PyLong_FromUnicodeObject(PyObject *u, int base);
 #endif
 
 #ifndef Py_LIMITED_API
@@ -155,8 +156,9 @@
 /* Format the object based on the format_spec, as defined in PEP 3101
    (Advanced String Formatting). */
 PyAPI_FUNC(PyObject *) _PyLong_FormatAdvanced(PyObject *obj,
-					      Py_UNICODE *format_spec,
-					      Py_ssize_t format_spec_len);
+					      PyObject *format_spec,
+					      Py_ssize_t start,
+					      Py_ssize_t end);
 #endif /* Py_LIMITED_API */
 
 /* These aren't really part of the long object, but they're handy. The
diff --git a/Include/pyerrors.h b/Include/pyerrors.h
--- a/Include/pyerrors.h
+++ b/Include/pyerrors.h
@@ -301,6 +301,12 @@
     Py_ssize_t end,
     const char *reason          /* UTF-8 encoded string */
     );
+PyAPI_FUNC(PyObject *) _PyUnicodeTranslateError_Create(
+    PyObject *object,
+    Py_ssize_t start,
+    Py_ssize_t end,
+    const char *reason          /* UTF-8 encoded string */
+    );
 #endif
 
 /* get the encoding attribute */
diff --git a/Include/pyport.h b/Include/pyport.h
--- a/Include/pyport.h
+++ b/Include/pyport.h
@@ -286,12 +286,15 @@
 /* fastest possible local call under MSVC */
 #define Py_LOCAL(type) static type __fastcall
 #define Py_LOCAL_INLINE(type) static __inline type __fastcall
+#define Py_LOCAL_CALLBACK(name) (__fastcall *name)
 #elif defined(USE_INLINE)
 #define Py_LOCAL(type) static type
 #define Py_LOCAL_INLINE(type) static inline type
+#define Py_LOCAL_CALLBACK(name) (*name)
 #else
 #define Py_LOCAL(type) static type
 #define Py_LOCAL_INLINE(type) static type
+#define Py_LOCAL_CALLBACK(name) (*name)
 #endif
 
 /* Py_MEMCPY can be used instead of memcpy in cases where the copied blocks
diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -64,16 +64,15 @@
 /* Python 3.x requires unicode */
 #define Py_USING_UNICODE
 
-/* FIXME: MvL's new implementation assumes that Py_UNICODE_SIZE is
-   properly set, but the default rules below doesn't set it.  I'll
-   sort this out some other day -- fredrik at pythonware.com */
-
-#ifndef Py_UNICODE_SIZE
-#error Must define Py_UNICODE_SIZE
+#ifndef SIZEOF_WCHAR_T
+#error Must define SIZEOF_WCHAR_T
 #endif
 
-/* Setting Py_UNICODE_WIDE enables UCS-4 storage.  Otherwise, Unicode
-   strings are stored as UCS-2 (with limited support for UTF-16) */
+#define Py_UNICODE_SIZE SIZEOF_WCHAR_T
+
+/* If wchar_t can be used for UCS-4 storage, set Py_UNICODE_WIDE.
+   Otherwise, Unicode strings are stored as UCS-2 (with limited support
+   for UTF-16) */
 
 #if Py_UNICODE_SIZE >= 4
 #define Py_UNICODE_WIDE
@@ -84,19 +83,14 @@
 /* #define HAVE_WCHAR_H */
 /* #define HAVE_USABLE_WCHAR_T */
 
-/* Defaults for various platforms */
-#ifndef PY_UNICODE_TYPE
+/* Py_UNICODE was the native Unicode storage format (code unit) used by
+   Python and represents a single Unicode element in the Unicode type.
+   With PEP 393, Py_UNICODE is deprected and replaced with a
+   typedef to wchar_t. */
 
-/* Windows has a usable wchar_t type (unless we're using UCS-4) */
-# if defined(MS_WIN32) && Py_UNICODE_SIZE == 2
-#  define HAVE_USABLE_WCHAR_T
-#  define PY_UNICODE_TYPE wchar_t
-# endif
-
-# if defined(Py_UNICODE_WIDE)
-#  define PY_UNICODE_TYPE Py_UCS4
-# endif
-
+#ifndef Py_LIMITED_API
+#define PY_UNICODE_TYPE wchar_t
+typedef wchar_t Py_UNICODE;
 #endif
 
 /* If the compiler provides a wchar_t type we try to support it
@@ -109,7 +103,7 @@
 # endif
 #endif
 
-#if defined(MS_WINDOWS) && defined(HAVE_USABLE_WCHAR_T)
+#if defined(MS_WINDOWS)
 #  define HAVE_MBCS
 #endif
 
@@ -121,200 +115,18 @@
 #  include <wchar.h>
 #endif
 
-/*
- * Use this typedef when you need to represent a UTF-16 surrogate pair
- * as single unsigned integer.
- */
+/* Py_UCS4 and Py_UCS2 are typdefs for the respecitve
+   unicode representations. */
 #if SIZEOF_INT >= 4
 typedef unsigned int Py_UCS4;
 #elif SIZEOF_LONG >= 4
 typedef unsigned long Py_UCS4;
+#else
+#error "Could not find a proper typedef for Py_UCS4"
 #endif
 
-/* Py_UNICODE is the native Unicode storage format (code unit) used by
-   Python and represents a single Unicode element in the Unicode
-   type. */
-
-#ifndef Py_LIMITED_API
-typedef PY_UNICODE_TYPE Py_UNICODE;
-#endif
-
-/* --- UCS-2/UCS-4 Name Mangling ------------------------------------------ */
-
-/* Unicode API names are mangled to assure that UCS-2 and UCS-4 builds
-   produce different external names and thus cause import errors in
-   case Python interpreters and extensions with mixed compiled in
-   Unicode width assumptions are combined. */
-
-#ifndef Py_UNICODE_WIDE
-
-# define PyUnicode_AsASCIIString PyUnicodeUCS2_AsASCIIString
-# define PyUnicode_AsCharmapString PyUnicodeUCS2_AsCharmapString
-# define PyUnicode_AsDecodedObject PyUnicodeUCS2_AsDecodedObject
-# define PyUnicode_AsDecodedUnicode PyUnicodeUCS2_AsDecodedUnicode
-# define PyUnicode_AsEncodedObject PyUnicodeUCS2_AsEncodedObject
-# define PyUnicode_AsEncodedString PyUnicodeUCS2_AsEncodedString
-# define PyUnicode_AsEncodedUnicode PyUnicodeUCS2_AsEncodedUnicode
-# define PyUnicode_AsLatin1String PyUnicodeUCS2_AsLatin1String
-# define PyUnicode_AsRawUnicodeEscapeString PyUnicodeUCS2_AsRawUnicodeEscapeString
-# define PyUnicode_AsUTF32String PyUnicodeUCS2_AsUTF32String
-# define PyUnicode_AsUTF16String PyUnicodeUCS2_AsUTF16String
-# define PyUnicode_AsUTF8String PyUnicodeUCS2_AsUTF8String
-# define PyUnicode_AsUnicode PyUnicodeUCS2_AsUnicode
-# define PyUnicode_AsUnicodeEscapeString PyUnicodeUCS2_AsUnicodeEscapeString
-# define PyUnicode_AsWideChar PyUnicodeUCS2_AsWideChar
-# define PyUnicode_AsWideCharString PyUnicodeUCS2_AsWideCharString
-# define PyUnicode_ClearFreeList PyUnicodeUCS2_ClearFreelist
-# define PyUnicode_Compare PyUnicodeUCS2_Compare
-# define PyUnicode_CompareWithASCIIString PyUnicodeUCS2_CompareWithASCIIString
-# define PyUnicode_Concat PyUnicodeUCS2_Concat
-# define PyUnicode_Append PyUnicodeUCS2_Append
-# define PyUnicode_AppendAndDel PyUnicodeUCS2_AppendAndDel
-# define PyUnicode_Contains PyUnicodeUCS2_Contains
-# define PyUnicode_Count PyUnicodeUCS2_Count
-# define PyUnicode_Decode PyUnicodeUCS2_Decode
-# define PyUnicode_DecodeASCII PyUnicodeUCS2_DecodeASCII
-# define PyUnicode_DecodeCharmap PyUnicodeUCS2_DecodeCharmap
-# define PyUnicode_DecodeLatin1 PyUnicodeUCS2_DecodeLatin1
-# define PyUnicode_DecodeFSDefault PyUnicodeUCS2_DecodeFSDefault
-# define PyUnicode_DecodeFSDefaultAndSize PyUnicodeUCS2_DecodeFSDefaultAndSize
-# define PyUnicode_DecodeRawUnicodeEscape PyUnicodeUCS2_DecodeRawUnicodeEscape
-# define PyUnicode_DecodeUTF32 PyUnicodeUCS2_DecodeUTF32
-# define PyUnicode_DecodeUTF32Stateful PyUnicodeUCS2_DecodeUTF32Stateful
-# define PyUnicode_DecodeUTF16 PyUnicodeUCS2_DecodeUTF16
-# define PyUnicode_DecodeUTF16Stateful PyUnicodeUCS2_DecodeUTF16Stateful
-# define PyUnicode_DecodeUTF8 PyUnicodeUCS2_DecodeUTF8
-# define PyUnicode_DecodeUTF8Stateful PyUnicodeUCS2_DecodeUTF8Stateful
-# define PyUnicode_DecodeUnicodeEscape PyUnicodeUCS2_DecodeUnicodeEscape
-# define PyUnicode_Encode PyUnicodeUCS2_Encode
-# define PyUnicode_EncodeASCII PyUnicodeUCS2_EncodeASCII
-# define PyUnicode_EncodeCharmap PyUnicodeUCS2_EncodeCharmap
-# define PyUnicode_EncodeDecimal PyUnicodeUCS2_EncodeDecimal
-# define PyUnicode_EncodeLatin1 PyUnicodeUCS2_EncodeLatin1
-# define PyUnicode_EncodeRawUnicodeEscape PyUnicodeUCS2_EncodeRawUnicodeEscape
-# define PyUnicode_EncodeUTF32 PyUnicodeUCS2_EncodeUTF32
-# define PyUnicode_EncodeUTF16 PyUnicodeUCS2_EncodeUTF16
-# define PyUnicode_EncodeUTF8 PyUnicodeUCS2_EncodeUTF8
-# define PyUnicode_EncodeUnicodeEscape PyUnicodeUCS2_EncodeUnicodeEscape
-# define PyUnicode_Find PyUnicodeUCS2_Find
-# define PyUnicode_Format PyUnicodeUCS2_Format
-# define PyUnicode_FromEncodedObject PyUnicodeUCS2_FromEncodedObject
-# define PyUnicode_FromFormat PyUnicodeUCS2_FromFormat
-# define PyUnicode_FromFormatV PyUnicodeUCS2_FromFormatV
-# define PyUnicode_FromObject PyUnicodeUCS2_FromObject
-# define PyUnicode_FromOrdinal PyUnicodeUCS2_FromOrdinal
-# define PyUnicode_FromString PyUnicodeUCS2_FromString
-# define PyUnicode_FromStringAndSize PyUnicodeUCS2_FromStringAndSize
-# define PyUnicode_FromUnicode PyUnicodeUCS2_FromUnicode
-# define PyUnicode_FromWideChar PyUnicodeUCS2_FromWideChar
-# define PyUnicode_FSConverter PyUnicodeUCS2_FSConverter
-# define PyUnicode_FSDecoder PyUnicodeUCS2_FSDecoder
-# define PyUnicode_GetDefaultEncoding PyUnicodeUCS2_GetDefaultEncoding
-# define PyUnicode_GetMax PyUnicodeUCS2_GetMax
-# define PyUnicode_GetSize PyUnicodeUCS2_GetSize
-# define PyUnicode_IsIdentifier PyUnicodeUCS2_IsIdentifier
-# define PyUnicode_Join PyUnicodeUCS2_Join
-# define PyUnicode_Partition PyUnicodeUCS2_Partition
-# define PyUnicode_RPartition PyUnicodeUCS2_RPartition
-# define PyUnicode_RSplit PyUnicodeUCS2_RSplit
-# define PyUnicode_Replace PyUnicodeUCS2_Replace
-# define PyUnicode_Resize PyUnicodeUCS2_Resize
-# define PyUnicode_RichCompare PyUnicodeUCS2_RichCompare
-# define PyUnicode_Split PyUnicodeUCS2_Split
-# define PyUnicode_Splitlines PyUnicodeUCS2_Splitlines
-# define PyUnicode_Tailmatch PyUnicodeUCS2_Tailmatch
-# define PyUnicode_Translate PyUnicodeUCS2_Translate
-# define PyUnicode_TranslateCharmap PyUnicodeUCS2_TranslateCharmap
-# define _PyUnicode_AsDefaultEncodedString _PyUnicodeUCS2_AsDefaultEncodedString
-# define _PyUnicode_Fini _PyUnicodeUCS2_Fini
-# define _PyUnicode_Init _PyUnicodeUCS2_Init
-# define PyUnicode_strdup PyUnicodeUCS2_strdup
-
-#else
-
-# define PyUnicode_AsASCIIString PyUnicodeUCS4_AsASCIIString
-# define PyUnicode_AsCharmapString PyUnicodeUCS4_AsCharmapString
-# define PyUnicode_AsDecodedObject PyUnicodeUCS4_AsDecodedObject
-# define PyUnicode_AsDecodedUnicode PyUnicodeUCS4_AsDecodedUnicode
-# define PyUnicode_AsEncodedObject PyUnicodeUCS4_AsEncodedObject
-# define PyUnicode_AsEncodedString PyUnicodeUCS4_AsEncodedString
-# define PyUnicode_AsEncodedUnicode PyUnicodeUCS4_AsEncodedUnicode
-# define PyUnicode_AsLatin1String PyUnicodeUCS4_AsLatin1String
-# define PyUnicode_AsRawUnicodeEscapeString PyUnicodeUCS4_AsRawUnicodeEscapeString
-# define PyUnicode_AsUTF32String PyUnicodeUCS4_AsUTF32String
-# define PyUnicode_AsUTF16String PyUnicodeUCS4_AsUTF16String
-# define PyUnicode_AsUTF8String PyUnicodeUCS4_AsUTF8String
-# define PyUnicode_AsUnicode PyUnicodeUCS4_AsUnicode
-# define PyUnicode_AsUnicodeEscapeString PyUnicodeUCS4_AsUnicodeEscapeString
-# define PyUnicode_AsWideChar PyUnicodeUCS4_AsWideChar
-# define PyUnicode_AsWideCharString PyUnicodeUCS4_AsWideCharString
-# define PyUnicode_ClearFreeList PyUnicodeUCS4_ClearFreelist
-# define PyUnicode_Compare PyUnicodeUCS4_Compare
-# define PyUnicode_CompareWithASCIIString PyUnicodeUCS4_CompareWithASCIIString
-# define PyUnicode_Concat PyUnicodeUCS4_Concat
-# define PyUnicode_Append PyUnicodeUCS4_Append
-# define PyUnicode_AppendAndDel PyUnicodeUCS4_AppendAndDel
-# define PyUnicode_Contains PyUnicodeUCS4_Contains
-# define PyUnicode_Count PyUnicodeUCS4_Count
-# define PyUnicode_Decode PyUnicodeUCS4_Decode
-# define PyUnicode_DecodeASCII PyUnicodeUCS4_DecodeASCII
-# define PyUnicode_DecodeCharmap PyUnicodeUCS4_DecodeCharmap
-# define PyUnicode_DecodeLatin1 PyUnicodeUCS4_DecodeLatin1
-# define PyUnicode_DecodeFSDefault PyUnicodeUCS4_DecodeFSDefault
-# define PyUnicode_DecodeFSDefaultAndSize PyUnicodeUCS4_DecodeFSDefaultAndSize
-# define PyUnicode_DecodeRawUnicodeEscape PyUnicodeUCS4_DecodeRawUnicodeEscape
-# define PyUnicode_DecodeUTF32 PyUnicodeUCS4_DecodeUTF32
-# define PyUnicode_DecodeUTF32Stateful PyUnicodeUCS4_DecodeUTF32Stateful
-# define PyUnicode_DecodeUTF16 PyUnicodeUCS4_DecodeUTF16
-# define PyUnicode_DecodeUTF16Stateful PyUnicodeUCS4_DecodeUTF16Stateful
-# define PyUnicode_DecodeUTF8 PyUnicodeUCS4_DecodeUTF8
-# define PyUnicode_DecodeUTF8Stateful PyUnicodeUCS4_DecodeUTF8Stateful
-# define PyUnicode_DecodeUnicodeEscape PyUnicodeUCS4_DecodeUnicodeEscape
-# define PyUnicode_Encode PyUnicodeUCS4_Encode
-# define PyUnicode_EncodeASCII PyUnicodeUCS4_EncodeASCII
-# define PyUnicode_EncodeCharmap PyUnicodeUCS4_EncodeCharmap
-# define PyUnicode_EncodeDecimal PyUnicodeUCS4_EncodeDecimal
-# define PyUnicode_EncodeLatin1 PyUnicodeUCS4_EncodeLatin1
-# define PyUnicode_EncodeRawUnicodeEscape PyUnicodeUCS4_EncodeRawUnicodeEscape
-# define PyUnicode_EncodeUTF32 PyUnicodeUCS4_EncodeUTF32
-# define PyUnicode_EncodeUTF16 PyUnicodeUCS4_EncodeUTF16
-# define PyUnicode_EncodeUTF8 PyUnicodeUCS4_EncodeUTF8
-# define PyUnicode_EncodeUnicodeEscape PyUnicodeUCS4_EncodeUnicodeEscape
-# define PyUnicode_Find PyUnicodeUCS4_Find
-# define PyUnicode_Format PyUnicodeUCS4_Format
-# define PyUnicode_FromEncodedObject PyUnicodeUCS4_FromEncodedObject
-# define PyUnicode_FromFormat PyUnicodeUCS4_FromFormat
-# define PyUnicode_FromFormatV PyUnicodeUCS4_FromFormatV
-# define PyUnicode_FromObject PyUnicodeUCS4_FromObject
-# define PyUnicode_FromOrdinal PyUnicodeUCS4_FromOrdinal
-# define PyUnicode_FromString PyUnicodeUCS4_FromString
-# define PyUnicode_FromStringAndSize PyUnicodeUCS4_FromStringAndSize
-# define PyUnicode_FromUnicode PyUnicodeUCS4_FromUnicode
-# define PyUnicode_FromWideChar PyUnicodeUCS4_FromWideChar
-# define PyUnicode_FSConverter PyUnicodeUCS4_FSConverter
-# define PyUnicode_FSDecoder PyUnicodeUCS4_FSDecoder
-# define PyUnicode_GetDefaultEncoding PyUnicodeUCS4_GetDefaultEncoding
-# define PyUnicode_GetMax PyUnicodeUCS4_GetMax
-# define PyUnicode_GetSize PyUnicodeUCS4_GetSize
-# define PyUnicode_IsIdentifier PyUnicodeUCS4_IsIdentifier
-# define PyUnicode_Join PyUnicodeUCS4_Join
-# define PyUnicode_Partition PyUnicodeUCS4_Partition
-# define PyUnicode_RPartition PyUnicodeUCS4_RPartition
-# define PyUnicode_RSplit PyUnicodeUCS4_RSplit
-# define PyUnicode_Replace PyUnicodeUCS4_Replace
-# define PyUnicode_Resize PyUnicodeUCS4_Resize
-# define PyUnicode_RichCompare PyUnicodeUCS4_RichCompare
-# define PyUnicode_Split PyUnicodeUCS4_Split
-# define PyUnicode_Splitlines PyUnicodeUCS4_Splitlines
-# define PyUnicode_Tailmatch PyUnicodeUCS4_Tailmatch
-# define PyUnicode_Translate PyUnicodeUCS4_Translate
-# define PyUnicode_TranslateCharmap PyUnicodeUCS4_TranslateCharmap
-# define _PyUnicode_AsDefaultEncodedString _PyUnicodeUCS4_AsDefaultEncodedString
-# define _PyUnicode_Fini _PyUnicodeUCS4_Fini
-# define _PyUnicode_Init _PyUnicodeUCS4_Init
-# define PyUnicode_strdup PyUnicodeUCS4_strdup
-
-#endif
+typedef unsigned short Py_UCS2;
+typedef unsigned char Py_UCS1;
 
 /* --- Internal Unicode Operations ---------------------------------------- */
 
@@ -354,7 +166,7 @@
     Py_UNICODE_ISDIGIT(ch) || \
     Py_UNICODE_ISNUMERIC(ch))
 
-#define Py_UNICODE_COPY(target, source, length)                         \
+#define Py_UNICODE_COPY(target, source, length) \
     Py_MEMCPY((target), (source), (length)*sizeof(Py_UNICODE))
 
 #define Py_UNICODE_FILL(target, value, length) \
@@ -375,9 +187,10 @@
    valid, and the substring must not be empty. */
 
 #define Py_UNICODE_MATCH(string, offset, substring) \
-    ((*((string)->str + (offset)) == *((substring)->str)) && \
-    ((*((string)->str + (offset) + (substring)->length-1) == *((substring)->str + (substring)->length-1))) && \
-     !memcmp((string)->str + (offset), (substring)->str, (substring)->length*sizeof(Py_UNICODE)))
+    ((*((string)->wstr + (offset)) == *((substring)->wstr)) && \
+     ((*((string)->wstr + (offset) + (substring)->wstr_length-1) == *((substring)->wstr + (substring)->wstr_length-1))) && \
+     !memcmp((string)->wstr + (offset), (substring)->wstr, (substring)->wstr_length*sizeof(Py_UNICODE)))
+
 #endif /* Py_LIMITED_API */
 
 #ifdef __cplusplus
@@ -387,41 +200,303 @@
 /* --- Unicode Type ------------------------------------------------------- */
 
 #ifndef Py_LIMITED_API
+
+/* ASCII-only strings created through PyUnicode_New use the PyASCIIObject
+   structure. state.ascii and state.compact are set, and the data
+   immediately follow the structure. utf8_length and wstr_length can be found
+   in the length field; the utf8 pointer is equal to the data pointer. */
 typedef struct {
     PyObject_HEAD
-    Py_ssize_t length;          /* Length of raw Unicode data in buffer */
-    Py_UNICODE *str;            /* Raw Unicode buffer */
+    Py_ssize_t length;          /* Number of code points in the string */
     Py_hash_t hash;             /* Hash value; -1 if not set */
-    int state;                  /* != 0 if interned. In this case the two
-                                 * references from the dictionary to this object
-                                 * are *not* counted in ob_refcnt. */
-    PyObject *defenc;           /* (Default) Encoded version as Python
-                                   string, or NULL; this is used for
-                                   implementing the buffer protocol */
+    struct {
+        /*
+           SSTATE_NOT_INTERNED (0)
+           SSTATE_INTERNED_MORTAL (1)
+           SSTATE_INTERNED_IMMORTAL (2)
+
+           If interned != SSTATE_NOT_INTERNED, the two references from the
+           dictionary to this object are *not* counted in ob_refcnt.
+         */
+        unsigned int interned:2;
+        /* Character size:
+
+           PyUnicode_WCHAR_KIND (0): wchar_t*
+           PyUnicode_1BYTE_KIND (1): Py_UCS1*
+           PyUnicode_2BYTE_KIND (2): Py_UCS2*
+           PyUnicode_4BYTE_KIND (3): Py_UCS4*
+         */
+        unsigned int kind:2;
+        /* Compact is with respect to the allocation scheme. Compact unicode
+           objects only require one memory block while non-compact objects use
+           one block for the PyUnicodeObject struct and another for its data
+           buffer. */
+        unsigned int compact:1;
+        /* Compact objects which are ASCII-only also have the state.compact
+           flag set, and use the PyASCIIObject struct. */
+        unsigned int ascii:1;
+        /* The ready flag indicates whether the object layout is initialized
+           completely. This means that this is either a compact object, or
+           the data pointer is filled out. The bit is redundant, and helps
+           to minimize the test in PyUnicode_IS_READY(). */
+        unsigned int ready:1;
+    } state;
+    wchar_t *wstr;              /* wchar_t representation (null-terminated) */
+} PyASCIIObject;
+
+/* Non-ASCII strings allocated through PyUnicode_New use the
+   PyCompactUnicodeOject structure. state.compact is set, and the data
+   immediately follow the structure. */
+typedef struct {
+    PyASCIIObject _base;
+    Py_ssize_t utf8_length;     /* Number of bytes in utf8, excluding the
+                                 * terminating \0. */
+    char *utf8;                 /* UTF-8 representation (null-terminated) */
+    Py_ssize_t wstr_length;     /* Number of code points in wstr, possible
+                                 * surrogates count as two code points. */
+} PyCompactUnicodeObject;
+
+/* Strings allocated through PyUnicode_FromUnicode(NULL, len) use the
+   PyUnicodeObject structure. The actual string data is initially in the wstr
+   block, and copied into the data block using PyUnicode_Ready. */
+typedef struct {
+    PyCompactUnicodeObject _base;
+    union {
+        void *any;
+        Py_UCS1 *latin1;
+        Py_UCS2 *ucs2;
+        Py_UCS4 *ucs4;
+    } data;                     /* Canonical, smallest-form Unicode buffer */
 } PyUnicodeObject;
 #endif
 
 PyAPI_DATA(PyTypeObject) PyUnicode_Type;
 PyAPI_DATA(PyTypeObject) PyUnicodeIter_Type;
 
-#define SSTATE_NOT_INTERNED 0
-#define SSTATE_INTERNED_MORTAL 1
-#define SSTATE_INTERNED_IMMORTAL 2
-
 #define PyUnicode_Check(op) \
                  PyType_FastSubclass(Py_TYPE(op), Py_TPFLAGS_UNICODE_SUBCLASS)
 #define PyUnicode_CheckExact(op) (Py_TYPE(op) == &PyUnicode_Type)
 
 /* Fast access macros */
 #ifndef Py_LIMITED_API
+
+#define PyUnicode_WSTR_LENGTH(op) \
+    (((PyASCIIObject*)op)->state.ascii ?    \
+     ((PyASCIIObject*)op)->length :                    \
+     ((PyCompactUnicodeObject*)op)->wstr_length)
+
+/* Returns the deprecated Py_UNICODE representation's size in code units
+   (this includes surrogate pairs as 2 units).
+   If the Py_UNICODE representation is not available, it will be computed
+   on request.  Use PyUnicode_GET_LENGTH() for the length in code points. */
+
 #define PyUnicode_GET_SIZE(op) \
-    (assert(PyUnicode_Check(op)),(((PyUnicodeObject *)(op))->length))
+    (assert(PyUnicode_Check(op)), \
+     (((PyASCIIObject *)(op))->wstr) ? \
+        PyUnicode_WSTR_LENGTH(op) :                   \
+        ((void)PyUnicode_AsUnicode((PyObject *)(op)), \
+         PyUnicode_WSTR_LENGTH(op)))
+
 #define PyUnicode_GET_DATA_SIZE(op) \
-    (assert(PyUnicode_Check(op)),(((PyUnicodeObject *)(op))->length * sizeof(Py_UNICODE)))
+    (PyUnicode_GET_SIZE(op) * Py_UNICODE_SIZE)
+
+/* Alias for PyUnicode_AsUnicode().  This will create a wchar_t/Py_UNICODE
+   representation on demand.  Using this macro is very inefficient now,
+   try to port your code to use the new PyUnicode_*BYTE_DATA() macros or
+   use PyUnicode_WRITE() and PyUnicode_READ(). */
+
 #define PyUnicode_AS_UNICODE(op) \
-    (assert(PyUnicode_Check(op)),(((PyUnicodeObject *)(op))->str))
+    (assert(PyUnicode_Check(op)), \
+     (((PyASCIIObject *)(op))->wstr) ? (((PyASCIIObject *)(op))->wstr) : \
+      PyUnicode_AsUnicode((PyObject *)(op)))
+
 #define PyUnicode_AS_DATA(op) \
-    (assert(PyUnicode_Check(op)),((const char *)((PyUnicodeObject *)(op))->str))
+    ((const char *)(PyUnicode_AS_UNICODE(op)))
+
+
+/* --- Flexible String Representaion Helper Macros (PEP 393) -------------- */
+
+/* Values for PyUnicodeObject.state: */
+
+/* Interning state. */
+#define SSTATE_NOT_INTERNED 0
+#define SSTATE_INTERNED_MORTAL 1
+#define SSTATE_INTERNED_IMMORTAL 2
+
+#define PyUnicode_IS_COMPACT_ASCII(op) (((PyASCIIObject*)op)->state.ascii)
+
+/* String contains only wstr byte characters.  This is only possible
+   when the string was created with a legacy API and PyUnicode_Ready()
+   has not been called yet.  */
+#define PyUnicode_WCHAR_KIND 0
+
+/* Return values of the PyUnicode_KIND() macro: */
+
+#define PyUnicode_1BYTE_KIND 1
+#define PyUnicode_2BYTE_KIND 2
+#define PyUnicode_4BYTE_KIND 3
+
+
+/* Return the number of bytes the string uses to represent single characters,
+   this can be 1, 2 or 4. */
+#define PyUnicode_CHARACTER_SIZE(op) \
+    (1 << (PyUnicode_KIND(op) - 1))
+
+/* Return pointers to the canonical representation casted as unsigned char,
+   Py_UCS2, or Py_UCS4 for direct character access.
+   No checks are performed, use PyUnicode_CHARACTER_SIZE or
+   PyUnicode_KIND() before to ensure these will work correctly. */
+
+#define PyUnicode_1BYTE_DATA(op) ((Py_UCS1*)PyUnicode_DATA(op))
+#define PyUnicode_2BYTE_DATA(op) ((Py_UCS2*)PyUnicode_DATA(op))
+#define PyUnicode_4BYTE_DATA(op) ((Py_UCS4*)PyUnicode_DATA(op))
+
+/* Return true if the string is compact or 0 if not.
+   No type checks or Ready calls are performed. */
+#define PyUnicode_IS_COMPACT(op) \
+    (((PyASCIIObject*)(op))->state.compact)
+
+/* Return one of the PyUnicode_*_KIND values defined above. */ 
+#define PyUnicode_KIND(op) \
+    (assert(PyUnicode_Check(op)), \
+     assert(PyUnicode_IS_READY(op)),            \
+     ((PyASCIIObject *)(op))->state.kind)
+
+/* Return a void pointer to the raw unicode buffer. */ 
+#define _PyUnicode_COMPACT_DATA(op)                     \
+    (PyUnicode_IS_COMPACT_ASCII(op) ?                   \
+     ((void*)((PyASCIIObject*)(op) + 1)) :              \
+     ((void*)((PyCompactUnicodeObject*)(op) + 1)))
+
+#define _PyUnicode_NONCOMPACT_DATA(op)                  \
+    (assert(((PyUnicodeObject*)(op))->data.any),        \
+     ((((PyUnicodeObject *)(op))->data.any)))
+
+#define PyUnicode_DATA(op) \
+    (assert(PyUnicode_Check(op)), \
+     PyUnicode_IS_COMPACT(op) ? _PyUnicode_COMPACT_DATA(op) :   \
+     _PyUnicode_NONCOMPACT_DATA(op))
+
+#define _PyUnicode_UTF8(op)                     \
+    (PyUnicode_IS_COMPACT_ASCII(op) ?           \
+     ((char*)((PyASCIIObject*)(op) + 1)) :      \
+     ((PyCompactUnicodeObject*)(op))->utf8)
+
+#define _PyUnicode_UTF8_LENGTH(op)                      \
+    (PyUnicode_IS_COMPACT_ASCII(op) ?                   \
+     ((PyASCIIObject*)(op))->length :                   \
+     ((PyCompactUnicodeObject*)(op))->utf8_length)
+
+/* Compute (index * char_size) where char_size is 2 ** (kind - 1).
+
+   The index is a character index, the result is a size in bytes. */
+#define PyUnicode_KIND_SIZE(kind, index) ((index) << ((kind) - 1))
+
+/* In the access macros below, "kind" may be evaluated more than once.
+   All other macro parameters are evaluated exactly once, so it is safe
+   to put side effects into them (such as increasing the index). */
+
+/* Write into the canonical representation, this macro does not do any sanity
+   checks and is intended for usage in loops.  The caller should cache the
+   kind and data pointers optained form other macro calls.
+   index is the index in the string (starts at 0) and value is the new
+   code point value which shoule be written to that location. */
+#define PyUnicode_WRITE(kind, data, index, value) \
+    do { \
+        switch ((kind)) { \
+        case PyUnicode_1BYTE_KIND: { \
+            ((Py_UCS1 *)(data))[(index)] = (Py_UCS1)(value); \
+            break; \
+        } \
+        case PyUnicode_2BYTE_KIND: { \
+            ((Py_UCS2 *)(data))[(index)] = (Py_UCS2)(value); \
+            break; \
+        } \
+        default: { \
+            assert((kind) == PyUnicode_4BYTE_KIND); \
+            ((Py_UCS4 *)(data))[(index)] = (Py_UCS4)(value); \
+        } \
+        } \
+    } while (0)
+
+/* Read a code point form the string's canonical representation.  No checks
+   or ready calls are performed. */
+#define PyUnicode_READ(kind, data, index) \
+    ((Py_UCS4) \
+    ((kind) == PyUnicode_1BYTE_KIND ? \
+        ((const unsigned char *)(data))[(index)] : \
+        ((kind) == PyUnicode_2BYTE_KIND ? \
+            ((const Py_UCS2 *)(data))[(index)] : \
+            ((const Py_UCS4 *)(data))[(index)] \
+        ) \
+    ))
+
+/* PyUnicode_READ_CHAR() is less efficient than PyUnicode_READ() because it
+   calls PyUnicode_KIND() and might call it twice.  For single reads, use
+   PyUnicode_READ_CHAR, for multiple consecutive reads callers should
+   cache kind and use PyUnicode_READ instead. */
+#define PyUnicode_READ_CHAR(unicode, index) \
+    ((Py_UCS4) \
+    (PyUnicode_KIND((unicode)) == PyUnicode_1BYTE_KIND ? \
+        ((const unsigned char *)(PyUnicode_DATA((unicode))))[(index)] : \
+        (PyUnicode_KIND((unicode)) == PyUnicode_2BYTE_KIND ? \
+            ((const Py_UCS2 *)(PyUnicode_DATA((unicode))))[(index)] : \
+            ((const Py_UCS4 *)(PyUnicode_DATA((unicode))))[(index)] \
+        ) \
+    ))
+
+/* Returns the length of the unicode string. The caller has to make sure that
+   the string has it's canonical representation set before calling
+   this macro.  Call PyUnicode_(FAST_)Ready to ensure that. */
+#define PyUnicode_GET_LENGTH(op)                \
+    (assert(PyUnicode_Check(op)),               \
+     assert(PyUnicode_IS_READY(op)),            \
+     ((PyASCIIObject *)(op))->length)
+
+
+/* Fast check to determine whether an object is ready. Equivalent to
+   PyUnicode_IS_COMPACT(op) || ((PyUnicodeObject*)(op))->data.any) */
+
+#define PyUnicode_IS_READY(op) (((PyASCIIObject*)op)->state.ready)
+
+/* PyUnicode_READY() does less work than PyUnicode_Ready() in the best
+   case.  If the canonical representation is not yet set, it will still call
+   PyUnicode_Ready().
+   Returns 0 on success and -1 on errors. */
+#define PyUnicode_READY(op)                        \
+    (assert(PyUnicode_Check(op)),                       \
+     (PyUnicode_IS_READY(op) ?                          \
+      0 : _PyUnicode_Ready((PyUnicodeObject *)(op))))
+
+/* Generic helper macro to convert characters of different types.
+   from_type and to_type have to be valid type names, begin and end
+   are pointers to the source characters which should be of type
+   "from_type *".  to is a pointer of type "to_type *" and points to the
+   buffer where the result characters are written to. */
+#define PyUnicode_CONVERT_BYTES(from_type, to_type, begin, end, to) \
+    do { \
+        const from_type *iter_; to_type *to_; \
+        for (iter_ = (begin), to_ = (to_type *)(to); \
+             iter_ < (end); \
+             ++iter_, ++to_) { \
+            *to_ = (to_type)*iter_; \
+        } \
+    } while (0)
+
+/* Return a maximum character value which is suitable for creating another
+   string based on op.  This is always an approximation but more efficient
+   than interating over the string. */
+#define PyUnicode_MAX_CHAR_VALUE(op) \
+    (assert(PyUnicode_IS_READY(op)),                                    \
+     (PyUnicode_IS_COMPACT_ASCII(op) ? 0x7f:                            \
+      (PyUnicode_KIND(op) == PyUnicode_1BYTE_KIND ?                     \
+       (PyUnicode_DATA(op) == (((PyCompactUnicodeObject *)(op))->utf8) ? \
+        (0x7fU) : (0xffU)                                                 \
+           ) :                                                          \
+       (PyUnicode_KIND(op) == PyUnicode_2BYTE_KIND ?                    \
+        (0xffffU) : (0x10ffffU)                                           \
+           ))))
+
 #endif
 
 /* --- Constants ---------------------------------------------------------- */
@@ -437,6 +512,52 @@
 
 /* --- Plain Py_UNICODE --------------------------------------------------- */
 
+/* With PEP 393, this is the recommended way to allocate a new unicode object.
+   This function will allocate the object and its buffer in a single memory
+   block.  Objects created using this function are not resizable. */
+#ifndef Py_LIMITED_API
+PyAPI_FUNC(PyObject*) PyUnicode_New(
+    Py_ssize_t size,            /* Number of code points in the new string */
+    Py_UCS4 maxchar             /* maximum code point value in the string */
+    );
+#endif
+
+/* Initializes the canonical string representation from a the deprected
+   wstr/Py_UNICODE representation.  This function is used to convert
+   unicode objects which were created using the old API to the new flexible 
+   format introduced with PEP 393.  The PyUnicode_READY() macro can be
+   more efficient if the string is already ready. */
+#ifndef Py_LIMITED_API
+PyAPI_FUNC(int) _PyUnicode_Ready(
+    PyUnicodeObject *unicode    /* Unicode object */
+    );
+#endif
+
+/* Copy character from one unicode object into another, this function performs
+   character conversion when nessesary and falls back to memcpy if possible.
+   Return -1 and raise an exception on error, return 0 on success. */
+#ifndef Py_LIMITED_API
+PyAPI_FUNC(int) PyUnicode_CopyCharacters(
+    PyObject *to,
+    Py_ssize_t to_start,
+    PyObject *from,
+    Py_ssize_t from_start,
+    Py_ssize_t how_many
+    );
+#endif
+
+/* Find the maximum code point and count the number of surrogate pairs so a
+   correct string length can be computed before converting a string to UCS4.
+   This function counts single surrogates as a character and not as a pair. */
+#ifndef Py_LIMITED_API
+PyAPI_FUNC(int) _PyUnicode_FindMaxCharAndNumSurrogatePairs(
+    const wchar_t *begin,
+    const wchar_t *end,
+    Py_UCS4 *maxchar,
+    Py_ssize_t *num_surrogates
+    );
+#endif
+
 /* Create a Unicode Object from the Py_UNICODE buffer u of the given
    size.
 
@@ -461,13 +582,43 @@
     );
 
 /* Similar to PyUnicode_FromUnicode(), but u points to null-terminated
-   UTF-8 encoded bytes */
+   UTF-8 encoded bytes.  The size is determined with strlen(). */
 PyAPI_FUNC(PyObject*) PyUnicode_FromString(
     const char *u              /* UTF-8 encoded string */
     );
 
+#ifndef Py_LIMITED_API
+PyAPI_FUNC(PyObject*) PyUnicode_FromKindAndData(
+    int kind,
+    const void *buffer,
+    Py_ssize_t size);
+#endif
+
+PyAPI_FUNC(PyObject*) PyUnicode_Substring(
+    PyObject *str,
+    Py_ssize_t start,
+    Py_ssize_t end);
+
+/* Copy the string into a UCS4 buffer including the null character is copy_null
+   is set. Return NULL and raise an exception on error. Raise a ValueError if
+   the buffer is smaller than the string. Return buffer on success.
+
+   buflen is the length of the buffer in (Py_UCS4) characters. */
+PyAPI_FUNC(Py_UCS4*) PyUnicode_AsUCS4(
+    PyObject *unicode,
+    Py_UCS4* buffer,
+    Py_ssize_t buflen,
+    int copy_null);
+
+/* Copy the string into a UCS4 buffer. A new buffer is allocated using
+ * PyMem_Malloc; if this fails, NULL is returned with a memory error
+   exception set. */
+PyAPI_FUNC(Py_UCS4*) PyUnicode_AsUCS4Copy(PyObject *unicode);
+
 /* Return a read-only pointer to the Unicode object's internal
-   Py_UNICODE buffer. */
+   Py_UNICODE buffer.
+   If the wchar_t/Py_UNICODE representation is not yet available, this
+   function will calculate it. */
 
 #ifndef Py_LIMITED_API
 PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
@@ -475,12 +626,47 @@
     );
 #endif
 
+/* Return a read-only pointer to the Unicode object's internal
+   Py_UNICODE buffer and save the length at size.
+   If the wchar_t/Py_UNICODE representation is not yet available, this
+   function will calculate it. */
+
+#ifndef Py_LIMITED_API
+PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicodeAndSize(
+    PyObject *unicode,          /* Unicode object */
+    Py_ssize_t *size            /* location where to save the length */
+    );
+#endif
+
 /* Get the length of the Unicode object. */
 
+PyAPI_FUNC(Py_ssize_t) PyUnicode_GetLength(
+    PyObject *unicode
+);
+
+/* Get the number of Py_UNICODE units in the 
+   string representation. */
+
 PyAPI_FUNC(Py_ssize_t) PyUnicode_GetSize(
     PyObject *unicode           /* Unicode object */
     );
 
+/* Read a character from the string. */
+
+PyAPI_FUNC(Py_UCS4) PyUnicode_ReadChar(
+    PyObject *unicode,
+    Py_ssize_t index
+    );
+
+/* Write a character to the string. The string must have been created through
+   PyUnicode_New, must not be shared, and must not have been hashed yet. */
+
+PyAPI_FUNC(int) PyUnicode_WriteChar(
+    PyObject *unicode,
+    Py_ssize_t index,
+    Py_UCS4 character
+    );
+
 #ifndef Py_LIMITED_API
 /* Get the maximum ordinal for a Unicode character. */
 PyAPI_FUNC(Py_UNICODE) PyUnicode_GetMax(void);
@@ -558,8 +744,9 @@
 /* Format the object based on the format_spec, as defined in PEP 3101
    (Advanced String Formatting). */
 PyAPI_FUNC(PyObject *) _PyUnicode_FormatAdvanced(PyObject *obj,
-                                                 Py_UNICODE *format_spec,
-                                                 Py_ssize_t format_spec_len);
+                                                 PyObject *format_spec,
+                                                 Py_ssize_t start,
+                                                 Py_ssize_t end);
 #endif
 
 PyAPI_FUNC(void) PyUnicode_InternInPlace(PyObject **);
@@ -572,7 +759,8 @@
 #endif
 
 /* Use only if you know it's a string */
-#define PyUnicode_CHECK_INTERNED(op) (((PyUnicodeObject *)(op))->state)
+#define PyUnicode_CHECK_INTERNED(op) \
+    (((PyASCIIObject *)(op))->state.interned)
 
 /* --- wchar_t support for platforms which support it --------------------- */
 
@@ -619,6 +807,8 @@
     Py_ssize_t *size            /* number of characters of the result */
     );
 
+PyAPI_FUNC(void*) _PyUnicode_AsKind(PyObject *s, unsigned int kind);
+
 #endif
 
 /* --- Unicode ordinals --------------------------------------------------- */
@@ -664,49 +854,42 @@
 
 /* --- Manage the default encoding ---------------------------------------- */
 
-/* Return a Python string holding the default encoded value of the
-   Unicode object.
-
-   Same as PyUnicode_AsUTF8String() except
-   the resulting string is cached in the Unicode object for subsequent
-   usage by this function. The cached version is needed to implement
-   the character buffer interface and will live (at least) as long as
-   the Unicode object itself.
-
-   The refcount of the string is *not* incremented.
-
-   *** Exported for internal use by the interpreter only !!! ***
-
-*/
-
-#ifndef Py_LIMITED_API
-PyAPI_FUNC(PyObject *) _PyUnicode_AsDefaultEncodedString(
-    PyObject *unicode);
-#endif
-
 /* Returns a pointer to the default encoding (UTF-8) of the
    Unicode object unicode and the size of the encoded representation
    in bytes stored in *size.
 
    In case of an error, no *size is set.
 
+   This funcation caches the UTF-8 encoded string in the unicodeobject 
+   and subsequent calls will return the same string.  The memory is relased
+   when the unicodeobject is deallocated.
+
+   _PyUnicode_AsStringAndSize is a #define for PyUnicode_AsUTF8AndSize to
+   support the previous internal function with the same behaviour.
+
    *** This API is for interpreter INTERNAL USE ONLY and will likely
    *** be removed or changed in the future.
 
    *** If you need to access the Unicode object as UTF-8 bytes string,
    *** please use PyUnicode_AsUTF8String() instead.
-
 */
 
 #ifndef Py_LIMITED_API
-PyAPI_FUNC(char *) _PyUnicode_AsStringAndSize(
+PyAPI_FUNC(char *) PyUnicode_AsUTF8AndSize(
     PyObject *unicode,
     Py_ssize_t *size);
+#define _PyUnicode_AsStringAndSize PyUnicode_AsUTF8AndSize
 #endif
 
 /* Returns a pointer to the default encoding (UTF-8) of the
    Unicode object unicode.
 
+   Like PyUnicode_AsUTF8AndSize(), this also caches the UTF-8 representation
+   in the unicodeobject.
+
+   _PyUnicode_AsString is a #define for PyUnicode_AsUTF8 to
+   support the previous internal function with the same behaviour.
+
    Use of this API is DEPRECATED since no size information can be
    extracted from the returned data.
 
@@ -719,7 +902,8 @@
 */
 
 #ifndef Py_LIMITED_API
-PyAPI_FUNC(char *) _PyUnicode_AsString(PyObject *unicode);
+PyAPI_FUNC(char *) PyUnicode_AsUTF8(PyObject *unicode);
+#define _PyUnicode_AsString PyUnicode_AsUTF8
 #endif
 
 /* Returns "utf-8".  */
@@ -846,6 +1030,10 @@
     );
 
 #ifndef Py_LIMITED_API
+PyAPI_FUNC(PyObject*) _PyUnicode_AsUTF8String(
+    PyObject *unicode,
+    const char *errors);
+
 PyAPI_FUNC(PyObject*) PyUnicode_EncodeUTF8(
     const Py_UNICODE *data,     /* Unicode char buffer */
     Py_ssize_t length,          /* number of Py_UNICODE chars to encode */
@@ -1076,6 +1264,10 @@
     );
 
 #ifndef Py_LIMITED_API
+PyAPI_FUNC(PyObject*) _PyUnicode_AsLatin1String(
+    PyObject* unicode,
+    const char* errors);
+
 PyAPI_FUNC(PyObject*) PyUnicode_EncodeLatin1(
     const Py_UNICODE *data,     /* Unicode char buffer */
     Py_ssize_t length,          /* Number of Py_UNICODE chars to encode */
@@ -1100,6 +1292,10 @@
     );
 
 #ifndef Py_LIMITED_API
+PyAPI_FUNC(PyObject*) _PyUnicode_AsASCIIString(
+    PyObject* unicode,
+    const char* errors);
+
 PyAPI_FUNC(PyObject*) PyUnicode_EncodeASCII(
     const Py_UNICODE *data,     /* Unicode char buffer */
     Py_ssize_t length,          /* Number of Py_UNICODE chars to encode */
@@ -1252,6 +1448,17 @@
     );
 #endif
 
+/* Similar to PyUnicode_TransformDecimalToASCII(), but takes a PyUnicodeObject
+   as argument instead of a raw buffer and length.  This function additionally
+   transforms spaces to ASCII because this is what the callers in longobject,
+   floatobject, and complexobject did anyways. */
+
+#ifndef Py_LIMITED_API
+PyAPI_FUNC(PyObject*) _PyUnicode_TransformDecimalAndSpaceToASCII(
+    PyObject *unicode           /* Unicode object */
+    );
+#endif
+
 /* --- File system encoding ---------------------------------------------- */
 
 /* ParseTuple converter: encode str objects to bytes using
@@ -1439,6 +1646,15 @@
     int direction               /* Find direction: +1 forward, -1 backward */
     );
 
+/* Like PyUnicode_Find, but search for single character only. */
+PyAPI_FUNC(Py_ssize_t) PyUnicode_FindChar(
+    PyObject *str,
+    Py_UCS4 ch,
+    Py_ssize_t start,
+    Py_ssize_t end,
+    int direction
+    );
+
 /* Count the number of occurrences of substr in str[start:end]. */
 
 PyAPI_FUNC(Py_ssize_t) PyUnicode_Count(
@@ -1542,13 +1758,15 @@
    into the string pointed to by buffer.  For the argument descriptions,
    see Objects/stringlib/localeutil.h */
 #ifndef Py_LIMITED_API
-PyAPI_FUNC(Py_ssize_t) _PyUnicode_InsertThousandsGrouping(Py_UNICODE *buffer,
-                                                   Py_ssize_t n_buffer,
-                                                   Py_UNICODE *digits,
-                                                   Py_ssize_t n_digits,
-                                                   Py_ssize_t min_width,
-                                                   const char *grouping,
-                                                   const char *thousands_sep);
+PyAPI_FUNC(Py_ssize_t) _PyUnicode_InsertThousandsGrouping(
+    int kind,
+    void *buffer,
+    Py_ssize_t n_buffer,
+    void *digits,
+    Py_ssize_t n_digits,
+    Py_ssize_t min_width,
+    const char *grouping,
+    const char *thousands_sep);
 #endif
 /* === Characters Type APIs =============================================== */
 
@@ -1673,6 +1891,43 @@
     Py_UNICODE c
     );
 
+PyAPI_FUNC(size_t) Py_UCS4_strlen(
+    const Py_UCS4 *u
+    );
+
+PyAPI_FUNC(Py_UCS4*) Py_UCS4_strcpy(
+    Py_UCS4 *s1,
+    const Py_UCS4 *s2);
+
+PyAPI_FUNC(Py_UCS4*) Py_UCS4_strcat(
+    Py_UCS4 *s1, const Py_UCS4 *s2);
+
+PyAPI_FUNC(Py_UCS4*) Py_UCS4_strncpy(
+    Py_UCS4 *s1,
+    const Py_UCS4 *s2,
+    size_t n);
+
+PyAPI_FUNC(int) Py_UCS4_strcmp(
+    const Py_UCS4 *s1,
+    const Py_UCS4 *s2
+    );
+
+PyAPI_FUNC(int) Py_UCS4_strncmp(
+    const Py_UCS4 *s1,
+    const Py_UCS4 *s2,
+    size_t n
+    );
+
+PyAPI_FUNC(Py_UCS4*) Py_UCS4_strchr(
+    const Py_UCS4 *s,
+    Py_UCS4 c
+    );
+
+PyAPI_FUNC(Py_UCS4*) Py_UCS4_strrchr(
+    const Py_UCS4 *s,
+    Py_UCS4 c
+    );
+
 /* Create a copy of a unicode string ending with a nul character. Return NULL
    and raise a MemoryError exception on memory allocation failure, otherwise
    return a new allocated buffer (use PyMem_Free() to free the buffer). */
diff --git a/Lib/json/decoder.py b/Lib/json/decoder.py
--- a/Lib/json/decoder.py
+++ b/Lib/json/decoder.py
@@ -121,8 +121,7 @@
                 msg = "Invalid \\uXXXX escape"
                 raise ValueError(errmsg(msg, s, end))
             uni = int(esc, 16)
-            # Check for surrogate pair on UCS-4 systems
-            if 0xd800 <= uni <= 0xdbff and sys.maxunicode > 65535:
+            if 0xd800 <= uni <= 0xdbff:
                 msg = "Invalid \\uXXXX\\uXXXX surrogate pair"
                 if not s[end + 5:end + 7] == '\\u':
                     raise ValueError(errmsg(msg, s, end))
diff --git a/Lib/test/json_tests/test_scanstring.py b/Lib/test/json_tests/test_scanstring.py
--- a/Lib/test/json_tests/test_scanstring.py
+++ b/Lib/test/json_tests/test_scanstring.py
@@ -9,14 +9,9 @@
             scanstring('"z\\ud834\\udd20x"', 1, True),
             ('z\U0001d120x', 16))
 
-        if sys.maxunicode == 65535:
-            self.assertEqual(
-                scanstring('"z\U0001d120x"', 1, True),
-                ('z\U0001d120x', 6))
-        else:
-            self.assertEqual(
-                scanstring('"z\U0001d120x"', 1, True),
-                ('z\U0001d120x', 5))
+        self.assertEqual(
+            scanstring('"z\U0001d120x"', 1, True),
+            ('z\U0001d120x', 5))
 
         self.assertEqual(
             scanstring('"\\u007b"', 1, True),
diff --git a/Lib/test/test_codeccallbacks.py b/Lib/test/test_codeccallbacks.py
--- a/Lib/test/test_codeccallbacks.py
+++ b/Lib/test/test_codeccallbacks.py
@@ -1,5 +1,6 @@
 import test.support, unittest
 import sys, codecs, html.entities, unicodedata
+import ctypes
 
 class PosReturn:
     # this can be used for configurable callbacks
@@ -577,8 +578,10 @@
                 UnicodeEncodeError("ascii", "\uffff", 0, 1, "ouch")),
             ("\\uffff", 1)
         )
-        # 1 on UCS-4 builds, 2 on UCS-2
-        len_wide = len("\U00010000")
+        if ctypes.sizeof(ctypes.c_wchar) == 2:
+            len_wide = 2
+        else:
+            len_wide = 1
         self.assertEqual(
             codecs.backslashreplace_errors(
                 UnicodeEncodeError("ascii", "\U00010000",
diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py
--- a/Lib/test/test_codecs.py
+++ b/Lib/test/test_codecs.py
@@ -622,6 +622,10 @@
                          b"abc\xed\xa0\x80def")
         self.assertEqual(b"abc\xed\xa0\x80def".decode("utf-8", "surrogatepass"),
                          "abc\ud800def")
+        self.assertEqual("\U00010fff\uD800".encode("utf-8", "surrogatepass"),
+                         b"\xf0\x90\xbf\xbf\xed\xa0\x80")
+        self.assertEqual(b"\xf0\x90\xbf\xbf\xed\xa0\x80".decode("utf-8", "surrogatepass"),
+                         "\U00010fff\uD800")
         self.assertTrue(codecs.lookup_error("surrogatepass"))
 
 class UTF7Test(ReadTest):
diff --git a/Lib/test/test_peepholer.py b/Lib/test/test_peepholer.py
--- a/Lib/test/test_peepholer.py
+++ b/Lib/test/test_peepholer.py
@@ -218,10 +218,6 @@
         # out of range
         asm = dis_single('"fuu"[10]')
         self.assertIn('BINARY_SUBSCR', asm)
-        # non-BMP char (see #5057)
-        asm = dis_single('"\U00012345"[0]')
-        self.assertIn('BINARY_SUBSCR', asm)
-
 
     def test_folding_of_unaryops_on_constants(self):
         for line, elem in (
diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py
--- a/Lib/test/test_re.py
+++ b/Lib/test/test_re.py
@@ -780,6 +780,13 @@
         self.assertRaises(OverflowError, _sre.compile, "abc", 0, [long_overflow])
         self.assertRaises(TypeError, _sre.compile, {}, 0, [])
 
+    def test_search_dot_unicode(self):
+        self.assertIsNotNone(re.search("123.*-", '123abc-'))
+        self.assertIsNotNone(re.search("123.*-", '123\xe9-'))
+        self.assertIsNotNone(re.search("123.*-", '123\u20ac-'))
+        self.assertIsNotNone(re.search("123.*-", '123\U0010ffff-'))
+        self.assertIsNotNone(re.search("123.*-", '123\xe9\u20ac\U0010ffff-'))
+
 def run_re_tests():
     from test.re_tests import tests, SUCCEED, FAIL, SYNTAX_ERROR
     if verbose:
diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py
--- a/Lib/test/test_sys.py
+++ b/Lib/test/test_sys.py
@@ -833,13 +833,39 @@
         class newstyleclass(object): pass
         check(newstyleclass, s)
         # unicode
-        usize = len('\0'.encode('unicode-internal'))
-        samples = ['', '1'*100]
-        # we need to test for both sizes, because we don't know if the string
-        # has been cached
+        # each tuple contains a string and its expected character size
+        # don't put any static strings here, as they may contain
+        # wchar_t or UTF-8 representations
+        samples = ['1'*100, '\xff'*50,
+                   '\u0100'*40, '\uffff'*100,
+                   '\U00010000'*30, '\U0010ffff'*100]
+        asciifields = h + "PPiP"
+        compactfields = asciifields + "PPP"
+        unicodefields = compactfields + "P"
         for s in samples:
-            basicsize =  size(h + 'PPPiP') + usize * (len(s) + 1)
-            check(s, basicsize)
+            maxchar = ord(max(s))
+            if maxchar < 128:
+                L = size(asciifields) + len(s) + 1
+            elif maxchar < 256:
+                L = size(compactfields) + len(s) + 1
+            elif maxchar < 65536:
+                L = size(compactfields) + 2*(len(s) + 1)
+            else:
+                L = size(compactfields) + 4*(len(s) + 1)
+            check(s, L)
+        # verify that the UTF-8 size is accounted for
+        s = chr(0x4000)   # 4 bytes canonical representation
+        check(s, size(compactfields) + 4)
+        try:
+            # FIXME: codecs.lookup(str) calls encoding.search_function() which
+            # calls __import__ using str in the module name. __import__ encodes
+            # the module name to the file system encoding (which is the locale
+            # encoding), so test_sys fails if the locale encoding is not UTF-8.
+            codecs.lookup(s) # produces 4 bytes UTF-8
+        except LookupError:
+            check(s, size(compactfields) + 4 + 4)
+        # TODO: add check that forces the presence of wchar_t representation
+        # TODO: add check that forces layout of unicodefields
         # weakref
         import weakref
         check(weakref.ref(int), size(h + '2Pl2P'))
diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -1583,16 +1583,32 @@
         self.assertRaises(OverflowError, 't\tt\t'.expandtabs, sys.maxsize)
 
     def test_raiseMemError(self):
-        # Ensure that the freelist contains a consistent object, even
-        # when a string allocation fails with a MemoryError.
-        # This used to crash the interpreter,
-        # or leak references when the number was smaller.
-        charwidth = 4 if sys.maxunicode >= 0x10000 else 2
-        # Note: sys.maxsize is half of the actual max allocation because of
-        # the signedness of Py_ssize_t.
-        alloc = lambda: "a" * (sys.maxsize // charwidth * 2)
-        self.assertRaises(MemoryError, alloc)
-        self.assertRaises(MemoryError, alloc)
+        if struct.calcsize('P') == 8:
+            # 64 bits pointers
+            ascii_struct_size = 64
+            compact_struct_size = 88
+        else:
+            # 32 bits pointers
+            ascii_struct_size = 32
+            compact_struct_size = 44
+
+        for char in ('a', '\xe9', '\u20ac', '\U0010ffff'):
+            code = ord(char)
+            if code < 0x100:
+                char_size = 1  # sizeof(Py_UCS1)
+                struct_size = ascii_struct_size
+            elif code < 0x10000:
+                char_size = 2  # sizeof(Py_UCS2)
+                struct_size = compact_struct_size
+            else:
+                char_size = 4  # sizeof(Py_UCS4)
+                struct_size = compact_struct_size
+            # Note: sys.maxsize is half of the actual max allocation because of
+            # the signedness of Py_ssize_t. -1 because of the null character.
+            maxlen = ((sys.maxsize - struct_size) // char_size) - 1
+            alloc = lambda: char * maxlen
+            self.assertRaises(MemoryError, alloc)
+            self.assertRaises(MemoryError, alloc)
 
     def test_format_subclass(self):
         class S(str):
@@ -1608,10 +1624,7 @@
         from ctypes import (pythonapi, py_object,
             c_int, c_long, c_longlong, c_ssize_t,
             c_uint, c_ulong, c_ulonglong, c_size_t)
-        if sys.maxunicode == 65535:
-            name = "PyUnicodeUCS2_FromFormat"
-        else:
-            name = "PyUnicodeUCS4_FromFormat"
+        name = "PyUnicode_FromFormat"
         _PyUnicode_FromFormat = getattr(pythonapi, name)
         _PyUnicode_FromFormat.restype = py_object
 
diff --git a/Makefile.pre.in b/Makefile.pre.in
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -629,7 +629,6 @@
 		$(srcdir)/Objects/stringlib/partition.h \
 		$(srcdir)/Objects/stringlib/split.h \
 		$(srcdir)/Objects/stringlib/stringdefs.h \
-		$(srcdir)/Objects/stringlib/string_format.h \
 		$(srcdir)/Objects/stringlib/transmogrify.h \
 		$(srcdir)/Objects/stringlib/unicodedefs.h \
 		$(srcdir)/Objects/stringlib/localeutil.h
@@ -639,7 +638,7 @@
 Objects/bytearrayobject.o: $(srcdir)/Objects/bytearrayobject.c $(BYTESTR_DEPS)
 
 Objects/unicodeobject.o: $(srcdir)/Objects/unicodeobject.c \
-				$(BYTESTR_DEPS)
+				$(BYTESTR_DEPS) $(srcdir)/Objects/stringlib/unicode_format.h
 
 Objects/dictobject.o: $(srcdir)/Objects/stringlib/eq.h
 Objects/setobject.o: $(srcdir)/Objects/stringlib/eq.h
@@ -650,8 +649,7 @@
 Python/ceval.o: $(OPCODETARGETS_H) Python/ceval_gil.h
 
 Python/formatter_unicode.o: $(srcdir)/Python/formatter_unicode.c \
-				$(BYTESTR_DEPS) \
-				$(srcdir)/Objects/stringlib/formatter.h
+				$(BYTESTR_DEPS)
 
 Objects/typeobject.o: $(srcdir)/Objects/typeslots.inc
 $(srcdir)/Objects/typeslots.inc: $(srcdir)/Include/typeslots.h $(srcdir)/Objects/typeslots.py
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,8 @@
 Core and Builtins
 -----------------
 
+- PEP 393: flexible string representation.
+
 - Issue #13012: The 'keepends' parameter to str.splitlines may now be passed
   as a keyword argument:  "my_string.splitlines(keepends=True)".  The same
   change also applies to bytes.splitlines and bytearray.splitlines.
diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c
--- a/Modules/_codecsmodule.c
+++ b/Modules/_codecsmodule.c
@@ -700,12 +700,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL)
+    if (str == NULL || PyUnicode_READY(str) == -1)
         return NULL;
-    v = codec_tuple(PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(str),
-                                         PyUnicode_GET_SIZE(str),
-                                         errors),
-                    PyUnicode_GET_SIZE(str));
+    v = codec_tuple(PyUnicode_AsEncodedString(str, "utf-8", errors),
+                    PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
     return v;
 }
diff --git a/Modules/_csv.c b/Modules/_csv.c
--- a/Modules/_csv.c
+++ b/Modules/_csv.c
@@ -128,7 +128,7 @@
         return Py_None;
     }
     else
-        return PyUnicode_FromUnicode((Py_UNICODE *)&c, 1);
+        return PyUnicode_FromOrdinal(c);
 }
 
 static PyObject *
diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c
--- a/Modules/_ctypes/_ctypes.c
+++ b/Modules/_ctypes/_ctypes.c
@@ -1843,11 +1843,9 @@
         return NULL;
     }
     if (PyUnicode_Check(proto)) {
-        PyObject *v = _PyUnicode_AsDefaultEncodedString(proto);
-        if (!v)
+        proto_str = PyUnicode_AsUTF8AndSize(proto, &proto_len);
+        if (!proto_str)
             goto error;
-        proto_str = PyBytes_AS_STRING(v);
-        proto_len = PyBytes_GET_SIZE(v);
     } else {
         PyErr_SetString(PyExc_TypeError,
             "class must define a '_type_' string attribute");
diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c
--- a/Modules/_ctypes/callproc.c
+++ b/Modules/_ctypes/callproc.c
@@ -658,13 +658,6 @@
 
 #ifdef CTYPES_UNICODE
     if (PyUnicode_Check(obj)) {
-#if Py_UNICODE_SIZE == SIZEOF_WCHAR_T
-        pa->ffi_type = &ffi_type_pointer;
-        pa->value.p = PyUnicode_AS_UNICODE(obj);
-        Py_INCREF(obj);
-        pa->keep = obj;
-        return 0;
-#else
         pa->ffi_type = &ffi_type_pointer;
         pa->value.p = PyUnicode_AsWideCharString(obj, NULL);
         if (pa->value.p == NULL)
@@ -675,7 +668,6 @@
             return -1;
         }
         return 0;
-#endif
     }
 #endif
 
diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c
--- a/Modules/_ctypes/cfield.c
+++ b/Modules/_ctypes/cfield.c
@@ -9,7 +9,6 @@
 
 #define CTYPES_CFIELD_CAPSULE_NAME_PYMEM "_ctypes/cfield.c pymem"
 
-#if Py_UNICODE_SIZE != SIZEOF_WCHAR_T
 static void pymem_destructor(PyObject *ptr)
 {
     void *p = PyCapsule_GetPointer(ptr, CTYPES_CFIELD_CAPSULE_NAME_PYMEM);
@@ -17,7 +16,6 @@
         PyMem_Free(p);
     }
 }
-#endif
 
 
 /******************************************************************/
@@ -1238,32 +1236,24 @@
 static PyObject *
 U_get(void *ptr, Py_ssize_t size)
 {
-    PyObject *result;
     Py_ssize_t len;
-    Py_UNICODE *p;
+    wchar_t *p;
 
     size /= sizeof(wchar_t); /* we count character units here, not bytes */
 
-    result = PyUnicode_FromWideChar((wchar_t *)ptr, size);
-    if (!result)
-        return NULL;
     /* We need 'result' to be able to count the characters with wcslen,
        since ptr may not be NUL terminated.  If the length is smaller (if
        it was actually NUL terminated, we construct a new one and throw
        away the result.
     */
     /* chop off at the first NUL character, if any. */
-    p = PyUnicode_AS_UNICODE(result);
-    for (len = 0; len < size; ++len)
+    p = (wchar_t*)ptr;
+    for (len = 0; len < size; ++len) {
         if (!p[len])
             break;
+    }
 
-    if (len < size) {
-        PyObject *ob = PyUnicode_FromWideChar((wchar_t *)ptr, len);
-        Py_DECREF(result);
-        return ob;
-    }
-    return result;
+    return PyUnicode_FromWideChar((wchar_t *)ptr, len);
 }
 
 static PyObject *
@@ -1401,6 +1391,9 @@
 static PyObject *
 Z_set(void *ptr, PyObject *value, Py_ssize_t size)
 {
+    PyObject *keep;
+    wchar_t *buffer;
+
     if (value == Py_None) {
         *(wchar_t **)ptr = NULL;
         Py_INCREF(value);
@@ -1420,37 +1413,20 @@
                      "unicode string or integer address expected instead of %s instance",
                      value->ob_type->tp_name);
         return NULL;
-    } else
-        Py_INCREF(value);
-#if Py_UNICODE_SIZE == SIZEOF_WCHAR_T
-    /* We can copy directly.  Hm, are unicode objects always NUL
-       terminated in Python, internally?
-     */
-    *(wchar_t **)ptr = (wchar_t *) PyUnicode_AS_UNICODE(value);
-    return value;
-#else
-    {
-        /* We must create a wchar_t* buffer from the unicode object,
-           and keep it alive */
-        PyObject *keep;
-        wchar_t *buffer;
+    }
 
-        buffer = PyUnicode_AsWideCharString(value, NULL);
-        if (!buffer) {
-            Py_DECREF(value);
-            return NULL;
-        }
-        keep = PyCapsule_New(buffer, CTYPES_CFIELD_CAPSULE_NAME_PYMEM, pymem_destructor);
-        if (!keep) {
-            Py_DECREF(value);
-            PyMem_Free(buffer);
-            return NULL;
-        }
-        *(wchar_t **)ptr = (wchar_t *)buffer;
-        Py_DECREF(value);
-        return keep;
+    /* We must create a wchar_t* buffer from the unicode object,
+       and keep it alive */
+    buffer = PyUnicode_AsWideCharString(value, NULL);
+    if (!buffer)
+        return NULL;
+    keep = PyCapsule_New(buffer, CTYPES_CFIELD_CAPSULE_NAME_PYMEM, pymem_destructor);
+    if (!keep) {
+        PyMem_Free(buffer);
+        return NULL;
     }
-#endif
+    *(wchar_t **)ptr = buffer;
+    return keep;
 }
 
 static PyObject *
diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c
--- a/Modules/_cursesmodule.c
+++ b/Modules/_cursesmodule.c
@@ -203,8 +203,11 @@
     } else if(PyBytes_Check(obj)
               && (PyBytes_Size(obj) == 1)) {
         *ch = (chtype) *PyBytes_AsString(obj);
-    } else if (PyUnicode_Check(obj) && PyUnicode_GetSize(obj) == 1) {
-        *ch = (chtype) *PyUnicode_AS_UNICODE(obj);
+    } else if (PyUnicode_Check(obj) && PyUnicode_GET_LENGTH(obj) == 1) {
+        Py_UCS4 ucs = PyUnicode_READ(PyUnicode_KIND(obj),
+                                     PyUnicode_DATA(obj),
+                                     0);
+        *ch = (chtype)ucs;
     } else {
         return 0;
     }
diff --git a/Modules/_datetimemodule.c b/Modules/_datetimemodule.c
--- a/Modules/_datetimemodule.c
+++ b/Modules/_datetimemodule.c
@@ -985,9 +985,8 @@
     if (tzinfo == Py_None)
         return repr;
     /* Get rid of the trailing ')'. */
-    assert(PyUnicode_AS_UNICODE(repr)[PyUnicode_GET_SIZE(repr)-1] == ')');
-    temp = PyUnicode_FromUnicode(PyUnicode_AS_UNICODE(repr),
-                                      PyUnicode_GET_SIZE(repr) - 1);
+    assert(PyUnicode_READ_CHAR(repr, PyUnicode_GET_LENGTH(repr)-1) == ')');
+    temp = PyUnicode_Substring(repr, 0, PyUnicode_GET_LENGTH(repr) - 1);
     Py_DECREF(repr);
     if (temp == NULL)
         return NULL;
@@ -4214,9 +4213,9 @@
 datetime_strptime(PyObject *cls, PyObject *args)
 {
     static PyObject *module = NULL;
-    const Py_UNICODE *string, *format;
-
-    if (!PyArg_ParseTuple(args, "uu:strptime", &string, &format))
+    PyObject *string, *format;
+
+    if (!PyArg_ParseTuple(args, "UU:strptime", &string, &format))
         return NULL;
 
     if (module == NULL) {
@@ -4224,7 +4223,7 @@
         if (module == NULL)
             return NULL;
     }
-    return PyObject_CallMethod(module, "_strptime_datetime", "Ouu",
+    return PyObject_CallMethod(module, "_strptime_datetime", "OOO",
                                cls, string, format);
 }
 
diff --git a/Modules/_dbmmodule.c b/Modules/_dbmmodule.c
--- a/Modules/_dbmmodule.c
+++ b/Modules/_dbmmodule.c
@@ -212,6 +212,7 @@
 {
     dbmobject *dp = (dbmobject *)self;
     datum key, val;
+    Py_ssize_t size;
 
     if ((dp)->di_dbm == NULL) {
         PyErr_SetString(DbmError,
@@ -219,8 +220,9 @@
          return -1;
     }
     if (PyUnicode_Check(arg)) {
-        arg = _PyUnicode_AsDefaultEncodedString(arg);
-        if (arg == NULL)
+        key.dptr = PyUnicode_AsUTF8AndSize(arg, &size);
+        key.dsize = size;
+        if (key.dptr == NULL)
             return -1;
     }
     if (!PyBytes_Check(arg)) {
@@ -229,8 +231,10 @@
                      arg->ob_type->tp_name);
         return -1;
     }
-    key.dptr = PyBytes_AS_STRING(arg);
-    key.dsize = PyBytes_GET_SIZE(arg);
+    else {
+        key.dptr = PyBytes_AS_STRING(arg);
+        key.dsize = PyBytes_GET_SIZE(arg);
+    }
     val = dbm_fetch(dp->di_dbm, key);
     return val.dptr != NULL;
 }
diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c
--- a/Modules/_elementtree.c
+++ b/Modules/_elementtree.c
@@ -723,13 +723,16 @@
     (ch == '/' || ch == '*' || ch == '[' || ch == '@' || ch == '.')
 
     if (PyUnicode_Check(tag)) {
-        Py_UNICODE *p = PyUnicode_AS_UNICODE(tag);
-        for (i = 0; i < PyUnicode_GET_SIZE(tag); i++) {
-            if (p[i] == '{')
+        const Py_ssize_t len = PyUnicode_GET_LENGTH(tag);
+        void *data = PyUnicode_DATA(tag);
+        unsigned int kind = PyUnicode_KIND(tag);
+        for (i = 0; i < len; i++) {
+            Py_UCS4 ch = PyUnicode_READ(kind, data, i);
+            if (ch == '{')
                 check = 0;
-            else if (p[i] == '}')
+            else if (ch == '}')
                 check = 1;
-            else if (check && PATHCHAR(p[i]))
+            else if (check && PATHCHAR(ch))
                 return 1;
         }
         return 0;
@@ -2401,9 +2404,10 @@
                                XML_Encoding *info)
 {
     PyObject* u;
-    Py_UNICODE* p;
     unsigned char s[256];
     int i;
+    void *data;
+    unsigned int kind;
 
     memset(info, 0, sizeof(XML_Encoding));
 
@@ -2413,17 +2417,20 @@
     u = PyUnicode_Decode((char*) s, 256, name, "replace");
     if (!u)
         return XML_STATUS_ERROR;
-
-    if (PyUnicode_GET_SIZE(u) != 256) {
+    if (PyUnicode_READY(u))
+        return XML_STATUS_ERROR;
+
+    if (PyUnicode_GET_LENGTH(u) != 256) {
         Py_DECREF(u);
         return XML_STATUS_ERROR;
     }
 
-    p = PyUnicode_AS_UNICODE(u);
-
+    kind = PyUnicode_KIND(u);
+    data = PyUnicode_DATA(u);
     for (i = 0; i < 256; i++) {
-        if (p[i] != Py_UNICODE_REPLACEMENT_CHARACTER)
-            info->map[i] = p[i];
+        Py_UCS4 ch = PyUnicode_READ(kind, data, i);
+        if (ch != Py_UNICODE_REPLACEMENT_CHARACTER)
+            info->map[i] = ch;
         else
             info->map[i] = -1;
     }
diff --git a/Modules/_io/_iomodule.h b/Modules/_io/_iomodule.h
--- a/Modules/_io/_iomodule.h
+++ b/Modules/_io/_iomodule.h
@@ -55,7 +55,7 @@
    Otherwise, the function will scan further and return garbage. */
 extern Py_ssize_t _PyIO_find_line_ending(
     int translated, int universal, PyObject *readnl,
-    Py_UNICODE *start, Py_UNICODE *end, Py_ssize_t *consumed);
+    int kind, char *start, char *end, Py_ssize_t *consumed);
 
 
 #define DEFAULT_BUFFER_SIZE (8 * 1024)  /* bytes */
diff --git a/Modules/_io/stringio.c b/Modules/_io/stringio.c
--- a/Modules/_io/stringio.c
+++ b/Modules/_io/stringio.c
@@ -9,7 +9,7 @@
 
 typedef struct {
     PyObject_HEAD
-    Py_UNICODE *buf;
+    Py_UCS4 *buf;
     Py_ssize_t pos;
     Py_ssize_t string_size;
     size_t buf_size;
@@ -21,7 +21,7 @@
     PyObject *decoder;
     PyObject *readnl;
     PyObject *writenl;
-    
+
     PyObject *dict;
     PyObject *weakreflist;
 } stringio;
@@ -56,7 +56,7 @@
     /* Here, unsigned types are used to avoid dealing with signed integer
        overflow, which is undefined in C. */
     size_t alloc = self->buf_size;
-    Py_UNICODE *new_buf = NULL;
+    Py_UCS4 *new_buf = NULL;
 
     assert(self->buf != NULL);
 
@@ -84,10 +84,9 @@
         alloc = size + 1;
     }
 
-    if (alloc > ((size_t)-1) / sizeof(Py_UNICODE))
+    if (alloc > PY_SIZE_MAX / sizeof(Py_UCS4))
         goto overflow;
-    new_buf = (Py_UNICODE *)PyMem_Realloc(self->buf,
-                                          alloc * sizeof(Py_UNICODE));
+    new_buf = (Py_UCS4 *)PyMem_Realloc(self->buf, alloc * sizeof(Py_UCS4));
     if (new_buf == NULL) {
         PyErr_NoMemory();
         return -1;
@@ -108,9 +107,9 @@
 static Py_ssize_t
 write_str(stringio *self, PyObject *obj)
 {
-    Py_UNICODE *str;
     Py_ssize_t len;
     PyObject *decoded = NULL;
+
     assert(self->buf != NULL);
     assert(self->pos >= 0);
 
@@ -132,8 +131,7 @@
         return -1;
 
     assert(PyUnicode_Check(decoded));
-    str = PyUnicode_AS_UNICODE(decoded);
-    len = PyUnicode_GET_SIZE(decoded);
+    len = PyUnicode_GET_LENGTH(decoded);
 
     assert(len >= 0);
 
@@ -161,18 +159,21 @@
 
         */
         memset(self->buf + self->string_size, '\0',
-               (self->pos - self->string_size) * sizeof(Py_UNICODE));
+               (self->pos - self->string_size) * sizeof(Py_UCS4));
     }
 
     /* Copy the data to the internal buffer, overwriting some of the
        existing data if self->pos < self->string_size. */
-    memcpy(self->buf + self->pos, str, len * sizeof(Py_UNICODE));
-    self->pos += len;
+    if (!PyUnicode_AsUCS4(decoded,
+                          self->buf + self->pos,
+                          self->buf_size - self->pos,
+                          0))
+        goto fail;
 
     /* Set the new length of the internal string if it has changed. */
-    if (self->string_size < self->pos) {
+    self->pos += len;
+    if (self->string_size < self->pos)
         self->string_size = self->pos;
-    }
 
     Py_DECREF(decoded);
     return 0;
@@ -190,7 +191,8 @@
 {
     CHECK_INITIALIZED(self);
     CHECK_CLOSED(self);
-    return PyUnicode_FromUnicode(self->buf, self->string_size);
+    return PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, self->buf,
+                                     self->string_size);
 }
 
 PyDoc_STRVAR(stringio_tell_doc,
@@ -214,7 +216,7 @@
 stringio_read(stringio *self, PyObject *args)
 {
     Py_ssize_t size, n;
-    Py_UNICODE *output;
+    Py_UCS4 *output;
     PyObject *arg = Py_None;
 
     CHECK_INITIALIZED(self);
@@ -247,19 +249,19 @@
 
     output = self->buf + self->pos;
     self->pos += size;
-    return PyUnicode_FromUnicode(output, size);
+    return PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, output, size);
 }
 
 /* Internal helper, used by stringio_readline and stringio_iternext */
 static PyObject *
 _stringio_readline(stringio *self, Py_ssize_t limit)
 {
-    Py_UNICODE *start, *end, old_char;
+    Py_UCS4 *start, *end, old_char;
     Py_ssize_t len, consumed;
 
     /* In case of overseek, return the empty string */
     if (self->pos >= self->string_size)
-        return PyUnicode_FromString("");
+        return PyUnicode_New(0, 0);
 
     start = self->buf + self->pos;
     if (limit < 0 || limit > self->string_size - self->pos)
@@ -270,14 +272,14 @@
     *end = '\0';
     len = _PyIO_find_line_ending(
         self->readtranslate, self->readuniversal, self->readnl,
-        start, end, &consumed);
+        PyUnicode_4BYTE_KIND, (char*)start, (char*)end, &consumed);
     *end = old_char;
     /* If we haven't found any line ending, we just return everything
        (`consumed` is ignored). */
     if (len < 0)
         len = limit;
     self->pos += len;
-    return PyUnicode_FromUnicode(start, len);
+    return PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, start, len);
 }
 
 PyDoc_STRVAR(stringio_readline_doc,
@@ -462,8 +464,10 @@
                      Py_TYPE(obj)->tp_name);
         return NULL;
     }
+    if (PyUnicode_READY(obj))
+        return NULL;
     CHECK_CLOSED(self);
-    size = PyUnicode_GET_SIZE(obj);
+    size = PyUnicode_GET_LENGTH(obj);
 
     if (size > 0 && write_str(self, obj) < 0)
         return NULL;
@@ -535,7 +539,7 @@
     /* tp_alloc initializes all the fields to zero. So we don't have to
        initialize them here. */
 
-    self->buf = (Py_UNICODE *)PyMem_Malloc(0);
+    self->buf = (Py_UCS4 *)PyMem_Malloc(0);
     if (self->buf == NULL) {
         Py_DECREF(self);
         return PyErr_NoMemory();
@@ -747,11 +751,22 @@
        once by __init__. So we do not take any chance and replace object's
        buffer completely. */
     {
-        Py_UNICODE *buf = PyUnicode_AS_UNICODE(PyTuple_GET_ITEM(state, 0));
-        Py_ssize_t bufsize = PyUnicode_GET_SIZE(PyTuple_GET_ITEM(state, 0));
-        if (resize_buffer(self, bufsize) < 0)
+        PyObject *item;
+        Py_UCS4 *buf;
+        Py_ssize_t bufsize;
+
+        item = PyTuple_GET_ITEM(state, 0);
+        buf = PyUnicode_AsUCS4Copy(item);
+        if (buf == NULL)
             return NULL;
-        memcpy(self->buf, buf, bufsize * sizeof(Py_UNICODE));
+        bufsize = PyUnicode_GET_LENGTH(item);
+
+        if (resize_buffer(self, bufsize) < 0) {
+            PyMem_Free(buf);
+            return NULL;
+        }
+        memcpy(self->buf, buf, bufsize * sizeof(Py_UCS4));
+        PyMem_Free(buf);
         self->string_size = bufsize;
     }
 
diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c
--- a/Modules/_io/textio.c
+++ b/Modules/_io/textio.c
@@ -274,18 +274,28 @@
         goto error;
     }
 
-    output_len = PyUnicode_GET_SIZE(output);
+    if (PyUnicode_READY(output) == -1)
+        goto error;
+
+    output_len = PyUnicode_GET_LENGTH(output);
     if (self->pendingcr && (final || output_len > 0)) {
-        Py_UNICODE *out;
-        PyObject *modified = PyUnicode_FromUnicode(NULL, output_len + 1);
+        /* Prefix output with CR */
+        int kind;
+        PyObject *modified;
+        char *out;
+
+        modified = PyUnicode_New(output_len + 1,
+                                 PyUnicode_MAX_CHAR_VALUE(output));
         if (modified == NULL)
             goto error;
-        out = PyUnicode_AS_UNICODE(modified);
-        out[0] = '\r';
-        memcpy(out + 1, PyUnicode_AS_UNICODE(output),
-               output_len * sizeof(Py_UNICODE));
+        kind = PyUnicode_KIND(modified);
+        out = PyUnicode_DATA(modified);
+        PyUnicode_WRITE(kind, PyUnicode_DATA(modified), 0, '\r');
+        memcpy(out + PyUnicode_KIND_SIZE(kind, 1),
+               PyUnicode_DATA(output),
+               PyUnicode_KIND_SIZE(kind, output_len));
         Py_DECREF(output);
-        output = modified;
+        output = modified; /* output remains ready */
         self->pendingcr = 0;
         output_len++;
     }
@@ -295,21 +305,13 @@
      */
     if (!final) {
         if (output_len > 0
-            && PyUnicode_AS_UNICODE(output)[output_len - 1] == '\r') {
-
-            if (Py_REFCNT(output) == 1) {
-                if (PyUnicode_Resize(&output, output_len - 1) < 0)
-                    goto error;
-            }
-            else {
-                PyObject *modified = PyUnicode_FromUnicode(
-                    PyUnicode_AS_UNICODE(output),
-                    output_len - 1);
-                if (modified == NULL)
-                    goto error;
-                Py_DECREF(output);
-                output = modified;
-            }
+            && PyUnicode_READ_CHAR(output, output_len - 1) == '\r')
+        {
+            PyObject *modified = PyUnicode_Substring(output, 0, output_len -1);
+            if (modified == NULL)
+                goto error;
+            Py_DECREF(output);
+            output = modified;
             self->pendingcr = 1;
         }
     }
@@ -317,13 +319,15 @@
     /* Record which newlines are read and do newline translation if desired,
        all in one pass. */
     {
-        Py_UNICODE *in_str;
+        void *in_str;
         Py_ssize_t len;
         int seennl = self->seennl;
         int only_lf = 0;
+        int kind;
 
-        in_str = PyUnicode_AS_UNICODE(output);
-        len = PyUnicode_GET_SIZE(output);
+        in_str = PyUnicode_DATA(output);
+        len = PyUnicode_GET_LENGTH(output);
+        kind = PyUnicode_KIND(output);
 
         if (len == 0)
             return output;
@@ -332,7 +336,7 @@
            for the \r *byte* with the libc's optimized memchr.
            */
         if (seennl == SEEN_LF || seennl == 0) {
-            only_lf = (memchr(in_str, '\r', len * sizeof(Py_UNICODE)) == NULL);
+            only_lf = (memchr(in_str, '\r', PyUnicode_KIND_SIZE(kind, len)) == NULL);
         }
 
         if (only_lf) {
@@ -340,21 +344,19 @@
                (there's nothing else to be done, even when in translation mode)
             */
             if (seennl == 0 &&
-                memchr(in_str, '\n', len * sizeof(Py_UNICODE)) != NULL) {
-                Py_UNICODE *s, *end;
-                s = in_str;
-                end = in_str + len;
+                memchr(in_str, '\n', PyUnicode_KIND_SIZE(kind, len)) != NULL) {
+                Py_ssize_t i = 0;
                 for (;;) {
                     Py_UNICODE c;
                     /* Fast loop for non-control characters */
-                    while (*s > '\n')
-                        s++;
-                    c = *s++;
+                    while (PyUnicode_READ(kind, in_str, i) > '\n')
+                        i++;
+                    c = PyUnicode_READ(kind, in_str, i++);
                     if (c == '\n') {
                         seennl |= SEEN_LF;
                         break;
                     }
-                    if (s > end)
+                    if (i >= len)
                         break;
                 }
             }
@@ -362,29 +364,27 @@
                need translating */
         }
         else if (!self->translate) {
-            Py_UNICODE *s, *end;
+            Py_ssize_t i = 0;
             /* We have already seen all newline types, no need to scan again */
             if (seennl == SEEN_ALL)
                 goto endscan;
-            s = in_str;
-            end = in_str + len;
             for (;;) {
-                Py_UNICODE c;
+                Py_UCS4 c;
                 /* Fast loop for non-control characters */
-                while (*s > '\r')
-                    s++;
-                c = *s++;
+                while (PyUnicode_READ(kind, in_str, i) > '\r')
+                    i++;
+                c = PyUnicode_READ(kind, in_str, i++);
                 if (c == '\n')
                     seennl |= SEEN_LF;
                 else if (c == '\r') {
-                    if (*s == '\n') {
+                    if (PyUnicode_READ(kind, in_str, i) == '\n') {
                         seennl |= SEEN_CRLF;
-                        s++;
+                        i++;
                     }
                     else
                         seennl |= SEEN_CR;
                 }
-                if (s > end)
+                if (i >= len)
                     break;
                 if (seennl == SEEN_ALL)
                     break;
@@ -393,61 +393,50 @@
             ;
         }
         else {
-            PyObject *translated = NULL;
-            Py_UNICODE *out_str;
-            Py_UNICODE *in, *out, *end;
-            if (Py_REFCNT(output) != 1) {
-                /* We could try to optimize this so that we only do a copy
-                   when there is something to translate. On the other hand,
-                   most decoders should only output non-shared strings, i.e.
-                   translation is done in place. */
-                translated = PyUnicode_FromUnicode(NULL, len);
-                if (translated == NULL)
-                    goto error;
-                assert(Py_REFCNT(translated) == 1);
-                memcpy(PyUnicode_AS_UNICODE(translated),
-                       PyUnicode_AS_UNICODE(output),
-                       len * sizeof(Py_UNICODE));
+            void *translated;
+            int kind = PyUnicode_KIND(output);
+            void *in_str = PyUnicode_DATA(output);
+            Py_ssize_t in, out;
+            /* XXX: Previous in-place translation here is disabled as
+               resizing is not possible anymore */
+            /* We could try to optimize this so that we only do a copy
+               when there is something to translate. On the other hand,
+               we already know there is a \r byte, so chances are high
+               that something needs to be done. */
+            translated = PyMem_Malloc(PyUnicode_KIND_SIZE(kind, len));
+            if (translated == NULL) {
+                PyErr_NoMemory();
+                goto error;
             }
-            else {
-                translated = output;
-            }
-            out_str = PyUnicode_AS_UNICODE(translated);
-            in = in_str;
-            out = out_str;
-            end = in_str + len;
+            in = out = 0;
             for (;;) {
-                Py_UNICODE c;
+                Py_UCS4 c;
                 /* Fast loop for non-control characters */
-                while ((c = *in++) > '\r')
-                    *out++ = c;
+                while ((c = PyUnicode_READ(kind, in_str, in++)) > '\r')
+                    PyUnicode_WRITE(kind, translated, out++, c);
                 if (c == '\n') {
-                    *out++ = c;
+                    PyUnicode_WRITE(kind, translated, out++, c);
                     seennl |= SEEN_LF;
                     continue;
                 }
                 if (c == '\r') {
-                    if (*in == '\n') {
+                    if (PyUnicode_READ(kind, in_str, in) == '\n') {
                         in++;
                         seennl |= SEEN_CRLF;
                     }
                     else
                         seennl |= SEEN_CR;
-                    *out++ = '\n';
+                    PyUnicode_WRITE(kind, translated, out++, '\n');
                     continue;
                 }
-                if (in > end)
+                if (in > len)
                     break;
-                *out++ = c;
+                PyUnicode_WRITE(kind, translated, out++, c);
             }
-            if (translated != output) {
-                Py_DECREF(output);
-                output = translated;
-            }
-            if (out - out_str != len) {
-                if (PyUnicode_Resize(&output, out - out_str) < 0)
-                    goto error;
-            }
+            Py_DECREF(output);
+            output = PyUnicode_FromKindAndData(kind, translated, out);
+            if (!output)
+                goto error;
         }
         self->seennl |= seennl;
     }
@@ -705,9 +694,7 @@
 static PyObject *
 ascii_encode(textio *self, PyObject *text)
 {
-    return PyUnicode_EncodeASCII(PyUnicode_AS_UNICODE(text),
-                                 PyUnicode_GET_SIZE(text),
-                                 PyBytes_AS_STRING(self->errors));
+    return _PyUnicode_AsASCIIString(text, PyBytes_AS_STRING(self->errors));
 }
 
 static PyObject *
@@ -777,17 +764,13 @@
 static PyObject *
 utf8_encode(textio *self, PyObject *text)
 {
-    return PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(text),
-                                PyUnicode_GET_SIZE(text),
-                                PyBytes_AS_STRING(self->errors));
+    return _PyUnicode_AsUTF8String(text, PyBytes_AS_STRING(self->errors));
 }
 
 static PyObject *
 latin1_encode(textio *self, PyObject *text)
 {
-    return PyUnicode_EncodeLatin1(PyUnicode_AS_UNICODE(text),
-                                  PyUnicode_GET_SIZE(text),
-                                  PyBytes_AS_STRING(self->errors));
+    return _PyUnicode_AsLatin1String(text, PyBytes_AS_STRING(self->errors));
 }
 
 /* Map normalized encoding names onto the specialized encoding funcs */
@@ -1213,18 +1196,6 @@
     return buffer;
 }
 
-Py_LOCAL_INLINE(const Py_UNICODE *)
-findchar(const Py_UNICODE *s, Py_ssize_t size, Py_UNICODE ch)
-{
-    /* like wcschr, but doesn't stop at NULL characters */
-    while (size-- > 0) {
-        if (*s == ch)
-            return s;
-        s++;
-    }
-    return NULL;
-}
-
 /* Flush the internal write buffer. This doesn't explicitly flush the
    underlying buffered object, though. */
 static int
@@ -1269,6 +1240,9 @@
         return NULL;
     }
 
+    if (PyUnicode_READY(text) == -1)
+        return NULL;
+
     CHECK_CLOSED(self);
 
     if (self->encoder == NULL)
@@ -1276,11 +1250,10 @@
 
     Py_INCREF(text);
 
-    textlen = PyUnicode_GetSize(text);
+    textlen = PyUnicode_GET_LENGTH(text);
 
     if ((self->writetranslate && self->writenl != NULL) || self->line_buffering)
-        if (findchar(PyUnicode_AS_UNICODE(text),
-                     PyUnicode_GET_SIZE(text), '\n'))
+        if (PyUnicode_FindChar(text, '\n', 0, PyUnicode_GET_LENGTH(text), 1) != -1)
             haslf = 1;
 
     if (haslf && self->writetranslate && self->writenl != NULL) {
@@ -1296,8 +1269,7 @@
         needflush = 1;
     else if (self->line_buffering &&
         (haslf ||
-         findchar(PyUnicode_AS_UNICODE(text),
-                  PyUnicode_GET_SIZE(text), '\r')))
+         PyUnicode_FindChar(text, '\r', 0, PyUnicode_GET_LENGTH(text), 1) != -1))
         needflush = 1;
 
     /* XXX What if we were just reading? */
@@ -1369,7 +1341,8 @@
     if (self->decoded_chars == NULL)
         return PyUnicode_FromStringAndSize(NULL, 0);
 
-    avail = (PyUnicode_GET_SIZE(self->decoded_chars)
+    /* decoded_chars is guaranteed to be "ready". */
+    avail = (PyUnicode_GET_LENGTH(self->decoded_chars)
              - self->decoded_chars_used);
 
     assert(avail >= 0);
@@ -1378,9 +1351,9 @@
         n = avail;
 
     if (self->decoded_chars_used > 0 || n < avail) {
-        chars = PyUnicode_FromUnicode(
-            PyUnicode_AS_UNICODE(self->decoded_chars)
-            + self->decoded_chars_used, n);
+        chars = PyUnicode_Substring(self->decoded_chars,
+                                    self->decoded_chars_used,
+                                    self->decoded_chars_used + n);
         if (chars == NULL)
             return NULL;
     }
@@ -1464,8 +1437,10 @@
     /* TODO sanity check: isinstance(decoded_chars, unicode) */
     if (decoded_chars == NULL)
         goto fail;
+    if (PyUnicode_READY(decoded_chars) == -1)
+        goto fail;
     textiowrapper_set_decoded_chars(self, decoded_chars);
-    nchars = PyUnicode_GET_SIZE(decoded_chars);
+    nchars = PyUnicode_GET_LENGTH(decoded_chars);
     if (nchars > 0)
         self->b2cratio = (double) nbytes / nchars;
     else
@@ -1553,7 +1528,9 @@
         result = textiowrapper_get_decoded_chars(self, n);
         if (result == NULL)
             goto fail;
-        remaining -= PyUnicode_GET_SIZE(result);
+        if (PyUnicode_READY(result) == -1)
+            goto fail;
+        remaining -= PyUnicode_GET_LENGTH(result);
 
         /* Keep reading chunks until we have n characters to return */
         while (remaining > 0) {
@@ -1573,7 +1550,7 @@
             result = textiowrapper_get_decoded_chars(self, remaining);
             if (result == NULL)
                 goto fail;
-            remaining -= PyUnicode_GET_SIZE(result);
+            remaining -= PyUnicode_GET_LENGTH(result);
         }
         if (chunks != NULL) {
             if (result != NULL && PyList_Append(chunks, result) < 0)
@@ -1596,33 +1573,34 @@
 /* NOTE: `end` must point to the real end of the Py_UNICODE storage,
    that is to the NUL character. Otherwise the function will produce
    incorrect results. */
-static Py_UNICODE *
-find_control_char(Py_UNICODE *start, Py_UNICODE *end, Py_UNICODE ch)
+static char *
+find_control_char(int kind, char *s, char *end, Py_UCS4 ch)
 {
-    Py_UNICODE *s = start;
+    int size = PyUnicode_KIND_SIZE(kind, 1);
     for (;;) {
-        while (*s > ch)
-            s++;
-        if (*s == ch)
+        while (PyUnicode_READ(kind, s, 0) > ch)
+            s += size;
+        if (PyUnicode_READ(kind, s, 0) == ch)
             return s;
         if (s == end)
             return NULL;
-        s++;
+        s += size;
     }
 }
 
 Py_ssize_t
 _PyIO_find_line_ending(
     int translated, int universal, PyObject *readnl,
-    Py_UNICODE *start, Py_UNICODE *end, Py_ssize_t *consumed)
+    int kind, char *start, char *end, Py_ssize_t *consumed)
 {
-    Py_ssize_t len = end - start;
+    int size = PyUnicode_KIND_SIZE(kind, 1);
+    Py_ssize_t len = ((char*)end - (char*)start)/size;
 
     if (translated) {
         /* Newlines are already translated, only search for \n */
-        Py_UNICODE *pos = find_control_char(start, end, '\n');
+        char *pos = find_control_char(kind, start, end, '\n');
         if (pos != NULL)
-            return pos - start + 1;
+            return (pos - start)/size + 1;
         else {
             *consumed = len;
             return -1;
@@ -1632,63 +1610,66 @@
         /* Universal newline search. Find any of \r, \r\n, \n
          * The decoder ensures that \r\n are not split in two pieces
          */
-        Py_UNICODE *s = start;
+        char *s = start;
         for (;;) {
-            Py_UNICODE ch;
+            Py_UCS4 ch;
             /* Fast path for non-control chars. The loop always ends
                since the Py_UNICODE storage is NUL-terminated. */
-            while (*s > '\r')
-                s++;
+            while (PyUnicode_READ(kind, s, 0) > '\r')
+                s += size;
             if (s >= end) {
                 *consumed = len;
                 return -1;
             }
-            ch = *s++;
+            ch = PyUnicode_READ(kind, s, 0);
+            s += size;
             if (ch == '\n')
-                return s - start;
+                return (s - start)/size;
             if (ch == '\r') {
-                if (*s == '\n')
-                    return s - start + 1;
+                if (PyUnicode_READ(kind, s, 0) == '\n')
+                    return (s - start)/size + 1;
                 else
-                    return s - start;
+                    return (s - start)/size;
             }
         }
     }
     else {
         /* Non-universal mode. */
-        Py_ssize_t readnl_len = PyUnicode_GET_SIZE(readnl);
-        Py_UNICODE *nl = PyUnicode_AS_UNICODE(readnl);
+        Py_ssize_t readnl_len = PyUnicode_GET_LENGTH(readnl);
+        char *nl = PyUnicode_DATA(readnl);
+        /* Assume that readnl is an ASCII character. */
+        assert(PyUnicode_KIND(readnl) == PyUnicode_1BYTE_KIND);
         if (readnl_len == 1) {
-            Py_UNICODE *pos = find_control_char(start, end, nl[0]);
+            char *pos = find_control_char(kind, start, end, nl[0]);
             if (pos != NULL)
-                return pos - start + 1;
+                return (pos - start)/size + 1;
             *consumed = len;
             return -1;
         }
         else {
-            Py_UNICODE *s = start;
-            Py_UNICODE *e = end - readnl_len + 1;
-            Py_UNICODE *pos;
+            char *s = start;
+            char *e = end - (readnl_len - 1)*size;
+            char *pos;
             if (e < s)
                 e = s;
             while (s < e) {
                 Py_ssize_t i;
-                Py_UNICODE *pos = find_control_char(s, end, nl[0]);
+                char *pos = find_control_char(kind, s, end, nl[0]);
                 if (pos == NULL || pos >= e)
                     break;
                 for (i = 1; i < readnl_len; i++) {
-                    if (pos[i] != nl[i])
+                    if (PyUnicode_READ(kind, pos, i) != nl[i])
                         break;
                 }
                 if (i == readnl_len)
-                    return pos - start + readnl_len;
-                s = pos + 1;
+                    return (pos - start)/size + readnl_len;
+                s = pos + size;
             }
-            pos = find_control_char(e, end, nl[0]);
+            pos = find_control_char(kind, e, end, nl[0]);
             if (pos == NULL)
                 *consumed = len;
             else
-                *consumed = pos - start;
+                *consumed = (pos - start)/size;
             return -1;
         }
     }
@@ -1709,14 +1690,15 @@
     chunked = 0;
 
     while (1) {
-        Py_UNICODE *ptr;
+        char *ptr;
         Py_ssize_t line_len;
+        int kind;
         Py_ssize_t consumed = 0;
 
         /* First, get some data if necessary */
         res = 1;
         while (!self->decoded_chars ||
-               !PyUnicode_GET_SIZE(self->decoded_chars)) {
+               !PyUnicode_GET_LENGTH(self->decoded_chars)) {
             res = textiowrapper_read_chunk(self);
             if (res < 0)
                 goto error;
@@ -1741,18 +1723,24 @@
             assert(self->decoded_chars_used == 0);
             line = PyUnicode_Concat(remaining, self->decoded_chars);
             start = 0;
-            offset_to_buffer = PyUnicode_GET_SIZE(remaining);
+            offset_to_buffer = PyUnicode_GET_LENGTH(remaining);
             Py_CLEAR(remaining);
             if (line == NULL)
                 goto error;
+            if (PyUnicode_READY(line) == -1)
+                goto error;
         }
 
-        ptr = PyUnicode_AS_UNICODE(line);
-        line_len = PyUnicode_GET_SIZE(line);
+        ptr = PyUnicode_DATA(line);
+        line_len = PyUnicode_GET_LENGTH(line);
+        kind = PyUnicode_KIND(line);
 
         endpos = _PyIO_find_line_ending(
             self->readtranslate, self->readuniversal, self->readnl,
-            ptr + start, ptr + line_len, &consumed);
+            kind,
+            ptr + PyUnicode_KIND_SIZE(kind, start),
+            ptr + PyUnicode_KIND_SIZE(kind, line_len),
+            &consumed);
         if (endpos >= 0) {
             endpos += start;
             if (limit >= 0 && (endpos - start) + chunked >= limit)
@@ -1776,21 +1764,20 @@
                 if (chunks == NULL)
                     goto error;
             }
-            s = PyUnicode_FromUnicode(ptr + start, endpos - start);
+            s = PyUnicode_Substring(line, start, endpos);
             if (s == NULL)
                 goto error;
             if (PyList_Append(chunks, s) < 0) {
                 Py_DECREF(s);
                 goto error;
             }
-            chunked += PyUnicode_GET_SIZE(s);
+            chunked += PyUnicode_GET_LENGTH(s);
             Py_DECREF(s);
         }
         /* There may be some remaining bytes we'll have to prepend to the
            next chunk of data */
         if (endpos < line_len) {
-            remaining = PyUnicode_FromUnicode(
-                    ptr + endpos, line_len - endpos);
+            remaining = PyUnicode_Substring(line, endpos, line_len);
             if (remaining == NULL)
                 goto error;
         }
@@ -1802,19 +1789,12 @@
     if (line != NULL) {
         /* Our line ends in the current buffer */
         self->decoded_chars_used = endpos - offset_to_buffer;
-        if (start > 0 || endpos < PyUnicode_GET_SIZE(line)) {
-            if (start == 0 && Py_REFCNT(line) == 1) {
-                if (PyUnicode_Resize(&line, endpos) < 0)
-                    goto error;
-            }
-            else {
-                PyObject *s = PyUnicode_FromUnicode(
-                        PyUnicode_AS_UNICODE(line) + start, endpos - start);
-                Py_CLEAR(line);
-                if (s == NULL)
-                    goto error;
-                line = s;
-            }
+        if (start > 0 || endpos < PyUnicode_GET_LENGTH(line)) {
+            PyObject *s = PyUnicode_Substring(line, start, endpos);
+            Py_CLEAR(line);
+            if (s == NULL)
+                goto error;
+            line = s;
         }
     }
     if (remaining != NULL) {
@@ -1828,16 +1808,20 @@
         Py_CLEAR(remaining);
     }
     if (chunks != NULL) {
-        if (line != NULL && PyList_Append(chunks, line) < 0)
-            goto error;
-        Py_CLEAR(line);
+        if (line != NULL) {
+            if (PyList_Append(chunks, line) < 0)
+                goto error;
+            Py_DECREF(line);
+        }
         line = PyUnicode_Join(_PyIO_empty_str, chunks);
         if (line == NULL)
             goto error;
-        Py_DECREF(chunks);
+        Py_CLEAR(chunks);
     }
-    if (line == NULL)
-        line = PyUnicode_FromStringAndSize(NULL, 0);
+    if (line == NULL) {
+        Py_INCREF(_PyIO_empty_str);
+        line = _PyIO_empty_str;
+    }
 
     return line;
 
@@ -2128,6 +2112,10 @@
 
         if (decoded == NULL)
             goto fail;
+        if (PyUnicode_READY(decoded) == -1) {
+            Py_DECREF(decoded);
+            goto fail;
+        }
 
         textiowrapper_set_decoded_chars(self, decoded);
 
@@ -2250,7 +2238,7 @@
         if (_decoded == NULL) \
             goto fail; \
         assert (PyUnicode_Check(_decoded)); \
-        res = PyUnicode_GET_SIZE(_decoded); \
+        res = PyUnicode_GET_LENGTH(_decoded); \
         Py_DECREF(_decoded); \
     } while (0)
 
@@ -2333,7 +2321,7 @@
         if (decoded == NULL)
             goto fail;
         assert (PyUnicode_Check(decoded));
-        chars_decoded += PyUnicode_GET_SIZE(decoded);
+        chars_decoded += PyUnicode_GET_LENGTH(decoded);
         Py_DECREF(decoded);
         cookie.need_eof = 1;
 
@@ -2559,10 +2547,10 @@
         }
     }
 
-    if (line == NULL)
+    if (line == NULL || PyUnicode_READY(line) == -1)
         return NULL;
 
-    if (PyUnicode_GET_SIZE(line) == 0) {
+    if (PyUnicode_GET_LENGTH(line) == 0) {
         /* Reached EOF or would have blocked */
         Py_DECREF(line);
         Py_CLEAR(self->snapshot);
diff --git a/Modules/_json.c b/Modules/_json.c
--- a/Modules/_json.c
+++ b/Modules/_json.c
@@ -238,13 +238,6 @@
 #define S_CHAR(c) (c >= ' ' && c <= '~' && c != '\\' && c != '"')
 #define IS_WHITESPACE(c) (((c) == ' ') || ((c) == '\t') || ((c) == '\n') || ((c) == '\r'))
 
-#define MIN_EXPANSION 6
-#ifdef Py_UNICODE_WIDE
-#define MAX_EXPANSION (2 * MIN_EXPANSION)
-#else
-#define MAX_EXPANSION MIN_EXPANSION
-#endif
-
 static int
 _convertPyInt_AsSsize_t(PyObject *o, Py_ssize_t *size_ptr)
 {
@@ -263,7 +256,7 @@
 }
 
 static Py_ssize_t
-ascii_escape_unichar(Py_UNICODE c, Py_UNICODE *output, Py_ssize_t chars)
+ascii_escape_unichar(Py_UCS4 c, unsigned char *output, Py_ssize_t chars)
 {
     /* Escape unicode code point c to ASCII escape sequences
     in char *output. output must have at least 12 bytes unused to
@@ -278,10 +271,9 @@
         case '\r': output[chars++] = 'r'; break;
         case '\t': output[chars++] = 't'; break;
         default:
-#ifdef Py_UNICODE_WIDE
             if (c >= 0x10000) {
                 /* UTF-16 surrogate pair */
-                Py_UNICODE v = c - 0x10000;
+                Py_UCS4 v = c - 0x10000;
                 c = 0xd800 | ((v >> 10) & 0x3ff);
                 output[chars++] = 'u';
                 output[chars++] = "0123456789abcdef"[(c >> 12) & 0xf];
@@ -291,7 +283,6 @@
                 c = 0xdc00 | (v & 0x3ff);
                 output[chars++] = '\\';
             }
-#endif
             output[chars++] = 'u';
             output[chars++] = "0123456789abcdef"[(c >> 12) & 0xf];
             output[chars++] = "0123456789abcdef"[(c >>  8) & 0xf];
@@ -308,54 +299,52 @@
     Py_ssize_t i;
     Py_ssize_t input_chars;
     Py_ssize_t output_size;
-    Py_ssize_t max_output_size;
     Py_ssize_t chars;
     PyObject *rval;
-    Py_UNICODE *output;
-    Py_UNICODE *input_unicode;
+    void *input;
+    unsigned char *output;
+    int kind;
 
-    input_chars = PyUnicode_GET_SIZE(pystr);
-    input_unicode = PyUnicode_AS_UNICODE(pystr);
+    if (PyUnicode_READY(pystr) == -1)
+        return NULL;
 
-    /* One char input can be up to 6 chars output, estimate 4 of these */
-    output_size = 2 + (MIN_EXPANSION * 4) + input_chars;
-    max_output_size = 2 + (input_chars * MAX_EXPANSION);
-    rval = PyUnicode_FromStringAndSize(NULL, output_size);
+    input_chars = PyUnicode_GET_LENGTH(pystr);
+    input = PyUnicode_DATA(pystr);
+    kind = PyUnicode_KIND(pystr);
+
+    /* Compute the output size */
+    for (i = 0, output_size = 2; i < input_chars; i++) {
+        Py_UCS4 c = PyUnicode_READ(kind, input, i);
+        if (S_CHAR(c))
+            output_size++;
+        else {
+            switch(c) {
+            case '\\': case '"': case '\b': case '\f': 
+            case '\n': case '\r': case '\t':
+                output_size += 2; break;
+            default:
+                output_size += c >= 0x10000 ? 12 : 6;
+            }
+        }
+    }
+
+    rval = PyUnicode_New(output_size, 127);
     if (rval == NULL) {
         return NULL;
     }
-    output = PyUnicode_AS_UNICODE(rval);
+    output = PyUnicode_1BYTE_DATA(rval);
     chars = 0;
     output[chars++] = '"';
     for (i = 0; i < input_chars; i++) {
-        Py_UNICODE c = input_unicode[i];
+        Py_UCS4 c = PyUnicode_READ(kind, input, i);
         if (S_CHAR(c)) {
             output[chars++] = c;
         }
         else {
             chars = ascii_escape_unichar(c, output, chars);
         }
-        if (output_size - chars < (1 + MAX_EXPANSION)) {
-            /* There's more than four, so let's resize by a lot */
-            Py_ssize_t new_output_size = output_size * 2;
-            /* This is an upper bound */
-            if (new_output_size > max_output_size) {
-                new_output_size = max_output_size;
-            }
-            /* Make sure that the output size changed before resizing */
-            if (new_output_size != output_size) {
-                output_size = new_output_size;
-                if (PyUnicode_Resize(&rval, output_size) == -1) {
-                    return NULL;
-                }
-                output = PyUnicode_AS_UNICODE(rval);
-            }
-        }
     }
     output[chars++] = '"';
-    if (PyUnicode_Resize(&rval, chars) == -1) {
-        return NULL;
-    }
     return rval;
 }
 
@@ -436,22 +425,30 @@
     Return value is a new PyUnicode
     */
     PyObject *rval = NULL;
-    Py_ssize_t len = PyUnicode_GET_SIZE(pystr);
+    Py_ssize_t len;
     Py_ssize_t begin = end - 1;
     Py_ssize_t next /* = begin */;
-    const Py_UNICODE *buf = PyUnicode_AS_UNICODE(pystr);
+    const void *buf;
+    int kind;
     PyObject *chunks = NULL;
     PyObject *chunk = NULL;
 
+    if (PyUnicode_READY(pystr) == -1)
+        return 0;
+
+    len = PyUnicode_GET_LENGTH(pystr);
+    buf = PyUnicode_DATA(pystr);
+    kind = PyUnicode_KIND(pystr);
+
     if (end < 0 || len <= end) {
         PyErr_SetString(PyExc_ValueError, "end is out of bounds");
         goto bail;
     }
     while (1) {
         /* Find the end of the string or the next escape */
-        Py_UNICODE c = 0;
+        Py_UCS4 c = 0;
         for (next = end; next < len; next++) {
-            c = buf[next];
+            c = PyUnicode_READ(kind, buf, next);
             if (c == '"' || c == '\\') {
                 break;
             }
@@ -467,7 +464,10 @@
         /* Pick up this chunk if it's not zero length */
         if (next != end) {
             APPEND_OLD_CHUNK
-            chunk = PyUnicode_FromUnicode(&buf[end], next - end);
+                chunk = PyUnicode_FromKindAndData(
+                    kind,
+                    (char*)buf + PyUnicode_KIND_SIZE(kind, end),
+                    next - end);
             if (chunk == NULL) {
                 goto bail;
             }
@@ -481,7 +481,7 @@
             raise_errmsg("Unterminated string starting at", pystr, begin);
             goto bail;
         }
-        c = buf[next];
+        c = PyUnicode_READ(kind, buf, next);
         if (c != 'u') {
             /* Non-unicode backslash escapes */
             end = next + 1;
@@ -511,7 +511,7 @@
             }
             /* Decode 4 hex digits */
             for (; next < end; next++) {
-                Py_UNICODE digit = buf[next];
+                Py_UCS4 digit = PyUnicode_READ(kind, buf, next);
                 c <<= 4;
                 switch (digit) {
                     case '0': case '1': case '2': case '3': case '4':
@@ -528,22 +528,22 @@
                         goto bail;
                 }
             }
-#ifdef Py_UNICODE_WIDE
             /* Surrogate pair */
             if ((c & 0xfc00) == 0xd800) {
-                Py_UNICODE c2 = 0;
+                Py_UCS4 c2 = 0;
                 if (end + 6 >= len) {
                     raise_errmsg("Unpaired high surrogate", pystr, end - 5);
                     goto bail;
                 }
-                if (buf[next++] != '\\' || buf[next++] != 'u') {
+                if (PyUnicode_READ(kind, buf, next++) != '\\' || 
+                    PyUnicode_READ(kind, buf, next++) != 'u') {
                     raise_errmsg("Unpaired high surrogate", pystr, end - 5);
                     goto bail;
                 }
                 end += 6;
                 /* Decode 4 hex digits */
                 for (; next < end; next++) {
-                    Py_UNICODE digit = buf[next];
+                    Py_UCS4 digit = PyUnicode_READ(kind, buf, next);
                     c2 <<= 4;
                     switch (digit) {
                         case '0': case '1': case '2': case '3': case '4':
@@ -570,10 +570,9 @@
                 raise_errmsg("Unpaired low surrogate", pystr, end - 5);
                 goto bail;
             }
-#endif
         }
         APPEND_OLD_CHUNK
-        chunk = PyUnicode_FromUnicode(&c, 1);
+        chunk = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, &c, 1);
         if (chunk == NULL) {
             goto bail;
         }
@@ -711,8 +710,9 @@
 
     Returns a new PyObject (usually a dict, but object_hook can change that)
     */
-    Py_UNICODE *str = PyUnicode_AS_UNICODE(pystr);
-    Py_ssize_t end_idx = PyUnicode_GET_SIZE(pystr) - 1;
+    void *str;
+    int kind;
+    Py_ssize_t end_idx;
     PyObject *val = NULL;
     PyObject *rval = NULL;
     PyObject *key = NULL;
@@ -720,6 +720,13 @@
     int has_pairs_hook = (s->object_pairs_hook != Py_None);
     Py_ssize_t next_idx;
 
+    if (PyUnicode_READY(pystr) == -1)
+        return NULL;
+
+    str = PyUnicode_DATA(pystr);
+    kind = PyUnicode_KIND(pystr);
+    end_idx = PyUnicode_GET_LENGTH(pystr) - 1;
+
     if (has_pairs_hook)
         rval = PyList_New(0);
     else
@@ -728,15 +735,15 @@
         return NULL;
 
     /* skip whitespace after { */
-    while (idx <= end_idx && IS_WHITESPACE(str[idx])) idx++;
+    while (idx <= end_idx && IS_WHITESPACE(PyUnicode_READ(kind,str, idx))) idx++;
 
     /* only loop if the object is non-empty */
-    if (idx <= end_idx && str[idx] != '}') {
+    if (idx <= end_idx && PyUnicode_READ(kind, str, idx) != '}') {
         while (idx <= end_idx) {
             PyObject *memokey;
 
             /* read key */
-            if (str[idx] != '"') {
+            if (PyUnicode_READ(kind, str, idx) != '"') {
                 raise_errmsg("Expecting property name", pystr, idx);
                 goto bail;
             }
@@ -756,13 +763,13 @@
             idx = next_idx;
 
             /* skip whitespace between key and : delimiter, read :, skip whitespace */
-            while (idx <= end_idx && IS_WHITESPACE(str[idx])) idx++;
-            if (idx > end_idx || str[idx] != ':') {
+            while (idx <= end_idx && IS_WHITESPACE(PyUnicode_READ(kind, str, idx))) idx++;
+            if (idx > end_idx || PyUnicode_READ(kind, str, idx) != ':') {
                 raise_errmsg("Expecting : delimiter", pystr, idx);
                 goto bail;
             }
             idx++;
-            while (idx <= end_idx && IS_WHITESPACE(str[idx])) idx++;
+            while (idx <= end_idx && IS_WHITESPACE(PyUnicode_READ(kind, str, idx))) idx++;
 
             /* read any JSON term */
             val = scan_once_unicode(s, pystr, idx, &next_idx);
@@ -790,26 +797,26 @@
             idx = next_idx;
 
             /* skip whitespace before } or , */
-            while (idx <= end_idx && IS_WHITESPACE(str[idx])) idx++;
+            while (idx <= end_idx && IS_WHITESPACE(PyUnicode_READ(kind, str, idx))) idx++;
 
             /* bail if the object is closed or we didn't get the , delimiter */
             if (idx > end_idx) break;
-            if (str[idx] == '}') {
+            if (PyUnicode_READ(kind, str, idx) == '}') {
                 break;
             }
-            else if (str[idx] != ',') {
+            else if (PyUnicode_READ(kind, str, idx) != ',') {
                 raise_errmsg("Expecting , delimiter", pystr, idx);
                 goto bail;
             }
             idx++;
 
             /* skip whitespace after , delimiter */
-            while (idx <= end_idx && IS_WHITESPACE(str[idx])) idx++;
+            while (idx <= end_idx && IS_WHITESPACE(PyUnicode_READ(kind, str, idx))) idx++;
         }
     }
 
     /* verify that idx < end_idx, str[idx] should be '}' */
-    if (idx > end_idx || str[idx] != '}') {
+    if (idx > end_idx || PyUnicode_READ(kind, str, idx) != '}') {
         raise_errmsg("Expecting object", pystr, end_idx);
         goto bail;
     }
@@ -845,19 +852,27 @@
 
     Returns a new PyList
     */
-    Py_UNICODE *str = PyUnicode_AS_UNICODE(pystr);
-    Py_ssize_t end_idx = PyUnicode_GET_SIZE(pystr) - 1;
+    void *str;
+    int kind;
+    Py_ssize_t end_idx;
     PyObject *val = NULL;
     PyObject *rval = PyList_New(0);
     Py_ssize_t next_idx;
     if (rval == NULL)
         return NULL;
 
+    if (PyUnicode_READY(pystr) == -1)
+        return NULL;
+
+    str = PyUnicode_DATA(pystr);
+    kind = PyUnicode_KIND(pystr);
+    end_idx = PyUnicode_GET_LENGTH(pystr) - 1;
+
     /* skip whitespace after [ */
-    while (idx <= end_idx && IS_WHITESPACE(str[idx])) idx++;
+    while (idx <= end_idx && IS_WHITESPACE(PyUnicode_READ(kind, str, idx))) idx++;
 
     /* only loop if the array is non-empty */
-    if (idx <= end_idx && str[idx] != ']') {
+    if (idx <= end_idx && PyUnicode_READ(kind, str, idx) != ']') {
         while (idx <= end_idx) {
 
             /* read any JSON term  */
@@ -872,26 +887,26 @@
             idx = next_idx;
 
             /* skip whitespace between term and , */
-            while (idx <= end_idx && IS_WHITESPACE(str[idx])) idx++;
+            while (idx <= end_idx && IS_WHITESPACE(PyUnicode_READ(kind, str, idx))) idx++;
 
             /* bail if the array is closed or we didn't get the , delimiter */
             if (idx > end_idx) break;
-            if (str[idx] == ']') {
+            if (PyUnicode_READ(kind, str, idx) == ']') {
                 break;
             }
-            else if (str[idx] != ',') {
+            else if (PyUnicode_READ(kind, str, idx) != ',') {
                 raise_errmsg("Expecting , delimiter", pystr, idx);
                 goto bail;
             }
             idx++;
 
             /* skip whitespace after , */
-            while (idx <= end_idx && IS_WHITESPACE(str[idx])) idx++;
+            while (idx <= end_idx && IS_WHITESPACE(PyUnicode_READ(kind, str, idx))) idx++;
         }
     }
 
-    /* verify that idx < end_idx, str[idx] should be ']' */
-    if (idx > end_idx || str[idx] != ']') {
+    /* verify that idx < end_idx, PyUnicode_READ(kind, str, idx) should be ']' */
+    if (idx > end_idx || PyUnicode_READ(kind, str, idx) != ']') {
         raise_errmsg("Expecting object", pystr, end_idx);
         goto bail;
     }
@@ -940,16 +955,24 @@
         PyInt, PyLong, or PyFloat.
         May return other types if parse_int or parse_float are set
     */
-    Py_UNICODE *str = PyUnicode_AS_UNICODE(pystr);
-    Py_ssize_t end_idx = PyUnicode_GET_SIZE(pystr) - 1;
+    void *str;
+    int kind;
+    Py_ssize_t end_idx;
     Py_ssize_t idx = start;
     int is_float = 0;
     PyObject *rval;
     PyObject *numstr = NULL;
     PyObject *custom_func;
 
+    if (PyUnicode_READY(pystr) == -1)
+        return NULL;
+
+    str = PyUnicode_DATA(pystr);
+    kind = PyUnicode_KIND(pystr);
+    end_idx = PyUnicode_GET_LENGTH(pystr) - 1;
+
     /* read a sign if it's there, make sure it's not the end of the string */
-    if (str[idx] == '-') {
+    if (PyUnicode_READ(kind, str, idx) == '-') {
         idx++;
         if (idx > end_idx) {
             PyErr_SetNone(PyExc_StopIteration);
@@ -958,12 +981,12 @@
     }
 
     /* read as many integer digits as we find as long as it doesn't start with 0 */
-    if (str[idx] >= '1' && str[idx] <= '9') {
+    if (PyUnicode_READ(kind, str, idx) >= '1' && PyUnicode_READ(kind, str, idx) <= '9') {
         idx++;
-        while (idx <= end_idx && str[idx] >= '0' && str[idx] <= '9') idx++;
+        while (idx <= end_idx && PyUnicode_READ(kind, str, idx) >= '0' && PyUnicode_READ(kind, str, idx) <= '9') idx++;
     }
     /* if it starts with 0 we only expect one integer digit */
-    else if (str[idx] == '0') {
+    else if (PyUnicode_READ(kind, str, idx) == '0') {
         idx++;
     }
     /* no integer digits, error */
@@ -973,25 +996,25 @@
     }
 
     /* if the next char is '.' followed by a digit then read all float digits */
-    if (idx < end_idx && str[idx] == '.' && str[idx + 1] >= '0' && str[idx + 1] <= '9') {
+    if (idx < end_idx && PyUnicode_READ(kind, str, idx) == '.' && PyUnicode_READ(kind, str, idx + 1) >= '0' && PyUnicode_READ(kind, str, idx + 1) <= '9') {
         is_float = 1;
         idx += 2;
-        while (idx <= end_idx && str[idx] >= '0' && str[idx] <= '9') idx++;
+        while (idx <= end_idx && PyUnicode_READ(kind, str, idx) >= '0' && PyUnicode_READ(kind, str, idx) <= '9') idx++;
     }
 
     /* if the next char is 'e' or 'E' then maybe read the exponent (or backtrack) */
-    if (idx < end_idx && (str[idx] == 'e' || str[idx] == 'E')) {
+    if (idx < end_idx && (PyUnicode_READ(kind, str, idx) == 'e' || PyUnicode_READ(kind, str, idx) == 'E')) {
         Py_ssize_t e_start = idx;
         idx++;
 
         /* read an exponent sign if present */
-        if (idx < end_idx && (str[idx] == '-' || str[idx] == '+')) idx++;
+        if (idx < end_idx && (PyUnicode_READ(kind, str, idx) == '-' || PyUnicode_READ(kind, str, idx) == '+')) idx++;
 
         /* read all digits */
-        while (idx <= end_idx && str[idx] >= '0' && str[idx] <= '9') idx++;
+        while (idx <= end_idx && PyUnicode_READ(kind, str, idx) >= '0' && PyUnicode_READ(kind, str, idx) <= '9') idx++;
 
         /* if we got a digit, then parse as float. if not, backtrack */
-        if (str[idx - 1] >= '0' && str[idx - 1] <= '9') {
+        if (PyUnicode_READ(kind, str, idx - 1) >= '0' && PyUnicode_READ(kind, str, idx - 1) <= '9') {
             is_float = 1;
         }
         else {
@@ -1008,7 +1031,9 @@
 
     if (custom_func) {
         /* copy the section we determined to be a number */
-        numstr = PyUnicode_FromUnicode(&str[start], idx - start);
+        numstr = PyUnicode_FromKindAndData(kind,
+                                           (char*)str + PyUnicode_KIND_SIZE(kind, start),
+                                           idx - start);
         if (numstr == NULL)
             return NULL;
         rval = PyObject_CallFunctionObjArgs(custom_func, numstr, NULL);
@@ -1024,7 +1049,7 @@
             return NULL;
         buf = PyBytes_AS_STRING(numstr);
         for (i = 0; i < n; i++) {
-            buf[i] = (char) str[i + start];
+            buf[i] = (char) PyUnicode_READ(kind, str, i + start);
         }
         if (is_float)
             rval = PyFloat_FromString(numstr);
@@ -1047,13 +1072,23 @@
     Returns a new PyObject representation of the term.
     */
     PyObject *res;
-    Py_UNICODE *str = PyUnicode_AS_UNICODE(pystr);
-    Py_ssize_t length = PyUnicode_GET_SIZE(pystr);
+    void *str;
+    int kind;
+    Py_ssize_t length;
+
+    if (PyUnicode_READY(pystr) == -1)
+        return NULL;
+
+    str = PyUnicode_DATA(pystr);
+    kind = PyUnicode_KIND(pystr);
+    length = PyUnicode_GET_LENGTH(pystr);
+
     if (idx >= length) {
         PyErr_SetNone(PyExc_StopIteration);
         return NULL;
     }
-    switch (str[idx]) {
+
+    switch (PyUnicode_READ(kind, str, idx)) {
         case '"':
             /* string */
             return scanstring_unicode(pystr, idx + 1,
@@ -1077,7 +1112,7 @@
             return res;
         case 'n':
             /* null */
-            if ((idx + 3 < length) && str[idx + 1] == 'u' && str[idx + 2] == 'l' && str[idx + 3] == 'l') {
+            if ((idx + 3 < length) && PyUnicode_READ(kind, str, idx + 1) == 'u' && PyUnicode_READ(kind, str, idx + 2) == 'l' && PyUnicode_READ(kind, str, idx + 3) == 'l') {
                 Py_INCREF(Py_None);
                 *next_idx_ptr = idx + 4;
                 return Py_None;
@@ -1085,7 +1120,7 @@
             break;
         case 't':
             /* true */
-            if ((idx + 3 < length) && str[idx + 1] == 'r' && str[idx + 2] == 'u' && str[idx + 3] == 'e') {
+            if ((idx + 3 < length) && PyUnicode_READ(kind, str, idx + 1) == 'r' && PyUnicode_READ(kind, str, idx + 2) == 'u' && PyUnicode_READ(kind, str, idx + 3) == 'e') {
                 Py_INCREF(Py_True);
                 *next_idx_ptr = idx + 4;
                 return Py_True;
@@ -1093,7 +1128,10 @@
             break;
         case 'f':
             /* false */
-            if ((idx + 4 < length) && str[idx + 1] == 'a' && str[idx + 2] == 'l' && str[idx + 3] == 's' && str[idx + 4] == 'e') {
+            if ((idx + 4 < length) && PyUnicode_READ(kind, str, idx + 1) == 'a' && 
+                PyUnicode_READ(kind, str, idx + 2) == 'l' && 
+                PyUnicode_READ(kind, str, idx + 3) == 's' && 
+                PyUnicode_READ(kind, str, idx + 4) == 'e') {
                 Py_INCREF(Py_False);
                 *next_idx_ptr = idx + 5;
                 return Py_False;
@@ -1101,19 +1139,33 @@
             break;
         case 'N':
             /* NaN */
-            if ((idx + 2 < length) && str[idx + 1] == 'a' && str[idx + 2] == 'N') {
+            if ((idx + 2 < length) && PyUnicode_READ(kind, str, idx + 1) == 'a' && 
+                PyUnicode_READ(kind, str, idx + 2) == 'N') {
                 return _parse_constant(s, "NaN", idx, next_idx_ptr);
             }
             break;
         case 'I':
             /* Infinity */
-            if ((idx + 7 < length) && str[idx + 1] == 'n' && str[idx + 2] == 'f' && str[idx + 3] == 'i' && str[idx + 4] == 'n' && str[idx + 5] == 'i' && str[idx + 6] == 't' && str[idx + 7] == 'y') {
+            if ((idx + 7 < length) && PyUnicode_READ(kind, str, idx + 1) == 'n' && 
+                PyUnicode_READ(kind, str, idx + 2) == 'f' && 
+                PyUnicode_READ(kind, str, idx + 3) == 'i' && 
+                PyUnicode_READ(kind, str, idx + 4) == 'n' &&
+                PyUnicode_READ(kind, str, idx + 5) == 'i' && 
+                PyUnicode_READ(kind, str, idx + 6) == 't' && 
+                PyUnicode_READ(kind, str, idx + 7) == 'y') {
                 return _parse_constant(s, "Infinity", idx, next_idx_ptr);
             }
             break;
         case '-':
             /* -Infinity */
-            if ((idx + 8 < length) && str[idx + 1] == 'I' && str[idx + 2] == 'n' && str[idx + 3] == 'f' && str[idx + 4] == 'i' && str[idx + 5] == 'n' && str[idx + 6] == 'i' && str[idx + 7] == 't' && str[idx + 8] == 'y') {
+            if ((idx + 8 < length) && PyUnicode_READ(kind, str, idx + 1) == 'I' && 
+                PyUnicode_READ(kind, str, idx + 2) == 'n' &&
+                PyUnicode_READ(kind, str, idx + 3) == 'f' &&
+                PyUnicode_READ(kind, str, idx + 4) == 'i' && 
+                PyUnicode_READ(kind, str, idx + 5) == 'n' &&
+                PyUnicode_READ(kind, str, idx + 6) == 'i' && 
+                PyUnicode_READ(kind, str, idx + 7) == 't' && 
+                PyUnicode_READ(kind, str, idx + 8) == 'y') {
                 return _parse_constant(s, "-Infinity", idx, next_idx_ptr);
             }
             break;
diff --git a/Modules/_pickle.c b/Modules/_pickle.c
--- a/Modules/_pickle.c
+++ b/Modules/_pickle.c
@@ -1867,9 +1867,7 @@
     if (self->bin) {
         char pdata[5];
 
-        encoded = PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(obj),
-                                    PyUnicode_GET_SIZE(obj),
-                                    "surrogatepass");
+        encoded = PyUnicode_AsEncodedString(obj, "utf-8", "surrogatepass");
         if (encoded == NULL)
             goto error;
 
diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c
--- a/Modules/_sqlite/connection.c
+++ b/Modules/_sqlite/connection.c
@@ -1436,10 +1436,11 @@
     PyObject* uppercase_name = 0;
     PyObject* name;
     PyObject* retval;
-    Py_UNICODE* chk;
     Py_ssize_t i, len;
     char *uppercase_name_str;
     int rc;
+    unsigned int kind;
+    void *data;
 
     if (!pysqlite_check_thread(self) || !pysqlite_check_connection(self)) {
         goto finally;
@@ -1454,12 +1455,16 @@
         goto finally;
     }
 
-    len = PyUnicode_GET_SIZE(uppercase_name);
-    chk = PyUnicode_AS_UNICODE(uppercase_name);
-    for (i=0; i<len; i++, chk++) {
-        if ((*chk >= '0' && *chk <= '9')
-         || (*chk >= 'A' && *chk <= 'Z')
-         || (*chk == '_'))
+    if (PyUnicode_READY(uppercase_name))
+        goto finally;
+    len = PyUnicode_GET_LENGTH(uppercase_name);
+    kind = PyUnicode_KIND(uppercase_name);
+    data = PyUnicode_DATA(uppercase_name);
+    for (i=0; i<len; i++) {
+        Py_UCS4 ch = PyUnicode_READ(kind, data, i);
+        if ((ch >= '0' && ch <= '9')
+         || (ch >= 'A' && ch <= 'Z')
+         || (ch == '_'))
         {
             continue;
         } else {
diff --git a/Modules/_sre.c b/Modules/_sre.c
--- a/Modules/_sre.c
+++ b/Modules/_sre.c
@@ -163,8 +163,6 @@
 
 /* unicode-specific character predicates */
 
-#if defined(HAVE_UNICODE)
-
 #define SRE_UNI_IS_DIGIT(ch) Py_UNICODE_ISDECIMAL((Py_UNICODE)(ch))
 #define SRE_UNI_IS_SPACE(ch) Py_UNICODE_ISSPACE((Py_UNICODE)(ch))
 #define SRE_UNI_IS_LINEBREAK(ch) Py_UNICODE_ISLINEBREAK((Py_UNICODE)(ch))
@@ -176,8 +174,6 @@
     return (unsigned int) Py_UNICODE_TOLOWER((Py_UNICODE)(ch));
 }
 
-#endif
-
 LOCAL(int)
 sre_category(SRE_CODE category, unsigned int ch)
 {
@@ -205,7 +201,6 @@
     case SRE_CATEGORY_LOC_NOT_WORD:
         return !SRE_LOC_IS_WORD(ch);
 
-#if defined(HAVE_UNICODE)
     case SRE_CATEGORY_UNI_DIGIT:
         return SRE_UNI_IS_DIGIT(ch);
     case SRE_CATEGORY_UNI_NOT_DIGIT:
@@ -222,24 +217,6 @@
         return SRE_UNI_IS_LINEBREAK(ch);
     case SRE_CATEGORY_UNI_NOT_LINEBREAK:
         return !SRE_UNI_IS_LINEBREAK(ch);
-#else
-    case SRE_CATEGORY_UNI_DIGIT:
-        return SRE_IS_DIGIT(ch);
-    case SRE_CATEGORY_UNI_NOT_DIGIT:
-        return !SRE_IS_DIGIT(ch);
-    case SRE_CATEGORY_UNI_SPACE:
-        return SRE_IS_SPACE(ch);
-    case SRE_CATEGORY_UNI_NOT_SPACE:
-        return !SRE_IS_SPACE(ch);
-    case SRE_CATEGORY_UNI_WORD:
-        return SRE_LOC_IS_WORD(ch);
-    case SRE_CATEGORY_UNI_NOT_WORD:
-        return !SRE_LOC_IS_WORD(ch);
-    case SRE_CATEGORY_UNI_LINEBREAK:
-        return SRE_IS_LINEBREAK(ch);
-    case SRE_CATEGORY_UNI_NOT_LINEBREAK:
-        return !SRE_IS_LINEBREAK(ch);
-#endif
     }
     return 0;
 }
@@ -280,6 +257,7 @@
 /* generate 8-bit version */
 
 #define SRE_CHAR unsigned char
+#define SRE_CHARGET(state, buf, index) ((unsigned char*)buf)[index]
 #define SRE_AT sre_at
 #define SRE_COUNT sre_count
 #define SRE_CHARSET sre_charset
@@ -287,15 +265,11 @@
 #define SRE_MATCH sre_match
 #define SRE_MATCH_CONTEXT sre_match_context
 #define SRE_SEARCH sre_search
-#define SRE_LITERAL_TEMPLATE sre_literal_template
-
-#if defined(HAVE_UNICODE)
 
 #define SRE_RECURSIVE
 #include "_sre.c"
 #undef SRE_RECURSIVE
 
-#undef SRE_LITERAL_TEMPLATE
 #undef SRE_SEARCH
 #undef SRE_MATCH
 #undef SRE_MATCH_CONTEXT
@@ -304,10 +278,15 @@
 #undef SRE_COUNT
 #undef SRE_AT
 #undef SRE_CHAR
-
-/* generate 16-bit unicode version */
-
-#define SRE_CHAR Py_UNICODE
+#undef SRE_CHARGET
+
+/* generate 8/16/32-bit unicode version */
+
+#define SRE_CHAR void
+#define SRE_CHARGET(state, buf, index) \
+    ((state->charsize==1) ? ((Py_UCS1*)buf)[index] : \
+     (state->charsize==2) ? ((Py_UCS2*)buf)[index] : \
+     ((Py_UCS4*)buf)[index])
 #define SRE_AT sre_uat
 #define SRE_COUNT sre_ucount
 #define SRE_CHARSET sre_ucharset
@@ -315,8 +294,6 @@
 #define SRE_MATCH sre_umatch
 #define SRE_MATCH_CONTEXT sre_umatch_context
 #define SRE_SEARCH sre_usearch
-#define SRE_LITERAL_TEMPLATE sre_uliteral_template
-#endif
 
 #endif /* SRE_RECURSIVE */
 
@@ -327,7 +304,7 @@
    settings */
 
 LOCAL(int)
-SRE_AT(SRE_STATE* state, SRE_CHAR* ptr, SRE_CODE at)
+SRE_AT(SRE_STATE* state, char* ptr, SRE_CODE at)
 {
     /* check if pointer is at given position */
 
@@ -341,16 +318,16 @@
 
     case SRE_AT_BEGINNING_LINE:
         return ((void*) ptr == state->beginning ||
-                SRE_IS_LINEBREAK((int) ptr[-1]));
+                SRE_IS_LINEBREAK((int) SRE_CHARGET(state, ptr, -1)));
 
     case SRE_AT_END:
-        return (((void*) (ptr+1) == state->end &&
-                 SRE_IS_LINEBREAK((int) ptr[0])) ||
+        return (((void*) (ptr+state->charsize) == state->end &&
+                 SRE_IS_LINEBREAK((int) SRE_CHARGET(state, ptr, 0))) ||
                 ((void*) ptr == state->end));
 
     case SRE_AT_END_LINE:
         return ((void*) ptr == state->end ||
-                SRE_IS_LINEBREAK((int) ptr[0]));
+                SRE_IS_LINEBREAK((int) SRE_CHARGET(state, ptr, 0)));
 
     case SRE_AT_END_STRING:
         return ((void*) ptr == state->end);
@@ -359,57 +336,55 @@
         if (state->beginning == state->end)
             return 0;
         thatp = ((void*) ptr > state->beginning) ?
-            SRE_IS_WORD((int) ptr[-1]) : 0;
+            SRE_IS_WORD((int) SRE_CHARGET(state, ptr, -1)) : 0;
         thisp = ((void*) ptr < state->end) ?
-            SRE_IS_WORD((int) ptr[0]) : 0;
+            SRE_IS_WORD((int) SRE_CHARGET(state, ptr, 0)) : 0;
         return thisp != thatp;
 
     case SRE_AT_NON_BOUNDARY:
         if (state->beginning == state->end)
             return 0;
         thatp = ((void*) ptr > state->beginning) ?
-            SRE_IS_WORD((int) ptr[-1]) : 0;
+            SRE_IS_WORD((int) SRE_CHARGET(state, ptr, -1)) : 0;
         thisp = ((void*) ptr < state->end) ?
-            SRE_IS_WORD((int) ptr[0]) : 0;
+            SRE_IS_WORD((int) SRE_CHARGET(state, ptr, 0)) : 0;
         return thisp == thatp;
 
     case SRE_AT_LOC_BOUNDARY:
         if (state->beginning == state->end)
             return 0;
         thatp = ((void*) ptr > state->beginning) ?
-            SRE_LOC_IS_WORD((int) ptr[-1]) : 0;
+            SRE_LOC_IS_WORD((int) SRE_CHARGET(state, ptr, -1)) : 0;
         thisp = ((void*) ptr < state->end) ?
-            SRE_LOC_IS_WORD((int) ptr[0]) : 0;
+            SRE_LOC_IS_WORD((int) SRE_CHARGET(state, ptr, 0)) : 0;
         return thisp != thatp;
 
     case SRE_AT_LOC_NON_BOUNDARY:
         if (state->beginning == state->end)
             return 0;
         thatp = ((void*) ptr > state->beginning) ?
-            SRE_LOC_IS_WORD((int) ptr[-1]) : 0;
+            SRE_LOC_IS_WORD((int) SRE_CHARGET(state, ptr, -1)) : 0;
         thisp = ((void*) ptr < state->end) ?
-            SRE_LOC_IS_WORD((int) ptr[0]) : 0;
+            SRE_LOC_IS_WORD((int) SRE_CHARGET(state, ptr, 0)) : 0;
         return thisp == thatp;
 
-#if defined(HAVE_UNICODE)
     case SRE_AT_UNI_BOUNDARY:
         if (state->beginning == state->end)
             return 0;
         thatp = ((void*) ptr > state->beginning) ?
-            SRE_UNI_IS_WORD((int) ptr[-1]) : 0;
+            SRE_UNI_IS_WORD((int) SRE_CHARGET(state, ptr, -1)) : 0;
         thisp = ((void*) ptr < state->end) ?
-            SRE_UNI_IS_WORD((int) ptr[0]) : 0;
+            SRE_UNI_IS_WORD((int) SRE_CHARGET(state, ptr, 0)) : 0;
         return thisp != thatp;
 
     case SRE_AT_UNI_NON_BOUNDARY:
         if (state->beginning == state->end)
             return 0;
         thatp = ((void*) ptr > state->beginning) ?
-            SRE_UNI_IS_WORD((int) ptr[-1]) : 0;
+            SRE_UNI_IS_WORD((int) SRE_CHARGET(state, ptr, -1)) : 0;
         thisp = ((void*) ptr < state->end) ?
-            SRE_UNI_IS_WORD((int) ptr[0]) : 0;
+            SRE_UNI_IS_WORD((int) SRE_CHARGET(state, ptr, 0)) : 0;
         return thisp == thatp;
-#endif
 
     }
 
@@ -476,7 +451,7 @@
             count = *(set++);
 
             if (sizeof(SRE_CODE) == 2) {
-                block = ((unsigned char*)set)[ch >> 8];
+                block = ((char*)set)[ch >> 8];
                 set += 128;
                 if (set[block*16 + ((ch & 255)>>4)] & (1 << (ch & 15)))
                     return ok;
@@ -486,7 +461,7 @@
                 /* !(c & ~N) == (c < N+1) for any unsigned c, this avoids
                  * warnings when c's type supports only numbers < N+1 */
                 if (!(ch & ~65535))
-                    block = ((unsigned char*)set)[ch >> 8];
+                    block = ((char*)set)[ch >> 8];
                 else
                     block = -1;
                 set += 64;
@@ -512,28 +487,29 @@
 SRE_COUNT(SRE_STATE* state, SRE_CODE* pattern, Py_ssize_t maxcount)
 {
     SRE_CODE chr;
-    SRE_CHAR* ptr = (SRE_CHAR *)state->ptr;
-    SRE_CHAR* end = (SRE_CHAR *)state->end;
+    char* ptr = (char *)state->ptr;
+    char* end = (char *)state->end;
     Py_ssize_t i;
 
     /* adjust end */
     if (maxcount < end - ptr && maxcount != 65535)
-        end = ptr + maxcount;
+        end = ptr + maxcount*state->charsize;
 
     switch (pattern[0]) {
 
     case SRE_OP_IN:
         /* repeated set */
         TRACE(("|%p|%p|COUNT IN\n", pattern, ptr));
-        while (ptr < end && SRE_CHARSET(pattern + 2, *ptr))
-            ptr++;
+        while (ptr < end && 
+               SRE_CHARSET(pattern + 2, SRE_CHARGET(state, ptr, 0)))
+            ptr += state->charsize;
         break;
 
     case SRE_OP_ANY:
         /* repeated dot wildcard. */
         TRACE(("|%p|%p|COUNT ANY\n", pattern, ptr));
-        while (ptr < end && !SRE_IS_LINEBREAK(*ptr))
-            ptr++;
+        while (ptr < end && !SRE_IS_LINEBREAK(SRE_CHARGET(state, ptr, 0)))
+            ptr += state->charsize;
         break;
 
     case SRE_OP_ANY_ALL:
@@ -547,38 +523,38 @@
         /* repeated literal */
         chr = pattern[1];
         TRACE(("|%p|%p|COUNT LITERAL %d\n", pattern, ptr, chr));
-        while (ptr < end && (SRE_CODE) *ptr == chr)
-            ptr++;
+        while (ptr < end && (SRE_CODE) SRE_CHARGET(state, ptr, 0) == chr)
+            ptr += state->charsize;
         break;
 
     case SRE_OP_LITERAL_IGNORE:
         /* repeated literal */
         chr = pattern[1];
         TRACE(("|%p|%p|COUNT LITERAL_IGNORE %d\n", pattern, ptr, chr));
-        while (ptr < end && (SRE_CODE) state->lower(*ptr) == chr)
-            ptr++;
+        while (ptr < end && (SRE_CODE) state->lower(SRE_CHARGET(state, ptr, 0)) == chr)
+            ptr += state->charsize;
         break;
 
     case SRE_OP_NOT_LITERAL:
         /* repeated non-literal */
         chr = pattern[1];
         TRACE(("|%p|%p|COUNT NOT_LITERAL %d\n", pattern, ptr, chr));
-        while (ptr < end && (SRE_CODE) *ptr != chr)
-            ptr++;
+        while (ptr < end && (SRE_CODE) SRE_CHARGET(state, ptr, 0) != chr)
+            ptr += state->charsize;
         break;
 
     case SRE_OP_NOT_LITERAL_IGNORE:
         /* repeated non-literal */
         chr = pattern[1];
         TRACE(("|%p|%p|COUNT NOT_LITERAL_IGNORE %d\n", pattern, ptr, chr));
-        while (ptr < end && (SRE_CODE) state->lower(*ptr) != chr)
-            ptr++;
+        while (ptr < end && (SRE_CODE) state->lower(SRE_CHARGET(state, ptr, 0)) != chr)
+            ptr += state->charsize;
         break;
 
     default:
         /* repeated single character pattern */
         TRACE(("|%p|%p|COUNT SUBPATTERN\n", pattern, ptr));
-        while ((SRE_CHAR*) state->ptr < end) {
+        while ((char*) state->ptr < end) {
             i = SRE_MATCH(state, pattern);
             if (i < 0)
                 return i;
@@ -586,12 +562,12 @@
                 break;
         }
         TRACE(("|%p|%p|COUNT %d\n", pattern, ptr,
-               (SRE_CHAR*) state->ptr - ptr));
-        return (SRE_CHAR*) state->ptr - ptr;
+               ((char*)state->ptr - ptr)/state->charsize));
+        return ((char*)state->ptr - ptr)/state->charsize;
     }
 
-    TRACE(("|%p|%p|COUNT %d\n", pattern, ptr, ptr - (SRE_CHAR*) state->ptr));
-    return ptr - (SRE_CHAR*) state->ptr;
+    TRACE(("|%p|%p|COUNT %d\n", pattern, ptr, (ptr - (char*) state->ptr)/state->charsize));
+    return (ptr - (char*) state->ptr)/state->charsize;
 }
 
 #if 0 /* not used in this release */
@@ -602,8 +578,8 @@
        returns the number of SRE_CODE objects to skip if successful, 0
        if no match */
 
-    SRE_CHAR* end = state->end;
-    SRE_CHAR* ptr = state->ptr;
+    char* end = state->end;
+    char* ptr = state->ptr;
     Py_ssize_t i;
 
     /* check minimal length */
@@ -614,7 +590,7 @@
     if (pattern[2] & SRE_INFO_PREFIX && pattern[5] > 1) {
         /* <length> <skip> <prefix data> <overlap data> */
         for (i = 0; i < pattern[5]; i++)
-            if ((SRE_CODE) ptr[i] != pattern[7 + i])
+            if ((SRE_CODE) SRE_CHARGET(state, ptr, i) != pattern[7 + i])
                 return 0;
         return pattern[0] + 2 * pattern[6];
     }
@@ -783,7 +759,7 @@
 typedef struct {
     Py_ssize_t last_ctx_pos;
     Py_ssize_t jump;
-    SRE_CHAR* ptr;
+    char* ptr;
     SRE_CODE* pattern;
     Py_ssize_t count;
     Py_ssize_t lastmark;
@@ -799,7 +775,7 @@
 LOCAL(Py_ssize_t)
 SRE_MATCH(SRE_STATE* state, SRE_CODE* pattern)
 {
-    SRE_CHAR* end = (SRE_CHAR *)state->end;
+    char* end = (char*)state->end;
     Py_ssize_t alloc_pos, ctx_pos = -1;
     Py_ssize_t i, ret = 0;
     Py_ssize_t jump;
@@ -818,12 +794,12 @@
 
 entrance:
 
-    ctx->ptr = (SRE_CHAR *)state->ptr;
+    ctx->ptr = (char *)state->ptr;
 
     if (ctx->pattern[0] == SRE_OP_INFO) {
         /* optimization info block */
         /* <INFO> <1=skip> <2=flags> <3=min> ... */
-        if (ctx->pattern[3] && (end - ctx->ptr) < ctx->pattern[3]) {
+        if (ctx->pattern[3] && (end - ctx->ptr)/state->charsize < ctx->pattern[3]) {
             TRACE(("reject (got %d chars, need %d)\n",
                    (end - ctx->ptr), ctx->pattern[3]));
             RETURN_FAILURE;
@@ -865,10 +841,10 @@
             /* <LITERAL> <code> */
             TRACE(("|%p|%p|LITERAL %d\n", ctx->pattern,
                    ctx->ptr, *ctx->pattern));
-            if (ctx->ptr >= end || (SRE_CODE) ctx->ptr[0] != ctx->pattern[0])
+            if (ctx->ptr >= end || (SRE_CODE) SRE_CHARGET(state, ctx->ptr, 0) != ctx->pattern[0])
                 RETURN_FAILURE;
             ctx->pattern++;
-            ctx->ptr++;
+            ctx->ptr += state->charsize;
             break;
 
         case SRE_OP_NOT_LITERAL:
@@ -876,10 +852,10 @@
             /* <NOT_LITERAL> <code> */
             TRACE(("|%p|%p|NOT_LITERAL %d\n", ctx->pattern,
                    ctx->ptr, *ctx->pattern));
-            if (ctx->ptr >= end || (SRE_CODE) ctx->ptr[0] == ctx->pattern[0])
+            if (ctx->ptr >= end || (SRE_CODE) SRE_CHARGET(state, ctx->ptr, 0) == ctx->pattern[0])
                 RETURN_FAILURE;
             ctx->pattern++;
-            ctx->ptr++;
+            ctx->ptr += state->charsize;
             break;
 
         case SRE_OP_SUCCESS:
@@ -902,19 +878,19 @@
             /* <CATEGORY> <code> */
             TRACE(("|%p|%p|CATEGORY %d\n", ctx->pattern,
                    ctx->ptr, *ctx->pattern));
-            if (ctx->ptr >= end || !sre_category(ctx->pattern[0], ctx->ptr[0]))
+            if (ctx->ptr >= end || !sre_category(ctx->pattern[0], SRE_CHARGET(state, ctx->ptr, 0)))
                 RETURN_FAILURE;
             ctx->pattern++;
-            ctx->ptr++;
+            ctx->ptr += state->charsize;
             break;
 
         case SRE_OP_ANY:
             /* match anything (except a newline) */
             /* <ANY> */
             TRACE(("|%p|%p|ANY\n", ctx->pattern, ctx->ptr));
-            if (ctx->ptr >= end || SRE_IS_LINEBREAK(ctx->ptr[0]))
-                RETURN_FAILURE;
-            ctx->ptr++;
+                if (ctx->ptr >= end || SRE_IS_LINEBREAK(SRE_CHARGET(state, ctx->ptr, 0)))
+                    RETURN_FAILURE;
+            ctx->ptr += state->charsize;
             break;
 
         case SRE_OP_ANY_ALL:
@@ -923,47 +899,47 @@
             TRACE(("|%p|%p|ANY_ALL\n", ctx->pattern, ctx->ptr));
             if (ctx->ptr >= end)
                 RETURN_FAILURE;
-            ctx->ptr++;
+            ctx->ptr += state->charsize;
             break;
 
         case SRE_OP_IN:
             /* match set member (or non_member) */
             /* <IN> <skip> <set> */
             TRACE(("|%p|%p|IN\n", ctx->pattern, ctx->ptr));
-            if (ctx->ptr >= end || !SRE_CHARSET(ctx->pattern + 1, *ctx->ptr))
-                RETURN_FAILURE;
+                if (ctx->ptr >= end || !SRE_CHARSET(ctx->pattern + 1, SRE_CHARGET(state, ctx->ptr, 0)))
+                    RETURN_FAILURE;
             ctx->pattern += ctx->pattern[0];
-            ctx->ptr++;
+            ctx->ptr += state->charsize;
             break;
 
         case SRE_OP_LITERAL_IGNORE:
             TRACE(("|%p|%p|LITERAL_IGNORE %d\n",
                    ctx->pattern, ctx->ptr, ctx->pattern[0]));
             if (ctx->ptr >= end ||
-                state->lower(*ctx->ptr) != state->lower(*ctx->pattern))
+                state->lower(SRE_CHARGET(state, ctx->ptr, 0)) != state->lower(*ctx->pattern))
                 RETURN_FAILURE;
             ctx->pattern++;
-            ctx->ptr++;
+            ctx->ptr += state->charsize;
             break;
 
         case SRE_OP_NOT_LITERAL_IGNORE:
             TRACE(("|%p|%p|NOT_LITERAL_IGNORE %d\n",
                    ctx->pattern, ctx->ptr, *ctx->pattern));
             if (ctx->ptr >= end ||
-                state->lower(*ctx->ptr) == state->lower(*ctx->pattern))
+                state->lower(SRE_CHARGET(state, ctx->ptr, 0)) == state->lower(*ctx->pattern))
                 RETURN_FAILURE;
             ctx->pattern++;
-            ctx->ptr++;
+            ctx->ptr += state->charsize;
             break;
 
         case SRE_OP_IN_IGNORE:
             TRACE(("|%p|%p|IN_IGNORE\n", ctx->pattern, ctx->ptr));
             if (ctx->ptr >= end
                 || !SRE_CHARSET(ctx->pattern+1,
-                                (SRE_CODE)state->lower(*ctx->ptr)))
+                                (SRE_CODE)state->lower(SRE_CHARGET(state, ctx->ptr, 0))))
                 RETURN_FAILURE;
             ctx->pattern += ctx->pattern[0];
-            ctx->ptr++;
+            ctx->ptr += state->charsize;
             break;
 
         case SRE_OP_JUMP:
@@ -986,11 +962,11 @@
             for (; ctx->pattern[0]; ctx->pattern += ctx->pattern[0]) {
                 if (ctx->pattern[1] == SRE_OP_LITERAL &&
                     (ctx->ptr >= end ||
-                     (SRE_CODE) *ctx->ptr != ctx->pattern[2]))
+                     (SRE_CODE) SRE_CHARGET(state, ctx->ptr, 0) != ctx->pattern[2]))
                     continue;
                 if (ctx->pattern[1] == SRE_OP_IN &&
                     (ctx->ptr >= end ||
-                     !SRE_CHARSET(ctx->pattern + 3, (SRE_CODE) *ctx->ptr)))
+                     !SRE_CHARSET(ctx->pattern + 3, (SRE_CODE) SRE_CHARGET(state, ctx->ptr, 0))))
                     continue;
                 state->ptr = ctx->ptr;
                 DO_JUMP(JUMP_BRANCH, jump_branch, ctx->pattern+1);
@@ -1021,7 +997,7 @@
             TRACE(("|%p|%p|REPEAT_ONE %d %d\n", ctx->pattern, ctx->ptr,
                    ctx->pattern[1], ctx->pattern[2]));
 
-            if (ctx->ptr + ctx->pattern[1] > end)
+            if (ctx->ptr + state->charsize * ctx->pattern[1] > end)
                 RETURN_FAILURE; /* cannot match */
 
             state->ptr = ctx->ptr;
@@ -1030,7 +1006,7 @@
             RETURN_ON_ERROR(ret);
             DATA_LOOKUP_AT(SRE_MATCH_CONTEXT, ctx, ctx_pos);
             ctx->count = ret;
-            ctx->ptr += ctx->count;
+            ctx->ptr += state->charsize * ctx->count;
 
             /* when we arrive here, count contains the number of
                matches, and ctx->ptr points to the tail of the target
@@ -1054,8 +1030,9 @@
                 ctx->u.chr = ctx->pattern[ctx->pattern[0]+1];
                 for (;;) {
                     while (ctx->count >= (Py_ssize_t) ctx->pattern[1] &&
-                           (ctx->ptr >= end || *ctx->ptr != ctx->u.chr)) {
-                        ctx->ptr--;
+                           (ctx->ptr >= end || 
+                            SRE_CHARGET(state, ctx->ptr, 0) != ctx->u.chr)) {
+                        ctx->ptr -= state->charsize;
                         ctx->count--;
                     }
                     if (ctx->count < (Py_ssize_t) ctx->pattern[1])
@@ -1070,7 +1047,7 @@
 
                     LASTMARK_RESTORE();
 
-                    ctx->ptr--;
+                    ctx->ptr -= state->charsize;
                     ctx->count--;
                 }
 
@@ -1084,7 +1061,7 @@
                         RETURN_ON_ERROR(ret);
                         RETURN_SUCCESS;
                     }
-                    ctx->ptr--;
+                    ctx->ptr -= state->charsize;
                     ctx->count--;
                     LASTMARK_RESTORE();
                 }
@@ -1104,7 +1081,7 @@
             TRACE(("|%p|%p|MIN_REPEAT_ONE %d %d\n", ctx->pattern, ctx->ptr,
                    ctx->pattern[1], ctx->pattern[2]));
 
-            if (ctx->ptr + ctx->pattern[1] > end)
+            if (ctx->ptr + state->charsize * ctx->pattern[1] > end)
                 RETURN_FAILURE; /* cannot match */
 
             state->ptr = ctx->ptr;
@@ -1121,7 +1098,7 @@
                     RETURN_FAILURE;
                 /* advance past minimum matches of repeat */
                 ctx->count = ret;
-                ctx->ptr += ctx->count;
+                ctx->ptr += state->charsize * ctx->count;
             }
 
             if (ctx->pattern[ctx->pattern[0]] == SRE_OP_SUCCESS) {
@@ -1148,7 +1125,7 @@
                     if (ret == 0)
                         break;
                     assert(ret == 1);
-                    ctx->ptr++;
+                    ctx->ptr += state->charsize;
                     ctx->count++;
                     LASTMARK_RESTORE();
                 }
@@ -1320,14 +1297,16 @@
                 if (groupref >= state->lastmark) {
                     RETURN_FAILURE;
                 } else {
-                    SRE_CHAR* p = (SRE_CHAR*) state->mark[groupref];
-                    SRE_CHAR* e = (SRE_CHAR*) state->mark[groupref+1];
+                    char* p = (char*) state->mark[groupref];
+                    char* e = (char*) state->mark[groupref+1];
                     if (!p || !e || e < p)
                         RETURN_FAILURE;
                     while (p < e) {
-                        if (ctx->ptr >= end || *ctx->ptr != *p)
+                        if (ctx->ptr >= end || 
+                            SRE_CHARGET(state, ctx->ptr, 0) != SRE_CHARGET(state, p, 0))
                             RETURN_FAILURE;
-                        p++; ctx->ptr++;
+                        p += state->charsize;
+                        ctx->ptr += state->charsize;
                     }
                 }
             }
@@ -1344,15 +1323,16 @@
                 if (groupref >= state->lastmark) {
                     RETURN_FAILURE;
                 } else {
-                    SRE_CHAR* p = (SRE_CHAR*) state->mark[groupref];
-                    SRE_CHAR* e = (SRE_CHAR*) state->mark[groupref+1];
+                    char* p = (char*) state->mark[groupref];
+                    char* e = (char*) state->mark[groupref+1];
                     if (!p || !e || e < p)
                         RETURN_FAILURE;
                     while (p < e) {
                         if (ctx->ptr >= end ||
-                            state->lower(*ctx->ptr) != state->lower(*p))
+                            state->lower(SRE_CHARGET(state, ctx->ptr, 0)) != state->lower(*p))
                             RETURN_FAILURE;
-                        p++; ctx->ptr++;
+                        p++;
+                        ctx->ptr += state->charsize;
                     }
                 }
             }
@@ -1386,7 +1366,7 @@
             /* <ASSERT> <skip> <back> <pattern> */
             TRACE(("|%p|%p|ASSERT %d\n", ctx->pattern,
                    ctx->ptr, ctx->pattern[1]));
-            state->ptr = ctx->ptr - ctx->pattern[1];
+            state->ptr = ctx->ptr - state->charsize * ctx->pattern[1];
             if (state->ptr < state->beginning)
                 RETURN_FAILURE;
             DO_JUMP(JUMP_ASSERT, jump_assert, ctx->pattern+2);
@@ -1399,7 +1379,7 @@
             /* <ASSERT_NOT> <skip> <back> <pattern> */
             TRACE(("|%p|%p|ASSERT_NOT %d\n", ctx->pattern,
                    ctx->ptr, ctx->pattern[1]));
-            state->ptr = ctx->ptr - ctx->pattern[1];
+            state->ptr = ctx->ptr - state->charsize * ctx->pattern[1];
             if (state->ptr >= state->beginning) {
                 DO_JUMP(JUMP_ASSERT_NOT, jump_assert_not, ctx->pattern+2);
                 if (ret) {
@@ -1481,8 +1461,8 @@
 LOCAL(Py_ssize_t)
 SRE_SEARCH(SRE_STATE* state, SRE_CODE* pattern)
 {
-    SRE_CHAR* ptr = (SRE_CHAR *)state->start;
-    SRE_CHAR* end = (SRE_CHAR *)state->end;
+    char* ptr = (char*)state->start;
+    char* end = (char*)state->end;
     Py_ssize_t status = 0;
     Py_ssize_t prefix_len = 0;
     Py_ssize_t prefix_skip = 0;
@@ -1500,9 +1480,9 @@
         if (pattern[3] > 1) {
             /* adjust end point (but make sure we leave at least one
                character in there, so literal search will work) */
-            end -= pattern[3]-1;
+            end -= (pattern[3]-1) * state->charsize;
             if (end <= ptr)
-                end = ptr+1;
+                end = ptr + state->charsize;
         }
 
         if (flags & SRE_INFO_PREFIX) {
@@ -1528,10 +1508,10 @@
         /* pattern starts with a known prefix.  use the overlap
            table to skip forward as fast as we possibly can */
         Py_ssize_t i = 0;
-        end = (SRE_CHAR *)state->end;
+        end = (char *)state->end;
         while (ptr < end) {
             for (;;) {
-                if ((SRE_CODE) ptr[0] != prefix[i]) {
+                if ((SRE_CODE) SRE_CHARGET(state, ptr, 0) != prefix[i]) {
                     if (!i)
                         break;
                     else
@@ -1540,8 +1520,8 @@
                     if (++i == prefix_len) {
                         /* found a potential match */
                         TRACE(("|%p|%p|SEARCH SCAN\n", pattern, ptr));
-                        state->start = ptr + 1 - prefix_len;
-                        state->ptr = ptr + 1 - prefix_len + prefix_skip;
+                        state->start = ptr - (prefix_len - 1) * state->charsize;
+                        state->ptr = ptr - (prefix_len - prefix_skip - 1) * state->charsize;
                         if (flags & SRE_INFO_LITERAL)
                             return 1; /* we got all of it */
                         status = SRE_MATCH(state, pattern + 2*prefix_skip);
@@ -1553,7 +1533,7 @@
                     break;
                 }
             }
-            ptr++;
+            ptr += state->charsize;
         }
         return 0;
     }
@@ -1563,15 +1543,16 @@
         /* pattern starts with a literal character.  this is used
            for short prefixes, and if fast search is disabled */
         SRE_CODE chr = pattern[1];
-        end = (SRE_CHAR *)state->end;
+        end = (char*)state->end;
         for (;;) {
-            while (ptr < end && (SRE_CODE) ptr[0] != chr)
-                ptr++;
+            while (ptr < end && (SRE_CODE) SRE_CHARGET(state, ptr, 0) != chr)
+                ptr += state->charsize;
             if (ptr >= end)
                 return 0;
             TRACE(("|%p|%p|SEARCH LITERAL\n", pattern, ptr));
             state->start = ptr;
-            state->ptr = ++ptr;
+            ptr += state->charsize;
+            state->ptr = ptr;
             if (flags & SRE_INFO_LITERAL)
                 return 1; /* we got all of it */
             status = SRE_MATCH(state, pattern + 2);
@@ -1580,10 +1561,10 @@
         }
     } else if (charset) {
         /* pattern starts with a character from a known set */
-        end = (SRE_CHAR *)state->end;
+        end = (char*)state->end;
         for (;;) {
-            while (ptr < end && !SRE_CHARSET(charset, ptr[0]))
-                ptr++;
+            while (ptr < end && !SRE_CHARSET(charset, SRE_CHARGET(state, ptr, 0)))
+                ptr += state->charsize;
             if (ptr >= end)
                 return 0;
             TRACE(("|%p|%p|SEARCH CHARSET\n", pattern, ptr));
@@ -1592,13 +1573,14 @@
             status = SRE_MATCH(state, pattern);
             if (status != 0)
                 break;
-            ptr++;
+            ptr += state->charsize;
         }
     } else
         /* general case */
         while (ptr <= end) {
             TRACE(("|%p|%p|SEARCH\n", pattern, ptr));
-            state->start = state->ptr = ptr++;
+            state->start = state->ptr = ptr;
+            ptr += state->charsize;
             status = SRE_MATCH(state, pattern);
             if (status != 0)
                 break;
@@ -1607,16 +1589,6 @@
     return status;
 }
 
-LOCAL(int)
-SRE_LITERAL_TEMPLATE(SRE_CHAR* ptr, Py_ssize_t len)
-{
-    /* check if given string is a literal template (i.e. no escapes) */
-    while (len-- > 0)
-        if (*ptr++ == '\\')
-            return 0;
-    return 1;
-}
-
 #if !defined(SRE_RECURSIVE)
 
 /* -------------------------------------------------------------------- */
@@ -1626,6 +1598,23 @@
 static PyObject*pattern_new_match(PatternObject*, SRE_STATE*, int);
 static PyObject*pattern_scanner(PatternObject*, PyObject*);
 
+static int
+sre_literal_template(int charsize, char* ptr, Py_ssize_t len)
+{
+    /* check if given string is a literal template (i.e. no escapes) */
+    struct {
+        int charsize;
+    } state = {
+        charsize
+    };
+    while (len-- > 0) {
+        if (SRE_CHARGET((&state), ptr, 0) == '\\')
+            return 0;
+        ptr += charsize;
+    }
+    return 1;
+}
+
 static PyObject *
 sre_codesize(PyObject* self, PyObject *unused)
 {
@@ -1641,11 +1630,7 @@
     if (flags & SRE_FLAG_LOCALE)
         return Py_BuildValue("i", sre_lower_locale(character));
     if (flags & SRE_FLAG_UNICODE)
-#if defined(HAVE_UNICODE)
         return Py_BuildValue("i", sre_lower_unicode(character));
-#else
-        return Py_BuildValue("i", sre_lower_locale(character));
-#endif
     return Py_BuildValue("i", sre_lower(character));
 }
 
@@ -1664,7 +1649,8 @@
 }
 
 static void*
-getstring(PyObject* string, Py_ssize_t* p_length, int* p_charsize)
+getstring(PyObject* string, Py_ssize_t* p_length,
+          int* p_logical_charsize, int* p_charsize)
 {
     /* given a python object, return a data pointer, a length (in
        characters), and a character size.  return NULL if the object
@@ -1679,9 +1665,12 @@
     /* Unicode objects do not support the buffer API. So, get the data
        directly instead. */
     if (PyUnicode_Check(string)) {
-        ptr = (void *)PyUnicode_AS_DATA(string);
-        *p_length = PyUnicode_GET_SIZE(string);
-        *p_charsize = sizeof(Py_UNICODE);
+        if (PyUnicode_READY(string) == -1)
+            return NULL;
+        ptr = PyUnicode_DATA(string);
+        *p_length = PyUnicode_GET_LENGTH(string);
+        *p_charsize = PyUnicode_CHARACTER_SIZE(string);
+        *p_logical_charsize = 4;
         return ptr;
     }
 
@@ -1713,10 +1702,8 @@
 
     if (PyBytes_Check(string) || bytes == size)
         charsize = 1;
-#if defined(HAVE_UNICODE)
     else if (bytes == (Py_ssize_t) (size * sizeof(Py_UNICODE)))
         charsize = sizeof(Py_UNICODE);
-#endif
     else {
         PyErr_SetString(PyExc_TypeError, "buffer size mismatch");
         return NULL;
@@ -1724,6 +1711,7 @@
 
     *p_length = size;
     *p_charsize = charsize;
+    *p_logical_charsize = charsize;
 
     if (ptr == NULL) {
             PyErr_SetString(PyExc_ValueError,
@@ -1739,7 +1727,7 @@
     /* prepare state object */
 
     Py_ssize_t length;
-    int charsize;
+    int logical_charsize, charsize;
     void* ptr;
 
     memset(state, 0, sizeof(SRE_STATE));
@@ -1747,16 +1735,16 @@
     state->lastmark = -1;
     state->lastindex = -1;
 
-    ptr = getstring(string, &length, &charsize);
+    ptr = getstring(string, &length, &logical_charsize, &charsize);
     if (!ptr)
         return NULL;
 
-	if (charsize == 1 && pattern->charsize > 1) {
+	if (logical_charsize == 1 && pattern->logical_charsize > 1) {
 		PyErr_SetString(PyExc_TypeError,
 			"can't use a string pattern on a bytes-like object");
 		return NULL;
 	}
-	if (charsize > 1 && pattern->charsize == 1) {
+	if (logical_charsize > 1 && pattern->logical_charsize == 1) {
 		PyErr_SetString(PyExc_TypeError,
 			"can't use a bytes pattern on a string-like object");
 		return NULL;
@@ -1773,6 +1761,7 @@
     else if (end > length)
         end = length;
 
+    state->logical_charsize = logical_charsize;
     state->charsize = charsize;
 
     state->beginning = ptr;
@@ -1788,11 +1777,7 @@
     if (pattern->flags & SRE_FLAG_LOCALE)
         state->lower = sre_lower_locale;
     else if (pattern->flags & SRE_FLAG_UNICODE)
-#if defined(HAVE_UNICODE)
         state->lower = sre_lower_unicode;
-#else
-        state->lower = sre_lower_locale;
-#endif
     else
         state->lower = sre_lower;
 
@@ -1891,12 +1876,10 @@
 
     TRACE(("|%p|%p|MATCH\n", PatternObject_GetCode(self), state.ptr));
 
-    if (state.charsize == 1) {
+    if (state.logical_charsize == 1) {
         status = sre_match(&state, PatternObject_GetCode(self));
     } else {
-#if defined(HAVE_UNICODE)
         status = sre_umatch(&state, PatternObject_GetCode(self));
-#endif
     }
 
     TRACE(("|%p|%p|END\n", PatternObject_GetCode(self), state.ptr));
@@ -1928,12 +1911,10 @@
 
     TRACE(("|%p|%p|SEARCH\n", PatternObject_GetCode(self), state.ptr));
 
-    if (state.charsize == 1) {
+    if (state.logical_charsize == 1) {
         status = sre_search(&state, PatternObject_GetCode(self));
     } else {
-#if defined(HAVE_UNICODE)
         status = sre_usearch(&state, PatternObject_GetCode(self));
-#endif
     }
 
     TRACE(("|%p|%p|END\n", PatternObject_GetCode(self), state.ptr));
@@ -2075,12 +2056,10 @@
 
         state.ptr = state.start;
 
-        if (state.charsize == 1) {
+        if (state.logical_charsize == 1) {
             status = sre_search(&state, PatternObject_GetCode(self));
         } else {
-#if defined(HAVE_UNICODE)
             status = sre_usearch(&state, PatternObject_GetCode(self));
-#endif
         }
 
 	if (PyErr_Occurred())
@@ -2205,12 +2184,10 @@
 
         state.ptr = state.start;
 
-        if (state.charsize == 1) {
+        if (state.logical_charsize == 1) {
             status = sre_search(&state, PatternObject_GetCode(self));
         } else {
-#if defined(HAVE_UNICODE)
             status = sre_usearch(&state, PatternObject_GetCode(self));
-#endif
         }
 
 	if (PyErr_Occurred())
@@ -2295,7 +2272,7 @@
     int status;
     Py_ssize_t n;
     Py_ssize_t i, b, e;
-    int bint;
+    int logical_charsize, charsize;
     int filter_is_callable;
 
     if (PyCallable_Check(ptemplate)) {
@@ -2306,16 +2283,10 @@
     } else {
         /* if not callable, check if it's a literal string */
         int literal;
-        ptr = getstring(ptemplate, &n, &bint);
-        b = bint;
+        ptr = getstring(ptemplate, &n, &logical_charsize, &charsize);
+        b = charsize;
         if (ptr) {
-            if (b == 1) {
-		    literal = sre_literal_template((unsigned char *)ptr, n);
-            } else {
-#if defined(HAVE_UNICODE)
-		    literal = sre_uliteral_template((Py_UNICODE *)ptr, n);
-#endif
-            }
+            literal = sre_literal_template(b, ptr, n);
         } else {
             PyErr_Clear();
             literal = 0;
@@ -2357,12 +2328,10 @@
 
         state.ptr = state.start;
 
-        if (state.charsize == 1) {
+        if (state.logical_charsize == 1) {
             status = sre_search(&state, PatternObject_GetCode(self));
         } else {
-#if defined(HAVE_UNICODE)
             status = sre_usearch(&state, PatternObject_GetCode(self));
-#endif
         }
 
 	if (PyErr_Occurred())
@@ -2694,15 +2663,18 @@
         return NULL;
     }
 
-	if (pattern == Py_None)
-		self->charsize = -1;
-	else {
-		Py_ssize_t p_length;
-		if (!getstring(pattern, &p_length, &self->charsize)) {
-			Py_DECREF(self);
-			return NULL;
-		}
-	}
+    if (pattern == Py_None) {
+        self->logical_charsize = -1;
+        self->charsize = -1;
+    } 
+    else {
+        Py_ssize_t p_length;
+        if (!getstring(pattern, &p_length, &self->logical_charsize,
+                       &self->charsize)) {
+            Py_DECREF(self);
+            return NULL;
+        }
+    }
 
     Py_INCREF(pattern);
     self->pattern = pattern;
@@ -3746,12 +3718,10 @@
 
     state->ptr = state->start;
 
-    if (state->charsize == 1) {
+    if (state->logical_charsize == 1) {
         status = sre_match(state, PatternObject_GetCode(self->pattern));
     } else {
-#if defined(HAVE_UNICODE)
         status = sre_umatch(state, PatternObject_GetCode(self->pattern));
-#endif
     }
     if (PyErr_Occurred())
         return NULL;
@@ -3779,12 +3749,10 @@
 
     state->ptr = state->start;
 
-    if (state->charsize == 1) {
+    if (state->logical_charsize == 1) {
         status = sre_search(state, PatternObject_GetCode(self->pattern));
     } else {
-#if defined(HAVE_UNICODE)
         status = sre_usearch(state, PatternObject_GetCode(self->pattern));
-#endif
     }
     if (PyErr_Occurred())
         return NULL;
diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -1355,7 +1355,7 @@
 test_Z_code(PyObject *self)
 {
     PyObject *tuple, *obj;
-    Py_UNICODE *value1, *value2;
+    const Py_UNICODE *value1, *value2;
     Py_ssize_t len1, len2;
 
     tuple = PyTuple_New(2);
diff --git a/Modules/_tkinter.c b/Modules/_tkinter.c
--- a/Modules/_tkinter.c
+++ b/Modules/_tkinter.c
@@ -80,18 +80,6 @@
 #error "Tk older than 8.3.1 not supported"
 #endif
 
-/* Unicode conversion assumes that Tcl_UniChar is two bytes.
-   We cannot test this directly, so we test UTF-8 size instead,
-   expecting that TCL_UTF_MAX is changed if Tcl ever supports
-   either UTF-16 or UCS-4.
-   Redhat 8 sets TCL_UTF_MAX to 6, and uses wchar_t for
-   Tcl_Unichar. This is also ok as long as Python uses UCS-4,
-   as well.
-*/
-#if TCL_UTF_MAX != 3 && !(defined(Py_UNICODE_WIDE) && TCL_UTF_MAX==6)
-#error "unsupported Tcl configuration"
-#endif
-
 #if !(defined(MS_WINDOWS) || defined(__CYGWIN__))
 #define HAVE_CREATEFILEHANDLER
 #endif
@@ -975,38 +963,44 @@
         return result;
     }
     else if (PyUnicode_Check(value)) {
-        Py_UNICODE *inbuf = PyUnicode_AS_UNICODE(value);
-        Py_ssize_t size = PyUnicode_GET_SIZE(value);
-        /* This #ifdef assumes that Tcl uses UCS-2.
-           See TCL_UTF_MAX test above. */
-#if defined(Py_UNICODE_WIDE) && TCL_UTF_MAX == 3
+        void *inbuf;
+        Py_ssize_t size;
+        int kind;
         Tcl_UniChar *outbuf = NULL;
         Py_ssize_t i;
-        size_t allocsize = ((size_t)size) * sizeof(Tcl_UniChar);
-        if (allocsize >= size)
-            outbuf = (Tcl_UniChar*)ckalloc(allocsize);
+        size_t allocsize;
+
+        if (PyUnicode_READY(value) == -1)
+            return NULL;
+
+        inbuf = PyUnicode_DATA(value);
+        size = PyUnicode_GET_LENGTH(value);
+        kind = PyUnicode_KIND(value);
+        allocsize = ((size_t)size) * sizeof(Tcl_UniChar);
+        outbuf = (Tcl_UniChar*)ckalloc(allocsize);
         /* Else overflow occurred, and we take the next exit */
         if (!outbuf) {
             PyErr_NoMemory();
             return NULL;
         }
         for (i = 0; i < size; i++) {
-            if (inbuf[i] >= 0x10000) {
+            Py_UCS4 ch = PyUnicode_READ(kind, inbuf, i);
+            /* We cannot test for sizeof(Tcl_UniChar) directly,
+               so we test for UTF-8 size instead. */
+#if TCL_UTF_MAX == 3
+            if (ch >= 0x10000) {
                 /* Tcl doesn't do UTF-16, yet. */
                 PyErr_SetString(PyExc_ValueError,
                                 "unsupported character");
                 ckfree(FREECAST outbuf);
                 return NULL;
+#endif
             }
-            outbuf[i] = inbuf[i];
+            outbuf[i] = ch;
         }
         result = Tcl_NewUnicodeObj(outbuf, size);
         ckfree(FREECAST outbuf);
         return result;
-#else
-        return Tcl_NewUnicodeObj(inbuf, size);
-#endif
-
     }
     else if(PyTclObject_Check(value)) {
         Tcl_Obj *v = ((PyTclObject*)value)->value;
@@ -1088,24 +1082,14 @@
     }
 
     if (value->typePtr == app->StringType) {
-#if defined(Py_UNICODE_WIDE) && TCL_UTF_MAX==3
-        PyObject *result;
-        int size;
-        Tcl_UniChar *input;
-        Py_UNICODE *output;
-
-        size = Tcl_GetCharLength(value);
-        result = PyUnicode_FromUnicode(NULL, size);
-        if (!result)
-            return NULL;
-        input = Tcl_GetUnicode(value);
-        output = PyUnicode_AS_UNICODE(result);
-        while (size--)
-            *output++ = *input++;
-        return result;
+#if TCL_UTF_MAX==3
+        return PyUnicode_FromKindAndData(
+            PyUnicode_2BYTE_KIND, Tcl_GetUnicode(value),
+            Tcl_GetCharLength(value));
 #else
-        return PyUnicode_FromUnicode(Tcl_GetUnicode(value),
-                                     Tcl_GetCharLength(value));
+        return PyUnicode_FromKindAndData(
+            PyUnicode_4BYTE_KIND, Tcl_GetUnicode(value),
+            Tcl_GetCharLength(value));
 #endif
     }
 
diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c
--- a/Modules/arraymodule.c
+++ b/Modules/arraymodule.c
@@ -2810,9 +2810,9 @@
 PyInit_array(void)
 {
     PyObject *m;
+    char buffer[PY_ARRAY_LENGTH(descriptors)], *p;
     PyObject *typecodes;
     Py_ssize_t size = 0;
-    register Py_UNICODE *p;
     struct arraydescr *descr;
 
     if (PyType_Ready(&Arraytype) < 0)
@@ -2831,13 +2831,13 @@
         size++;
     }
 
-    typecodes = PyUnicode_FromStringAndSize(NULL, size);
-    p = PyUnicode_AS_UNICODE(typecodes);
+    p = buffer;
     for (descr = descriptors; descr->typecode != '\0'; descr++) {
         *p++ = (char)descr->typecode;
     }
+    typecodes = PyUnicode_DecodeASCII(buffer, p - buffer, NULL);
 
-    PyModule_AddObject(m, "typecodes", (PyObject *)typecodes);
+    PyModule_AddObject(m, "typecodes", typecodes);
 
     if (PyErr_Occurred()) {
         Py_DECREF(m);
diff --git a/Modules/md5module.c b/Modules/md5module.c
--- a/Modules/md5module.c
+++ b/Modules/md5module.c
@@ -376,7 +376,7 @@
     unsigned char digest[MD5_DIGESTSIZE];
     struct md5_state temp;
     PyObject *retval;
-    Py_UNICODE *hex_digest;
+    Py_UCS1 *hex_digest;
     int i, j;
 
     /* Get the raw (binary) digest value */
@@ -384,14 +384,10 @@
     md5_done(&temp, digest);
 
     /* Create a new string */
-    retval = PyUnicode_FromStringAndSize(NULL, MD5_DIGESTSIZE * 2);
+    retval = PyUnicode_New(MD5_DIGESTSIZE * 2, 127);
     if (!retval)
             return NULL;
-    hex_digest = PyUnicode_AS_UNICODE(retval);
-    if (!hex_digest) {
-            Py_DECREF(retval);
-            return NULL;
-    }
+    hex_digest = PyUnicode_1BYTE_DATA(retval);
 
     /* Make hex version of the digest */
     for(i=j=0; i<MD5_DIGESTSIZE; i++) {
diff --git a/Modules/operator.c b/Modules/operator.c
--- a/Modules/operator.c
+++ b/Modules/operator.c
@@ -402,7 +402,8 @@
     for (idx = 0; idx < nattrs; ++idx) {
         PyObject *item = PyTuple_GET_ITEM(args, idx);
         Py_ssize_t item_len;
-        Py_UNICODE *item_buffer;
+        void *data;
+        unsigned int kind;
         int dot_count;
 
         if (!PyUnicode_Check(item)) {
@@ -411,13 +412,18 @@
             Py_DECREF(attr);
             return NULL;
         }
-        item_len = PyUnicode_GET_SIZE(item);
-        item_buffer = PyUnicode_AS_UNICODE(item);
+        if (PyUnicode_READY(item)) {
+            Py_DECREF(attr);
+            return NULL;
+        }
+        item_len = PyUnicode_GET_LENGTH(item);
+        kind = PyUnicode_KIND(item);
+        data = PyUnicode_DATA(item);
 
         /* check whethere the string is dotted */
         dot_count = 0;
         for (char_idx = 0; char_idx < item_len; ++char_idx) {
-            if (item_buffer[char_idx] == (Py_UNICODE)'.')
+            if (PyUnicode_READ(kind, data, char_idx) == '.')
                 ++dot_count;
         }
 
@@ -438,12 +444,12 @@
             }
 
             for (; dot_count > 0; --dot_count) {
-                while (item_buffer[unibuff_till] != (Py_UNICODE)'.') {
+                while (PyUnicode_READ(kind, data, unibuff_till) != '.') {
                     ++unibuff_till;
                 }
-                attr_chain_item = PyUnicode_FromUnicode(
-                                      item_buffer + unibuff_from,
-                                      unibuff_till - unibuff_from);
+                attr_chain_item = PyUnicode_Substring(item,
+                                      unibuff_from,
+                                      unibuff_till);
                 if (attr_chain_item == NULL) {
                     Py_DECREF(attr_chain);
                     Py_DECREF(attr);
@@ -456,9 +462,8 @@
             }
 
             /* now add the last dotless name */
-            attr_chain_item = PyUnicode_FromUnicode(
-                                  item_buffer + unibuff_from,
-                                  item_len - unibuff_from);
+            attr_chain_item = PyUnicode_Substring(item,
+                                                  unibuff_from, item_len);
             if (attr_chain_item == NULL) {
                 Py_DECREF(attr_chain);
                 Py_DECREF(attr);
diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c
--- a/Modules/pyexpat.c
+++ b/Modules/pyexpat.c
@@ -1102,17 +1102,22 @@
     PyUnicodeObject *_u_string = NULL;
     int result = 0;
     int i;
+    int kind;
+    void *data;
 
     /* Yes, supports only 8bit encodings */
     _u_string = (PyUnicodeObject *)
         PyUnicode_Decode(template_buffer, 256, name, "replace");
 
-    if (_u_string == NULL)
+    if (_u_string == NULL || PyUnicode_READY(_u_string) == -1)
         return result;
 
+    kind = PyUnicode_KIND(_u_string);
+    data = PyUnicode_DATA(_u_string);
+
     for (i = 0; i < 256; i++) {
         /* Stupid to access directly, but fast */
-        Py_UNICODE c = _u_string->str[i];
+        Py_UCS4 c = PyUnicode_READ(kind, data, i);
         if (c == Py_UNICODE_REPLACEMENT_CHARACTER)
             info->map[i] = -1;
         else
@@ -1229,7 +1234,7 @@
 static PyObject *
 xmlparse_getattro(xmlparseobject *self, PyObject *nameobj)
 {
-    Py_UNICODE *name;
+    const Py_UNICODE *name;
     int handlernum = -1;
 
     if (!PyUnicode_Check(nameobj))
diff --git a/Modules/sha1module.c b/Modules/sha1module.c
--- a/Modules/sha1module.c
+++ b/Modules/sha1module.c
@@ -352,7 +352,7 @@
     unsigned char digest[SHA1_DIGESTSIZE];
     struct sha1_state temp;
     PyObject *retval;
-    Py_UNICODE *hex_digest;
+    Py_UCS1 *hex_digest;
     int i, j;
 
     /* Get the raw (binary) digest value */
@@ -360,14 +360,10 @@
     sha1_done(&temp, digest);
 
     /* Create a new string */
-    retval = PyUnicode_FromStringAndSize(NULL, SHA1_DIGESTSIZE * 2);
+    retval = PyUnicode_New(SHA1_DIGESTSIZE * 2, 127);
     if (!retval)
             return NULL;
-    hex_digest = PyUnicode_AS_UNICODE(retval);
-    if (!hex_digest) {
-            Py_DECREF(retval);
-            return NULL;
-    }
+    hex_digest = PyUnicode_1BYTE_DATA(retval);
 
     /* Make hex version of the digest */
     for(i=j=0; i<SHA1_DIGESTSIZE; i++) {
diff --git a/Modules/sha256module.c b/Modules/sha256module.c
--- a/Modules/sha256module.c
+++ b/Modules/sha256module.c
@@ -445,7 +445,7 @@
     unsigned char digest[SHA_DIGESTSIZE];
     SHAobject temp;
     PyObject *retval;
-    Py_UNICODE *hex_digest;
+    Py_UCS1 *hex_digest;
     int i, j;
 
     /* Get the raw (binary) digest value */
@@ -453,14 +453,10 @@
     sha_final(digest, &temp);
 
     /* Create a new string */
-    retval = PyUnicode_FromStringAndSize(NULL, self->digestsize * 2);
+    retval = PyUnicode_New(self->digestsize * 2, 127);
     if (!retval)
             return NULL;
-    hex_digest = PyUnicode_AS_UNICODE(retval);
-    if (!hex_digest) {
-            Py_DECREF(retval);
-            return NULL;
-    }
+    hex_digest = PyUnicode_1BYTE_DATA(retval);
 
     /* Make hex version of the digest */
     for(i=j=0; i<self->digestsize; i++) {
diff --git a/Modules/sha512module.c b/Modules/sha512module.c
--- a/Modules/sha512module.c
+++ b/Modules/sha512module.c
@@ -511,7 +511,7 @@
     unsigned char digest[SHA_DIGESTSIZE];
     SHAobject temp;
     PyObject *retval;
-    Py_UNICODE *hex_digest;
+    Py_UCS1 *hex_digest;
     int i, j;
 
     /* Get the raw (binary) digest value */
@@ -519,14 +519,10 @@
     sha512_final(digest, &temp);
 
     /* Create a new string */
-    retval = PyUnicode_FromStringAndSize(NULL, self->digestsize * 2);
+    retval = PyUnicode_New(self->digestsize * 2, 127);
     if (!retval)
             return NULL;
-    hex_digest = PyUnicode_AS_UNICODE(retval);
-    if (!hex_digest) {
-            Py_DECREF(retval);
-            return NULL;
-    }
+    hex_digest = PyUnicode_1BYTE_DATA(retval);
 
     /* Make hex version of the digest */
     for (i=j=0; i<self->digestsize; i++) {
diff --git a/Modules/sre.h b/Modules/sre.h
--- a/Modules/sre.h
+++ b/Modules/sre.h
@@ -30,7 +30,8 @@
     PyObject* pattern; /* pattern source (or None) */
     int flags; /* flags used when compiling pattern source */
     PyObject *weakreflist; /* List of weak references */
-	int charsize; /* pattern charsize (or -1) */
+    int logical_charsize; /* pattern charsize (or -1) */
+    int charsize;
     /* pattern code */
     Py_ssize_t codesize;
     SRE_CODE code[1];
@@ -71,6 +72,7 @@
     PyObject* string;
     Py_ssize_t pos, endpos;
     /* character size */
+    int logical_charsize; /* kind of thing: 1 - bytes, 2/4 - unicode */
     int charsize;
     /* registers */
     Py_ssize_t lastindex;
diff --git a/Modules/syslogmodule.c b/Modules/syslogmodule.c
--- a/Modules/syslogmodule.c
+++ b/Modules/syslogmodule.c
@@ -70,7 +70,7 @@
 
     Py_ssize_t argv_len, scriptlen;
     PyObject *scriptobj;
-    Py_UNICODE *atslash, *atstart;
+    Py_ssize_t slash;
     PyObject *argv = PySys_GetObject("argv");
 
     if (argv == NULL) {
@@ -95,11 +95,13 @@
         return(NULL);
     }
 
-    atstart = PyUnicode_AS_UNICODE(scriptobj);
-    atslash = Py_UNICODE_strrchr(atstart, SEP);
-    if (atslash) {
-        return(PyUnicode_FromUnicode(atslash + 1,
-                                     scriptlen - (atslash - atstart) - 1));
+    slash = PyUnicode_FindChar(scriptobj, SEP,
+                               0, PyUnicode_GET_LENGTH(scriptobj), -1);
+    if (slash == -2)
+        return NULL;
+    if (slash != -1) {
+        return PyUnicode_Substring(scriptobj, slash,
+                                   PyUnicode_GET_LENGTH(scriptobj));
     } else {
         Py_INCREF(scriptobj);
         return(scriptobj);
diff --git a/Modules/unicodedata.c b/Modules/unicodedata.c
--- a/Modules/unicodedata.c
+++ b/Modules/unicodedata.c
@@ -92,16 +92,13 @@
 
 static Py_UCS4 getuchar(PyUnicodeObject *obj)
 {
-    Py_UNICODE *v = PyUnicode_AS_UNICODE(obj);
-
-    if (PyUnicode_GET_SIZE(obj) == 1)
-        return *v;
-#ifndef Py_UNICODE_WIDE
-    else if ((PyUnicode_GET_SIZE(obj) == 2) &&
-             (0xD800 <= v[0] && v[0] <= 0xDBFF) &&
-             (0xDC00 <= v[1] && v[1] <= 0xDFFF))
-        return (((v[0] & 0x3FF)<<10) | (v[1] & 0x3FF)) + 0x10000;
-#endif
+    if (PyUnicode_READY(obj))
+        return (Py_UCS4)-1;
+    if (PyUnicode_GET_LENGTH(obj) == 1) {
+        if (PyUnicode_READY(obj))
+            return (Py_UCS4)-1;
+        return PyUnicode_READ_CHAR(obj, 0);
+    }
     PyErr_SetString(PyExc_TypeError,
                     "need a single Unicode character as parameter");
     return (Py_UCS4)-1;
@@ -1142,7 +1139,6 @@
 unicodedata_lookup(PyObject* self, PyObject* args)
 {
     Py_UCS4 code;
-    Py_UNICODE str[2];
 
     char* name;
     int namelen;
@@ -1155,15 +1151,7 @@
         return NULL;
     }
 
-#ifndef Py_UNICODE_WIDE
-    if (code >= 0x10000) {
-        str[0] = 0xd800 + ((code - 0x10000) >> 10);
-        str[1] = 0xdc00 + ((code - 0x10000) & 0x3ff);
-        return PyUnicode_FromUnicode(str, 2);
-    }
-#endif
-    str[0] = (Py_UNICODE) code;
-    return PyUnicode_FromUnicode(str, 1);
+    return PyUnicode_FromOrdinal(code);
 }
 
 /* XXX Add doc strings. */
diff --git a/Modules/zipimport.c b/Modules/zipimport.c
--- a/Modules/zipimport.c
+++ b/Modules/zipimport.c
@@ -64,7 +64,7 @@
 zipimporter_init(ZipImporter *self, PyObject *args, PyObject *kwds)
 {
     PyObject *pathobj, *files;
-    Py_UNICODE *path, *p, *prefix, buf[MAXPATHLEN+2];
+    Py_UCS4 *path, *p, *prefix, buf[MAXPATHLEN+2];
     Py_ssize_t len;
 
     if (!_PyArg_NoKeywords("zipimporter()", kwds))
@@ -74,8 +74,11 @@
                           PyUnicode_FSDecoder, &pathobj))
         return -1;
 
+    if (PyUnicode_READY(pathobj) == -1)
+        return -1;
+
     /* copy path to buf */
-    len = PyUnicode_GET_SIZE(pathobj);
+    len = PyUnicode_GET_LENGTH(pathobj);
     if (len == 0) {
         PyErr_SetString(ZipImportError, "archive path is empty");
         goto error;
@@ -85,7 +88,8 @@
                         "archive path too long");
         goto error;
     }
-    Py_UNICODE_strcpy(buf, PyUnicode_AS_UNICODE(pathobj));
+    if (!PyUnicode_AsUCS4(pathobj, buf, PY_ARRAY_LENGTH(buf), 1))
+        goto error;
 
 #ifdef ALTSEP
     for (p = buf; *p; p++) {
@@ -101,7 +105,8 @@
         int rv;
 
         if (pathobj == NULL) {
-            pathobj = PyUnicode_FromUnicode(buf, len);
+            pathobj = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                                buf, len);
             if (pathobj == NULL)
                 goto error;
         }
@@ -116,7 +121,7 @@
         else if (PyErr_Occurred())
             goto error;
         /* back up one path element */
-        p = Py_UNICODE_strrchr(buf, SEP);
+        p = Py_UCS4_strrchr(buf, SEP);
         if (prefix != NULL)
             *prefix = SEP;
         if (p == NULL)
@@ -148,7 +153,7 @@
 
     if (prefix != NULL) {
         prefix++;
-        len = Py_UNICODE_strlen(prefix);
+        len = Py_UCS4_strlen(prefix);
         if (prefix[len-1] != SEP) {
             /* add trailing SEP */
             prefix[len] = SEP;
@@ -158,7 +163,8 @@
     }
     else
         len = 0;
-    self->prefix = PyUnicode_FromUnicode(prefix, len);
+    self->prefix = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                               prefix, len);
     if (self->prefix == NULL)
         goto error;
 
@@ -193,7 +199,7 @@
 {
     if (self->archive == NULL)
         return PyUnicode_FromString("<zipimporter object \"???\">");
-    else if (self->prefix != NULL && PyUnicode_GET_SIZE(self->prefix) != 0)
+    else if (self->prefix != NULL && PyUnicode_GET_LENGTH(self->prefix) != 0)
         return PyUnicode_FromFormat("<zipimporter object \"%U%c%U\">",
                                     self->archive, SEP, self->prefix);
     else
@@ -206,16 +212,24 @@
 get_subname(PyObject *fullname)
 {
     Py_ssize_t len;
-    Py_UNICODE *subname;
-    subname = Py_UNICODE_strrchr(PyUnicode_AS_UNICODE(fullname), '.');
+    Py_UCS4 *subname, *fullname_ucs4;
+    fullname_ucs4 = PyUnicode_AsUCS4Copy(fullname);
+    if (!fullname_ucs4)
+        return NULL;
+    subname = Py_UCS4_strrchr(fullname_ucs4, '.');
     if (subname == NULL) {
+        PyMem_Free(fullname_ucs4);
         Py_INCREF(fullname);
         return fullname;
     } else {
+        PyObject *result;
         subname++;
-        len = PyUnicode_GET_SIZE(fullname);
-        len -= subname - PyUnicode_AS_UNICODE(fullname);
-        return PyUnicode_FromUnicode(subname, len);
+        len = PyUnicode_GET_LENGTH(fullname);
+        len -= subname - fullname_ucs4;
+        result = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                           subname, len);
+        PyMem_Free(fullname_ucs4);
+        return result;
     }
 }
 
@@ -228,23 +242,29 @@
 make_filename(PyObject *prefix, PyObject *name)
 {
     PyObject *pathobj;
-    Py_UNICODE *p;
+    Py_UCS4 *p, *buf;
+    Py_ssize_t len;
 
-    pathobj = PyUnicode_FromUnicode(NULL,
-                                    PyUnicode_GET_SIZE(prefix)
-                                    + PyUnicode_GET_SIZE(name));
-    if (pathobj == NULL)
+    len = PyUnicode_GET_LENGTH(prefix) + PyUnicode_GET_LENGTH(name) + 1;
+    p = buf = PyMem_Malloc(sizeof(Py_UCS4) * len);
+    if (buf == NULL) {
+        PyErr_NoMemory();
         return NULL;
+    }
 
-    p = PyUnicode_AS_UNICODE(pathobj);
-
-    Py_UNICODE_strcpy(p, PyUnicode_AS_UNICODE(prefix));
-    p += PyUnicode_GET_SIZE(prefix);
-    Py_UNICODE_strcpy(p, PyUnicode_AS_UNICODE(name));
+    if (!PyUnicode_AsUCS4(prefix, p, len, 0))
+        return NULL;
+    p += PyUnicode_GET_LENGTH(prefix);
+    len -= PyUnicode_GET_LENGTH(prefix);
+    if (!PyUnicode_AsUCS4(name, p, len, 1))
+        return NULL;
     for (; *p; p++) {
         if (*p == '.')
             *p = SEP;
     }
+    pathobj = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                        buf, p-buf);
+    PyMem_Free(buf);
     return pathobj;
 }
 
@@ -330,6 +350,8 @@
     if (!PyArg_ParseTuple(args, "U:zipimporter.load_module",
                           &fullname))
         return NULL;
+    if (PyUnicode_READY(fullname) == -1)
+        return NULL;
 
     code = get_module_code(self, fullname, &ispackage, &modpath);
     if (code == NULL)
@@ -426,46 +448,53 @@
     return PyBool_FromLong(mi == MI_PACKAGE);
 }
 
+
 static PyObject *
 zipimporter_get_data(PyObject *obj, PyObject *args)
 {
     ZipImporter *self = (ZipImporter *)obj;
     PyObject *pathobj, *key;
-    const Py_UNICODE *path;
+    const Py_UCS4 *path;
 #ifdef ALTSEP
-    Py_UNICODE *p, buf[MAXPATHLEN + 1];
+    Py_UCS4 *p;
 #endif
-    Py_UNICODE *archive;
     PyObject *toc_entry;
     Py_ssize_t path_len, len;
+    Py_UCS4 buf[MAXPATHLEN + 1], archive[MAXPATHLEN + 1];
 
     if (!PyArg_ParseTuple(args, "U:zipimporter.get_data", &pathobj))
         return NULL;
 
-    path_len = PyUnicode_GET_SIZE(pathobj);
-    path = PyUnicode_AS_UNICODE(pathobj);
-#ifdef ALTSEP
+    if (PyUnicode_READY(pathobj) == -1)
+        return NULL;
+
+    path_len = PyUnicode_GET_LENGTH(pathobj);
     if (path_len >= MAXPATHLEN) {
         PyErr_SetString(ZipImportError, "path too long");
         return NULL;
     }
-    Py_UNICODE_strcpy(buf, path);
+    if (!PyUnicode_AsUCS4(pathobj, buf, PY_ARRAY_LENGTH(buf), 1))
+        return NULL;
+    path = buf;
+#ifdef ALTSEP
     for (p = buf; *p; p++) {
         if (*p == ALTSEP)
             *p = SEP;
     }
-    path = buf;
 #endif
-    archive = PyUnicode_AS_UNICODE(self->archive);
-    len = PyUnicode_GET_SIZE(self->archive);
-    if ((size_t)len < Py_UNICODE_strlen(path) &&
-        Py_UNICODE_strncmp(path, archive, len) == 0 &&
-        path[len] == SEP) {
-        path += len + 1;
-        path_len -= len + 1;
+    len = PyUnicode_GET_LENGTH(self->archive);
+    if ((size_t)len < Py_UCS4_strlen(path)) {
+        if (!PyUnicode_AsUCS4(self->archive, archive, PY_ARRAY_LENGTH(archive), 1))
+            return NULL;
+        if (Py_UCS4_strncmp(path, archive, len) == 0 &&
+            path[len] == SEP) {
+            path += len + 1;
+            path_len -= len + 1;
+        }
     }
 
-    key = PyUnicode_FromUnicode(path, path_len);
+    key = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                    path, path_len);
     if (key == NULL)
         return NULL;
     toc_entry = PyDict_GetItem(self->files, key);
@@ -725,9 +754,10 @@
     unsigned short flags;
     long compress, crc, data_size, file_size, file_offset, date, time;
     long header_offset, name_size, header_size, header_position;
-    long i, l, count;
+    long l, count;
+    Py_ssize_t i;
     size_t length;
-    Py_UNICODE path[MAXPATHLEN + 5];
+    Py_UCS4 path[MAXPATHLEN + 5];
     char name[MAXPATHLEN + 5];
     PyObject *nameobj = NULL;
     char *p, endof_central_dir[22];
@@ -736,12 +766,13 @@
     const char *charset;
     int bootstrap;
 
-    if (PyUnicode_GET_SIZE(archive) > MAXPATHLEN) {
+    if (PyUnicode_GET_LENGTH(archive) > MAXPATHLEN) {
         PyErr_SetString(PyExc_OverflowError,
                         "Zip path name is too long");
         return NULL;
     }
-    Py_UNICODE_strcpy(path, PyUnicode_AS_UNICODE(archive));
+    if (!PyUnicode_AsUCS4(archive, path, PY_ARRAY_LENGTH(path), 1))
+        return NULL;
 
     fp = _Py_fopen(archive, "rb");
     if (fp == NULL) {
@@ -771,7 +802,7 @@
     if (files == NULL)
         goto error;
 
-    length = Py_UNICODE_strlen(path);
+    length = Py_UCS4_strlen(path);
     path[length] = SEP;
 
     /* Start of Central Directory */
@@ -802,7 +833,7 @@
             name_size = MAXPATHLEN;
 
         p = name;
-        for (i = 0; i < name_size; i++) {
+        for (i = 0; i < (Py_ssize_t)name_size; i++) {
             *p = (char)getc(fp);
             if (*p == '/')
                 *p = SEP;
@@ -827,6 +858,8 @@
         else
             charset = "cp437";
         nameobj = PyUnicode_Decode(name, name_size, charset, NULL);
+        if (PyUnicode_READY(nameobj) == -1)
+            goto error;
         if (nameobj == NULL) {
             if (bootstrap)
                 PyErr_Format(PyExc_NotImplementedError,
@@ -835,11 +868,12 @@
                     PY_MAJOR_VERSION, PY_MINOR_VERSION);
             goto error;
         }
-        Py_UNICODE_strncpy(path + length + 1,
-                           PyUnicode_AS_UNICODE(nameobj),
-                           MAXPATHLEN - length - 1);
-
-        pathobj = PyUnicode_FromUnicode(path, Py_UNICODE_strlen(path));
+        for (i = 0; (i < MAXPATHLEN - length - 1) &&
+                 (i < PyUnicode_GET_LENGTH(nameobj)); i++)
+            path[length + 1 + i] = PyUnicode_READ_CHAR(nameobj, i);
+        path[length + 1 + i] = 0;
+        pathobj = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                            path, Py_UCS4_strlen(path));
         if (pathobj == NULL)
             goto error;
         t = Py_BuildValue("Niiiiiii", pathobj, compress, data_size,
@@ -1148,8 +1182,11 @@
     time_t mtime;
 
     /* strip 'c' or 'o' from *.py[co] */
-    stripped = PyUnicode_FromUnicode(PyUnicode_AS_UNICODE(path),
-                                     PyUnicode_GET_SIZE(path) - 1);
+    if (PyUnicode_READY(path) == -1)
+        return (time_t)-1;
+    stripped = PyUnicode_FromKindAndData(PyUnicode_KIND(path),
+                                         PyUnicode_DATA(path),
+                                         PyUnicode_GET_LENGTH(path) - 1);
     if (stripped == NULL)
         return (time_t)-1;
 
diff --git a/Objects/abstract.c b/Objects/abstract.c
--- a/Objects/abstract.c
+++ b/Objects/abstract.c
@@ -1379,9 +1379,7 @@
                                 PyBytes_GET_SIZE(o));
     if (PyUnicode_Check(o))
         /* The above check is done in PyLong_FromUnicode(). */
-        return PyLong_FromUnicode(PyUnicode_AS_UNICODE(o),
-                                  PyUnicode_GET_SIZE(o),
-                                  10);
+        return PyLong_FromUnicodeObject(o, 10);
     if (!PyObject_AsCharBuffer(o, &buffer, &buffer_len))
         return long_from_string(buffer, buffer_len);
 
diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c
--- a/Objects/bytearrayobject.c
+++ b/Objects/bytearrayobject.c
@@ -854,83 +854,79 @@
     const char *quote_prefix = "bytearray(b";
     const char *quote_postfix = ")";
     Py_ssize_t length = Py_SIZE(self);
-    /* 14 == strlen(quote_prefix) + 2 + strlen(quote_postfix) */
+    /* 15 == strlen(quote_prefix) + 2 + strlen(quote_postfix) + 1 */
     size_t newsize;
     PyObject *v;
-    if (length > (PY_SSIZE_T_MAX - 14) / 4) {
+    register Py_ssize_t i;
+    register char c;
+    register char *p;
+    int quote;
+    char *test, *start;
+    char *buffer;
+
+    if (length > (PY_SSIZE_T_MAX - 15) / 4) {
         PyErr_SetString(PyExc_OverflowError,
             "bytearray object is too large to make repr");
         return NULL;
     }
-    newsize = 14 + 4 * length;
-    v = PyUnicode_FromUnicode(NULL, newsize);
-    if (v == NULL) {
+
+    newsize = 15 + length * 4;
+    buffer = PyMem_Malloc(newsize);
+    if (buffer == NULL) {
+        PyErr_NoMemory();
         return NULL;
     }
-    else {
-        register Py_ssize_t i;
-        register Py_UNICODE c;
-        register Py_UNICODE *p;
-        int quote;
-
-        /* Figure out which quote to use; single is preferred */
-        quote = '\'';
-        {
-            char *test, *start;
-            start = PyByteArray_AS_STRING(self);
-            for (test = start; test < start+length; ++test) {
-                if (*test == '"') {
-                    quote = '\''; /* back to single */
-                    goto decided;
-                }
-                else if (*test == '\'')
-                    quote = '"';
-            }
-          decided:
-            ;
+
+    /* Figure out which quote to use; single is preferred */
+    quote = '\'';
+    start = PyByteArray_AS_STRING(self);
+    for (test = start; test < start+length; ++test) {
+        if (*test == '"') {
+            quote = '\''; /* back to single */
+            break;
         }
-
-        p = PyUnicode_AS_UNICODE(v);
-        while (*quote_prefix)
-            *p++ = *quote_prefix++;
-        *p++ = quote;
-
-        for (i = 0; i < length; i++) {
-            /* There's at least enough room for a hex escape
-               and a closing quote. */
-            assert(newsize - (p - PyUnicode_AS_UNICODE(v)) >= 5);
-            c = self->ob_bytes[i];
-            if (c == '\'' || c == '\\')
-                *p++ = '\\', *p++ = c;
-            else if (c == '\t')
-                *p++ = '\\', *p++ = 't';
-            else if (c == '\n')
-                *p++ = '\\', *p++ = 'n';
-            else if (c == '\r')
-                *p++ = '\\', *p++ = 'r';
-            else if (c == 0)
-                *p++ = '\\', *p++ = 'x', *p++ = '0', *p++ = '0';
-            else if (c < ' ' || c >= 0x7f) {
-                *p++ = '\\';
-                *p++ = 'x';
-                *p++ = hexdigits[(c & 0xf0) >> 4];
-                *p++ = hexdigits[c & 0xf];
-            }
-            else
-                *p++ = c;
+        else if (*test == '\'')
+            quote = '"';
+    }
+
+    p = buffer;
+    while (*quote_prefix)
+        *p++ = *quote_prefix++;
+    *p++ = quote;
+
+    for (i = 0; i < length; i++) {
+        /* There's at least enough room for a hex escape
+           and a closing quote. */
+        assert(newsize - (p - buffer) >= 5);
+        c = self->ob_bytes[i];
+        if (c == '\'' || c == '\\')
+            *p++ = '\\', *p++ = c;
+        else if (c == '\t')
+            *p++ = '\\', *p++ = 't';
+        else if (c == '\n')
+            *p++ = '\\', *p++ = 'n';
+        else if (c == '\r')
+            *p++ = '\\', *p++ = 'r';
+        else if (c == 0)
+            *p++ = '\\', *p++ = 'x', *p++ = '0', *p++ = '0';
+        else if (c < ' ' || c >= 0x7f) {
+            *p++ = '\\';
+            *p++ = 'x';
+            *p++ = hexdigits[(c & 0xf0) >> 4];
+            *p++ = hexdigits[c & 0xf];
         }
-        assert(newsize - (p - PyUnicode_AS_UNICODE(v)) >= 1);
-        *p++ = quote;
-        while (*quote_postfix) {
-           *p++ = *quote_postfix++;
-        }
-        *p = '\0';
-        if (PyUnicode_Resize(&v, (p - PyUnicode_AS_UNICODE(v)))) {
-            Py_DECREF(v);
-            return NULL;
-        }
-        return v;
+        else
+            *p++ = c;
     }
+    assert(newsize - (p - buffer) >= 1);
+    *p++ = quote;
+    while (*quote_postfix) {
+       *p++ = *quote_postfix++;
+    }
+
+    v = PyUnicode_DecodeASCII(buffer, p - buffer, NULL);
+    PyMem_Free(buffer);
+    return v;
 }
 
 static PyObject *
@@ -1034,6 +1030,8 @@
 /* -------------------------------------------------------------------- */
 /* Methods */
 
+#define FASTSEARCH fastsearch
+#define STRINGLIB(F) stringlib_##F
 #define STRINGLIB_CHAR char
 #define STRINGLIB_LEN PyByteArray_GET_SIZE
 #define STRINGLIB_STR PyByteArray_AS_STRING
@@ -2651,15 +2649,20 @@
 {
     PyObject *newbytes, *hexobj;
     char *buf;
-    Py_UNICODE *hex;
     Py_ssize_t hexlen, byteslen, i, j;
     int top, bot;
+    void *data;
+    unsigned int kind;
 
     if (!PyArg_ParseTuple(args, "U:fromhex", &hexobj))
         return NULL;
     assert(PyUnicode_Check(hexobj));
-    hexlen = PyUnicode_GET_SIZE(hexobj);
-    hex = PyUnicode_AS_UNICODE(hexobj);
+    if (PyUnicode_READY(hexobj))
+        return NULL;
+    kind = PyUnicode_KIND(hexobj);
+    data = PyUnicode_DATA(hexobj);
+    hexlen = PyUnicode_GET_LENGTH(hexobj);
+
     byteslen = hexlen/2; /* This overestimates if there are spaces */
     newbytes = PyByteArray_FromStringAndSize(NULL, byteslen);
     if (!newbytes)
@@ -2667,12 +2670,12 @@
     buf = PyByteArray_AS_STRING(newbytes);
     for (i = j = 0; i < hexlen; i += 2) {
         /* skip over spaces in the input */
-        while (hex[i] == ' ')
+        while (PyUnicode_READ(kind, data, i) == ' ')
             i++;
         if (i >= hexlen)
             break;
-        top = hex_digit_to_int(hex[i]);
-        bot = hex_digit_to_int(hex[i+1]);
+        top = hex_digit_to_int(PyUnicode_READ(kind, data, i));
+        bot = hex_digit_to_int(PyUnicode_READ(kind, data, i+1));
         if (top == -1 || bot == -1) {
             PyErr_Format(PyExc_ValueError,
                          "non-hexadecimal number found in "
diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -566,74 +566,68 @@
 {
     static const char *hexdigits = "0123456789abcdef";
     register PyBytesObject* op = (PyBytesObject*) obj;
-    Py_ssize_t length = Py_SIZE(op);
-    size_t newsize;
+    Py_ssize_t i, length = Py_SIZE(op);
+    size_t newsize, squotes, dquotes;
     PyObject *v;
-    if (length > (PY_SSIZE_T_MAX - 3) / 4) {
+    unsigned char quote, *s, *p;
+
+    /* Compute size of output string */
+    squotes = dquotes = 0;
+    newsize = 3; /* b'' */
+    s = (unsigned char*)op->ob_sval;
+    for (i = 0; i < length; i++) {
+        switch(s[i]) {
+        case '\'': squotes++; newsize++; break;
+        case '"':  dquotes++; newsize++; break;
+        case '\\': case '\t': case '\n': case '\r':
+            newsize += 2; break; /* \C */
+        default:
+            if (s[i] < ' ' || s[i] >= 0x7f)
+                newsize += 4; /* \xHH */
+            else
+                newsize++;
+        }
+    }
+    quote = '\'';
+    if (smartquotes && squotes && !dquotes)
+        quote = '"';
+    if (squotes && quote == '\'')
+        newsize += squotes;
+    
+    if (newsize > (PY_SSIZE_T_MAX - sizeof(PyUnicodeObject) - 1)) {
         PyErr_SetString(PyExc_OverflowError,
             "bytes object is too large to make repr");
         return NULL;
     }
-    newsize = 3 + 4 * length;
-    v = PyUnicode_FromUnicode(NULL, newsize);
+
+    v = PyUnicode_New(newsize, 127);
     if (v == NULL) {
         return NULL;
     }
-    else {
-        register Py_ssize_t i;
-        register Py_UNICODE c;
-        register Py_UNICODE *p = PyUnicode_AS_UNICODE(v);
-        int quote;
-
-        /* Figure out which quote to use; single is preferred */
-        quote = '\'';
-        if (smartquotes) {
-            char *test, *start;
-            start = PyBytes_AS_STRING(op);
-            for (test = start; test < start+length; ++test) {
-                if (*test == '"') {
-                    quote = '\''; /* back to single */
-                    goto decided;
-                }
-                else if (*test == '\'')
-                    quote = '"';
-            }
-            decided:
-            ;
+    p = PyUnicode_1BYTE_DATA(v);
+
+    *p++ = 'b', *p++ = quote;
+    for (i = 0; i < length; i++) {
+        unsigned char c = op->ob_sval[i];
+        if (c == quote || c == '\\')
+            *p++ = '\\', *p++ = c;
+        else if (c == '\t')
+            *p++ = '\\', *p++ = 't';
+        else if (c == '\n')
+            *p++ = '\\', *p++ = 'n';
+        else if (c == '\r')
+            *p++ = '\\', *p++ = 'r';
+        else if (c < ' ' || c >= 0x7f) {
+            *p++ = '\\';
+            *p++ = 'x';
+            *p++ = hexdigits[(c & 0xf0) >> 4];
+            *p++ = hexdigits[c & 0xf];
         }
-
-        *p++ = 'b', *p++ = quote;
-        for (i = 0; i < length; i++) {
-            /* There's at least enough room for a hex escape
-               and a closing quote. */
-            assert(newsize - (p - PyUnicode_AS_UNICODE(v)) >= 5);
-            c = op->ob_sval[i];
-            if (c == quote || c == '\\')
-                *p++ = '\\', *p++ = c;
-            else if (c == '\t')
-                *p++ = '\\', *p++ = 't';
-            else if (c == '\n')
-                *p++ = '\\', *p++ = 'n';
-            else if (c == '\r')
-                *p++ = '\\', *p++ = 'r';
-            else if (c < ' ' || c >= 0x7f) {
-                *p++ = '\\';
-                *p++ = 'x';
-                *p++ = hexdigits[(c & 0xf0) >> 4];
-                *p++ = hexdigits[c & 0xf];
-            }
-            else
-                *p++ = c;
-        }
-        assert(newsize - (p - PyUnicode_AS_UNICODE(v)) >= 1);
-        *p++ = quote;
-        *p = '\0';
-        if (PyUnicode_Resize(&v, (p - PyUnicode_AS_UNICODE(v)))) {
-            Py_DECREF(v);
-            return NULL;
-        }
-        return v;
+        else
+            *p++ = c;
     }
+    *p++ = quote;
+    return v;
 }
 
 static PyObject *
@@ -2356,15 +2350,20 @@
 {
     PyObject *newstring, *hexobj;
     char *buf;
-    Py_UNICODE *hex;
     Py_ssize_t hexlen, byteslen, i, j;
     int top, bot;
+    void *data;
+    unsigned int kind;
 
     if (!PyArg_ParseTuple(args, "U:fromhex", &hexobj))
         return NULL;
     assert(PyUnicode_Check(hexobj));
-    hexlen = PyUnicode_GET_SIZE(hexobj);
-    hex = PyUnicode_AS_UNICODE(hexobj);
+    if (PyUnicode_READY(hexobj))
+        return NULL;
+    kind = PyUnicode_KIND(hexobj);
+    data = PyUnicode_DATA(hexobj);
+    hexlen = PyUnicode_GET_LENGTH(hexobj);
+
     byteslen = hexlen/2; /* This overestimates if there are spaces */
     newstring = PyBytes_FromStringAndSize(NULL, byteslen);
     if (!newstring)
@@ -2372,12 +2371,12 @@
     buf = PyBytes_AS_STRING(newstring);
     for (i = j = 0; i < hexlen; i += 2) {
         /* skip over spaces in the input */
-        while (hex[i] == ' ')
+        while (PyUnicode_READ(kind, data, i) == ' ')
             i++;
         if (i >= hexlen)
             break;
-        top = hex_digit_to_int(hex[i]);
-        bot = hex_digit_to_int(hex[i+1]);
+        top = hex_digit_to_int(PyUnicode_READ(kind, data, i));
+        bot = hex_digit_to_int(PyUnicode_READ(kind, data, i+1));
         if (top == -1 || bot == -1) {
             PyErr_Format(PyExc_ValueError,
                          "non-hexadecimal number found in "
diff --git a/Objects/codeobject.c b/Objects/codeobject.c
--- a/Objects/codeobject.c
+++ b/Objects/codeobject.c
@@ -8,19 +8,24 @@
 /* all_name_chars(s): true iff all chars in s are valid NAME_CHARS */
 
 static int
-all_name_chars(Py_UNICODE *s)
+all_name_chars(PyObject *o)
 {
     static char ok_name_char[256];
     static unsigned char *name_chars = (unsigned char *)NAME_CHARS;
+    PyUnicodeObject *u = (PyUnicodeObject *)o;
+    const unsigned char *s;
+
+    if (!PyUnicode_Check(o) || PyUnicode_READY(u) == -1 ||
+        PyUnicode_MAX_CHAR_VALUE(u) >= 128)
+        return 0;
 
     if (ok_name_char[*name_chars] == 0) {
         unsigned char *p;
         for (p = name_chars; *p; p++)
             ok_name_char[*p] = 1;
     }
+    s = PyUnicode_1BYTE_DATA(u);
     while (*s) {
-        if (*s >= 128)
-            return 0;
         if (ok_name_char[*s++] == 0)
             return 0;
     }
@@ -77,9 +82,7 @@
     /* Intern selected string constants */
     for (i = PyTuple_GET_SIZE(consts); --i >= 0; ) {
         PyObject *v = PyTuple_GetItem(consts, i);
-        if (!PyUnicode_Check(v))
-            continue;
-        if (!all_name_chars(PyUnicode_AS_UNICODE(v)))
+        if (!all_name_chars(v))
             continue;
         PyUnicode_InternInPlace(&PyTuple_GET_ITEM(consts, i));
     }
diff --git a/Objects/complexobject.c b/Objects/complexobject.c
--- a/Objects/complexobject.c
+++ b/Objects/complexobject.c
@@ -702,9 +702,8 @@
 
     if (!PyArg_ParseTuple(args, "U:__format__", &format_spec))
     return NULL;
-    return _PyComplex_FormatAdvanced(self,
-                                     PyUnicode_AS_UNICODE(format_spec),
-                                     PyUnicode_GET_SIZE(format_spec));
+    return _PyComplex_FormatAdvanced(self, format_spec, 0,
+                                     PyUnicode_GET_LENGTH(format_spec));
 }
 
 #if 0
@@ -755,20 +754,10 @@
     Py_ssize_t len;
 
     if (PyUnicode_Check(v)) {
-        Py_ssize_t i, buflen = PyUnicode_GET_SIZE(v);
-        Py_UNICODE *bufptr;
-        s_buffer = PyUnicode_TransformDecimalToASCII(
-            PyUnicode_AS_UNICODE(v), buflen);
+        s_buffer = _PyUnicode_TransformDecimalAndSpaceToASCII(v);
         if (s_buffer == NULL)
             return NULL;
-        /* Replace non-ASCII whitespace with ' ' */
-        bufptr = PyUnicode_AS_UNICODE(s_buffer);
-        for (i = 0; i < buflen; i++) {
-            Py_UNICODE ch = bufptr[i];
-            if (ch > 127 && Py_UNICODE_ISSPACE(ch))
-                bufptr[i] = ' ';
-        }
-        s = _PyUnicode_AsStringAndSize(s_buffer, &len);
+        s = PyUnicode_AsUTF8AndSize(s_buffer, &len);
         if (s == NULL)
             goto error;
     }
diff --git a/Objects/dictobject.c b/Objects/dictobject.c
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -710,7 +710,7 @@
     if (!PyDict_Check(op))
         return NULL;
     if (!PyUnicode_CheckExact(key) ||
-        (hash = ((PyUnicodeObject *) key)->hash) == -1)
+        (hash = ((PyASCIIObject *) key)->hash) == -1)
     {
         hash = PyObject_Hash(key);
         if (hash == -1) {
@@ -762,7 +762,7 @@
         return NULL;
     }
     if (!PyUnicode_CheckExact(key) ||
-        (hash = ((PyUnicodeObject *) key)->hash) == -1)
+        (hash = ((PyASCIIObject *) key)->hash) == -1)
     {
         hash = PyObject_Hash(key);
         if (hash == -1) {
@@ -797,7 +797,7 @@
     assert(value);
     mp = (PyDictObject *)op;
     if (!PyUnicode_CheckExact(key) ||
-        (hash = ((PyUnicodeObject *) key)->hash) == -1)
+        (hash = ((PyASCIIObject *) key)->hash) == -1)
     {
         hash = PyObject_Hash(key);
         if (hash == -1)
@@ -842,7 +842,7 @@
     }
     assert(key);
     if (!PyUnicode_CheckExact(key) ||
-        (hash = ((PyUnicodeObject *) key)->hash) == -1) {
+        (hash = ((PyASCIIObject *) key)->hash) == -1) {
         hash = PyObject_Hash(key);
         if (hash == -1)
             return -1;
@@ -1122,7 +1122,7 @@
     PyDictEntry *ep;
     assert(mp->ma_table != NULL);
     if (!PyUnicode_CheckExact(key) ||
-        (hash = ((PyUnicodeObject *) key)->hash) == -1) {
+        (hash = ((PyASCIIObject *) key)->hash) == -1) {
         hash = PyObject_Hash(key);
         if (hash == -1)
             return NULL;
@@ -1726,7 +1726,7 @@
     PyDictEntry *ep;
 
     if (!PyUnicode_CheckExact(key) ||
-        (hash = ((PyUnicodeObject *) key)->hash) == -1) {
+        (hash = ((PyASCIIObject *) key)->hash) == -1) {
         hash = PyObject_Hash(key);
         if (hash == -1)
             return NULL;
@@ -1750,7 +1750,7 @@
         return NULL;
 
     if (!PyUnicode_CheckExact(key) ||
-        (hash = ((PyUnicodeObject *) key)->hash) == -1) {
+        (hash = ((PyASCIIObject *) key)->hash) == -1) {
         hash = PyObject_Hash(key);
         if (hash == -1)
             return NULL;
@@ -1779,7 +1779,7 @@
         return NULL;
 
     if (!PyUnicode_CheckExact(key) ||
-        (hash = ((PyUnicodeObject *) key)->hash) == -1) {
+        (hash = ((PyASCIIObject *) key)->hash) == -1) {
         hash = PyObject_Hash(key);
         if (hash == -1)
             return NULL;
@@ -1824,7 +1824,7 @@
         return NULL;
     }
     if (!PyUnicode_CheckExact(key) ||
-        (hash = ((PyUnicodeObject *) key)->hash) == -1) {
+        (hash = ((PyASCIIObject *) key)->hash) == -1) {
         hash = PyObject_Hash(key);
         if (hash == -1)
             return NULL;
@@ -2033,7 +2033,7 @@
     PyDictEntry *ep;
 
     if (!PyUnicode_CheckExact(key) ||
-        (hash = ((PyUnicodeObject *) key)->hash) == -1) {
+        (hash = ((PyASCIIObject *) key)->hash) == -1) {
         hash = PyObject_Hash(key);
         if (hash == -1)
             return -1;
diff --git a/Objects/exceptions.c b/Objects/exceptions.c
--- a/Objects/exceptions.c
+++ b/Objects/exceptions.c
@@ -962,21 +962,18 @@
 static PyObject*
 my_basename(PyObject *name)
 {
-    Py_UNICODE *unicode;
     Py_ssize_t i, size, offset;
-
-    unicode = PyUnicode_AS_UNICODE(name);
-    size = PyUnicode_GET_SIZE(name);
+    int kind = PyUnicode_KIND(name);
+    void *data = PyUnicode_DATA(name);
+    size = PyUnicode_GET_LENGTH(name);
     offset = 0;
     for(i=0; i < size; i++) {
-        if (unicode[i] == SEP)
+        if (PyUnicode_READ(kind, data, i) == SEP)
             offset = i + 1;
     }
-    if (offset != 0) {
-        return PyUnicode_FromUnicode(
-            PyUnicode_AS_UNICODE(name) + offset,
-            size - offset);
-    } else {
+    if (offset != 0)
+        return PyUnicode_Substring(name, offset, size);
+    else {
         Py_INCREF(name);
         return name;
     }
@@ -1712,6 +1709,7 @@
 };
 PyObject *PyExc_UnicodeTranslateError = (PyObject *)&_PyExc_UnicodeTranslateError;
 
+/* Deprecated. */
 PyObject *
 PyUnicodeTranslateError_Create(
     const Py_UNICODE *object, Py_ssize_t length,
@@ -1721,6 +1719,14 @@
                                  object, length, start, end, reason);
 }
 
+PyObject *
+_PyUnicodeTranslateError_Create(
+    PyObject *object,
+    Py_ssize_t start, Py_ssize_t end, const char *reason)
+{
+    return PyObject_CallFunction(PyExc_UnicodeTranslateError, "Ons",
+                                 object, start, end, reason);
+}
 
 /*
  *    AssertionError extends Exception
diff --git a/Objects/fileobject.c b/Objects/fileobject.c
--- a/Objects/fileobject.c
+++ b/Objects/fileobject.c
@@ -103,23 +103,18 @@
         }
     }
     if (n < 0 && result != NULL && PyUnicode_Check(result)) {
-        Py_UNICODE *s = PyUnicode_AS_UNICODE(result);
-        Py_ssize_t len = PyUnicode_GET_SIZE(result);
+        Py_ssize_t len = PyUnicode_GET_LENGTH(result);
         if (len == 0) {
             Py_DECREF(result);
             result = NULL;
             PyErr_SetString(PyExc_EOFError,
                             "EOF when reading a line");
         }
-        else if (s[len-1] == '\n') {
-            if (result->ob_refcnt == 1)
-                PyUnicode_Resize(&result, len-1);
-            else {
-                PyObject *v;
-                v = PyUnicode_FromUnicode(s, len-1);
-                Py_DECREF(result);
-                result = v;
-            }
+        else if (PyUnicode_READ_CHAR(result, len-1) == '\n') {
+            PyObject *v;
+            v = PyUnicode_Substring(result, 0, len-1);
+            Py_DECREF(result);
+            result = v;
         }
     }
     return result;
diff --git a/Objects/floatobject.c b/Objects/floatobject.c
--- a/Objects/floatobject.c
+++ b/Objects/floatobject.c
@@ -174,20 +174,10 @@
     PyObject *result = NULL;
 
     if (PyUnicode_Check(v)) {
-        Py_ssize_t i, buflen = PyUnicode_GET_SIZE(v);
-        Py_UNICODE *bufptr;
-        s_buffer = PyUnicode_TransformDecimalToASCII(
-            PyUnicode_AS_UNICODE(v), buflen);
+        s_buffer = _PyUnicode_TransformDecimalAndSpaceToASCII(v);
         if (s_buffer == NULL)
             return NULL;
-        /* Replace non-ASCII whitespace with ' ' */
-        bufptr = PyUnicode_AS_UNICODE(s_buffer);
-        for (i = 0; i < buflen; i++) {
-            Py_UNICODE ch = bufptr[i];
-            if (ch > 127 && Py_UNICODE_ISSPACE(ch))
-                bufptr[i] = ' ';
-        }
-        s = _PyUnicode_AsStringAndSize(s_buffer, &len);
+        s = PyUnicode_AsUTF8AndSize(s_buffer, &len);
         if (s == NULL) {
             Py_DECREF(s_buffer);
             return NULL;
@@ -1741,9 +1731,8 @@
 
     if (!PyArg_ParseTuple(args, "U:__format__", &format_spec))
         return NULL;
-    return _PyFloat_FormatAdvanced(self,
-                                   PyUnicode_AS_UNICODE(format_spec),
-                                   PyUnicode_GET_SIZE(format_spec));
+    return _PyFloat_FormatAdvanced(self, format_spec, 0,
+                                   PyUnicode_GET_LENGTH(format_spec));
 }
 
 PyDoc_STRVAR(float__format__doc,
diff --git a/Objects/longobject.c b/Objects/longobject.c
--- a/Objects/longobject.c
+++ b/Objects/longobject.c
@@ -1551,7 +1551,7 @@
     PyObject *str;
     Py_ssize_t size, strlen, size_a, i, j;
     digit *pout, *pin, rem, tenpow;
-    Py_UNICODE *p;
+    unsigned char *p;
     int negative;
 
     a = (PyLongObject *)aa;
@@ -1619,14 +1619,15 @@
         tenpow *= 10;
         strlen++;
     }
-    str = PyUnicode_FromUnicode(NULL, strlen);
+    str = PyUnicode_New(strlen, '9');
     if (str == NULL) {
         Py_DECREF(scratch);
         return NULL;
     }
 
     /* fill the string right-to-left */
-    p = PyUnicode_AS_UNICODE(str) + strlen;
+    assert(PyUnicode_KIND(str) == PyUnicode_1BYTE_KIND);
+    p = PyUnicode_1BYTE_DATA(str) + strlen;
     *p = '\0';
     /* pout[0] through pout[size-2] contribute exactly
        _PyLong_DECIMAL_SHIFT digits each */
@@ -1649,7 +1650,7 @@
         *--p = '-';
 
     /* check we've counted correctly */
-    assert(p == PyUnicode_AS_UNICODE(str));
+    assert(p == PyUnicode_1BYTE_DATA(str));
     Py_DECREF(scratch);
     return (PyObject *)str;
 }
@@ -1662,10 +1663,12 @@
 _PyLong_Format(PyObject *aa, int base)
 {
     register PyLongObject *a = (PyLongObject *)aa;
-    PyObject *str;
+    PyObject *v;
     Py_ssize_t i, sz;
     Py_ssize_t size_a;
-    Py_UNICODE *p, sign = '\0';
+    char *p;
+    char sign = '\0';
+    char *buffer;
     int bits;
 
     assert(base == 2 || base == 8 || base == 10 || base == 16);
@@ -1695,7 +1698,7 @@
     }
     /* compute length of output string: allow 2 characters for prefix and
        1 for possible '-' sign. */
-    if (size_a > (PY_SSIZE_T_MAX - 3) / PyLong_SHIFT) {
+    if (size_a > (PY_SSIZE_T_MAX - 3) / PyLong_SHIFT / sizeof(Py_UCS4)) {
         PyErr_SetString(PyExc_OverflowError,
                         "int is too large to format");
         return NULL;
@@ -1704,11 +1707,12 @@
        is safe from overflow */
     sz = 3 + (size_a * PyLong_SHIFT + (bits - 1)) / bits;
     assert(sz >= 0);
-    str = PyUnicode_FromUnicode(NULL, sz);
-    if (str == NULL)
+    buffer = PyMem_Malloc(sz);
+    if (buffer == NULL) {
+        PyErr_NoMemory();
         return NULL;
-    p = PyUnicode_AS_UNICODE(str) + sz;
-    *p = '\0';
+    }
+    p = &buffer[sz];
     if (Py_SIZE(a) < 0)
         sign = '-';
 
@@ -1724,10 +1728,10 @@
             accumbits += PyLong_SHIFT;
             assert(accumbits >= bits);
             do {
-                Py_UNICODE cdigit;
-                cdigit = (Py_UNICODE)(accum & (base - 1));
+                char cdigit;
+                cdigit = (char)(accum & (base - 1));
                 cdigit += (cdigit < 10) ? '0' : 'a'-10;
-                assert(p > PyUnicode_AS_UNICODE(str));
+                assert(p > buffer);
                 *--p = cdigit;
                 accumbits -= bits;
                 accum >>= bits;
@@ -1744,19 +1748,9 @@
     *--p = '0';
     if (sign)
         *--p = sign;
-    if (p != PyUnicode_AS_UNICODE(str)) {
-        Py_UNICODE *q = PyUnicode_AS_UNICODE(str);
-        assert(p > q);
-        do {
-        } while ((*q++ = *p++) != '\0');
-        q--;
-        if (PyUnicode_Resize(&str,(Py_ssize_t) (q -
-                                        PyUnicode_AS_UNICODE(str)))) {
-            Py_DECREF(str);
-            return NULL;
-        }
-    }
-    return (PyObject *)str;
+    v = PyUnicode_DecodeASCII(p, &buffer[sz] - p, NULL);
+    PyMem_Free(buffer);
+    return v;
 }
 
 /* Table of digit values for 8-bit string -> integer conversion.
@@ -2134,23 +2128,26 @@
 PyObject *
 PyLong_FromUnicode(Py_UNICODE *u, Py_ssize_t length, int base)
 {
+    PyObject *v, *unicode = PyUnicode_FromUnicode(u, length);
+    if (unicode == NULL)
+        return NULL;
+    v = PyLong_FromUnicodeObject(unicode, base);
+    Py_DECREF(unicode);
+    return v;
+}
+
+PyObject *
+PyLong_FromUnicodeObject(PyObject *u, int base)
+{
     PyObject *result;
     PyObject *asciidig;
     char *buffer, *end;
-    Py_ssize_t i, buflen;
-    Py_UNICODE *ptr;
-
-    asciidig = PyUnicode_TransformDecimalToASCII(u, length);
+    Py_ssize_t buflen;
+
+    asciidig = _PyUnicode_TransformDecimalAndSpaceToASCII(u);
     if (asciidig == NULL)
         return NULL;
-    /* Replace non-ASCII whitespace with ' ' */
-    ptr = PyUnicode_AS_UNICODE(asciidig);
-    for (i = 0; i < length; i++) {
-        Py_UNICODE ch = ptr[i];
-        if (ch > 127 && Py_UNICODE_ISSPACE(ch))
-            ptr[i] = ' ';
-    }
-    buffer = _PyUnicode_AsStringAndSize(asciidig, &buflen);
+    buffer = PyUnicode_AsUTF8AndSize(asciidig, &buflen);
     if (buffer == NULL) {
         Py_DECREF(asciidig);
         return NULL;
@@ -4144,9 +4141,7 @@
     }
 
     if (PyUnicode_Check(x))
-        return PyLong_FromUnicode(PyUnicode_AS_UNICODE(x),
-                                  PyUnicode_GET_SIZE(x),
-                                  (int)base);
+        return PyLong_FromUnicodeObject(x, (int)base);
     else if (PyByteArray_Check(x) || PyBytes_Check(x)) {
         /* Since PyLong_FromString doesn't have a length parameter,
          * check here for possible NULs in the string. */
@@ -4228,9 +4223,8 @@
 
     if (!PyArg_ParseTuple(args, "U:__format__", &format_spec))
         return NULL;
-    return _PyLong_FormatAdvanced(self,
-                                  PyUnicode_AS_UNICODE(format_spec),
-                                  PyUnicode_GET_SIZE(format_spec));
+    return _PyLong_FormatAdvanced(self, format_spec, 0,
+                                  PyUnicode_GET_LENGTH(format_spec));
 }
 
 /* Return a pair (q, r) such that a = b * q + r, and
diff --git a/Objects/moduleobject.c b/Objects/moduleobject.c
--- a/Objects/moduleobject.c
+++ b/Objects/moduleobject.c
@@ -285,8 +285,8 @@
     pos = 0;
     while (PyDict_Next(d, &pos, &key, &value)) {
         if (value != Py_None && PyUnicode_Check(key)) {
-            Py_UNICODE *u = PyUnicode_AS_UNICODE(key);
-            if (u[0] == '_' && u[1] != '_') {
+            if (PyUnicode_READ_CHAR(key, 0) == '_' && 
+                PyUnicode_READ_CHAR(key, 1) != '_') {
                 if (Py_VerboseFlag > 1) {
                     const char *s = _PyUnicode_AsString(key);
                     if (s != NULL)
@@ -303,9 +303,8 @@
     pos = 0;
     while (PyDict_Next(d, &pos, &key, &value)) {
         if (value != Py_None && PyUnicode_Check(key)) {
-            Py_UNICODE *u = PyUnicode_AS_UNICODE(key);
-            if (u[0] != '_'
-                || PyUnicode_CompareWithASCIIString(key, "__builtins__") != 0)
+            if (PyUnicode_READ_CHAR(key, 0) != '_' ||
+                PyUnicode_CompareWithASCIIString(key, "__builtins__") != 0)
             {
                 if (Py_VerboseFlag > 1) {
                     const char *s = _PyUnicode_AsString(key);
diff --git a/Objects/object.c b/Objects/object.c
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -295,9 +295,7 @@
             }
             else if (PyUnicode_Check(s)) {
                 PyObject *t;
-                t = PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(s),
-                                         PyUnicode_GET_SIZE(s),
-                                         "backslashreplace");
+                t = PyUnicode_AsEncodedString(s, "utf-8", "backslashreplace");
                 if (t == NULL)
                     ret = 0;
                 else {
@@ -439,11 +437,7 @@
         return NULL;
 
     /* repr is guaranteed to be a PyUnicode object by PyObject_Repr */
-    ascii = PyUnicode_EncodeASCII(
-        PyUnicode_AS_UNICODE(repr),
-        PyUnicode_GET_SIZE(repr),
-        "backslashreplace");
-
+    ascii = _PyUnicode_AsASCIIString(repr, "backslashreplace");
     Py_DECREF(repr);
     if (ascii == NULL)
         return NULL;
diff --git a/Objects/setobject.c b/Objects/setobject.c
--- a/Objects/setobject.c
+++ b/Objects/setobject.c
@@ -386,7 +386,7 @@
     register Py_ssize_t n_used;
 
     if (!PyUnicode_CheckExact(key) ||
-        (hash = ((PyUnicodeObject *) key)->hash) == -1) {
+        (hash = ((PyASCIIObject *) key)->hash) == -1) {
         hash = PyObject_Hash(key);
         if (hash == -1)
             return -1;
@@ -434,7 +434,7 @@
     assert (PyAnySet_Check(so));
 
     if (!PyUnicode_CheckExact(key) ||
-        (hash = ((PyUnicodeObject *) key)->hash) == -1) {
+        (hash = ((PyASCIIObject *) key)->hash) == -1) {
         hash = PyObject_Hash(key);
         if (hash == -1)
             return -1;
@@ -579,11 +579,8 @@
 static PyObject *
 set_repr(PySetObject *so)
 {
-    PyObject *keys, *result=NULL;
-    Py_UNICODE *u;
+    PyObject *result=NULL, *keys, *listrepr, *tmp;
     int status = Py_ReprEnter((PyObject*)so);
-    PyObject *listrepr;
-    Py_ssize_t newsize;
 
     if (status != 0) {
         if (status < 0)
@@ -601,31 +598,24 @@
     if (keys == NULL)
         goto done;
 
+    /* repr(keys)[1:-1] */
     listrepr = PyObject_Repr(keys);
     Py_DECREF(keys);
     if (listrepr == NULL)
         goto done;
-    newsize = PyUnicode_GET_SIZE(listrepr);
-    result = PyUnicode_FromUnicode(NULL, newsize);
-    if (result == NULL)
+    tmp = PyUnicode_Substring(listrepr, 1, PyUnicode_GET_LENGTH(listrepr)-1);
+    Py_DECREF(listrepr);
+    if (tmp == NULL)
         goto done;
+    listrepr = tmp;
 
-    u = PyUnicode_AS_UNICODE(result);
-    *u++ = '{';
-    /* Omit the brackets from the listrepr */
-    Py_UNICODE_COPY(u, PyUnicode_AS_UNICODE(listrepr)+1,
-                       newsize-2);
-    u += newsize-2;
-    *u++ = '}';
+    if (Py_TYPE(so) != &PySet_Type)
+        result = PyUnicode_FromFormat("%s({%U})",
+                                      Py_TYPE(so)->tp_name,
+                                      listrepr);
+    else
+        result = PyUnicode_FromFormat("{%U}", listrepr);
     Py_DECREF(listrepr);
-
-    if (Py_TYPE(so) != &PySet_Type) {
-        PyObject *tmp = PyUnicode_FromFormat("%s(%U)",
-                                             Py_TYPE(so)->tp_name,
-                                             result);
-        Py_DECREF(result);
-        result = tmp;
-    }
 done:
     Py_ReprLeave((PyObject*)so);
     return result;
@@ -684,7 +674,7 @@
     setentry *entry;
 
     if (!PyUnicode_CheckExact(key) ||
-        (hash = ((PyUnicodeObject *) key)->hash) == -1) {
+        (hash = ((PyASCIIObject *) key)->hash) == -1) {
         hash = PyObject_Hash(key);
         if (hash == -1)
             return -1;
diff --git a/Objects/stringlib/count.h b/Objects/stringlib/count.h
--- a/Objects/stringlib/count.h
+++ b/Objects/stringlib/count.h
@@ -1,14 +1,11 @@
 /* stringlib: count implementation */
 
-#ifndef STRINGLIB_COUNT_H
-#define STRINGLIB_COUNT_H
-
 #ifndef STRINGLIB_FASTSEARCH_H
 #error must include "stringlib/fastsearch.h" before including this module
 #endif
 
 Py_LOCAL_INLINE(Py_ssize_t)
-stringlib_count(const STRINGLIB_CHAR* str, Py_ssize_t str_len,
+STRINGLIB(count)(const STRINGLIB_CHAR* str, Py_ssize_t str_len,
                 const STRINGLIB_CHAR* sub, Py_ssize_t sub_len,
                 Py_ssize_t maxcount)
 {
@@ -19,7 +16,7 @@
     if (sub_len == 0)
         return (str_len < maxcount) ? str_len + 1 : maxcount;
 
-    count = fastsearch(str, str_len, sub, sub_len, maxcount, FAST_COUNT);
+    count = FASTSEARCH(str, str_len, sub, sub_len, maxcount, FAST_COUNT);
 
     if (count < 0)
         return 0; /* no match */
@@ -27,4 +24,4 @@
     return count;
 }
 
-#endif
+
diff --git a/Objects/stringlib/eq.h b/Objects/stringlib/eq.h
--- a/Objects/stringlib/eq.h
+++ b/Objects/stringlib/eq.h
@@ -9,13 +9,26 @@
     register PyUnicodeObject *a = (PyUnicodeObject *)aa;
     register PyUnicodeObject *b = (PyUnicodeObject *)bb;
 
-    if (a->length != b->length)
+    if (PyUnicode_READY(a) == -1 || PyUnicode_READY(b) == -1) {
+        assert(0 && "unicode_eq ready fail");
         return 0;
-    if (a->length == 0)
+    }
+
+    if (PyUnicode_GET_LENGTH(a) != PyUnicode_GET_LENGTH(b))
+        return 0;
+    if (PyUnicode_GET_LENGTH(a) == 0)
         return 1;
-    if (a->str[0] != b->str[0])
+    if (PyUnicode_KIND(a) != PyUnicode_KIND(b))
         return 0;
-    if (a->length == 1)
+    /* Just comparing the first byte is enough to see if a and b differ.
+     * If they are 2 byte or 4 byte character most differences will happen in
+     * the lower bytes anyways.
+     */
+    if (PyUnicode_1BYTE_DATA(a)[0] != PyUnicode_1BYTE_DATA(b)[0])
+        return 0;
+    if (PyUnicode_KIND(a) == PyUnicode_1BYTE_KIND &&
+        PyUnicode_GET_LENGTH(a) == 1)
         return 1;
-    return memcmp(a->str, b->str, a->length * sizeof(Py_UNICODE)) == 0;
+    return memcmp(PyUnicode_1BYTE_DATA(a), PyUnicode_1BYTE_DATA(b),
+                  PyUnicode_GET_LENGTH(a) * PyUnicode_CHARACTER_SIZE(a)) == 0;
 }
diff --git a/Objects/stringlib/fastsearch.h b/Objects/stringlib/fastsearch.h
--- a/Objects/stringlib/fastsearch.h
+++ b/Objects/stringlib/fastsearch.h
@@ -1,6 +1,5 @@
 /* stringlib: fastsearch implementation */
 
-#ifndef STRINGLIB_FASTSEARCH_H
 #define STRINGLIB_FASTSEARCH_H
 
 /* fast search/count implementation, based on a mix between boyer-
@@ -34,7 +33,7 @@
     ((mask &  (1UL << ((ch) & (STRINGLIB_BLOOM_WIDTH -1)))))
 
 Py_LOCAL_INLINE(Py_ssize_t)
-fastsearch(const STRINGLIB_CHAR* s, Py_ssize_t n,
+FASTSEARCH(const STRINGLIB_CHAR* s, Py_ssize_t n,
            const STRINGLIB_CHAR* p, Py_ssize_t m,
            Py_ssize_t maxcount, int mode)
 {
@@ -157,4 +156,3 @@
     return count;
 }
 
-#endif
diff --git a/Objects/stringlib/find.h b/Objects/stringlib/find.h
--- a/Objects/stringlib/find.h
+++ b/Objects/stringlib/find.h
@@ -1,14 +1,11 @@
 /* stringlib: find/index implementation */
 
-#ifndef STRINGLIB_FIND_H
-#define STRINGLIB_FIND_H
-
 #ifndef STRINGLIB_FASTSEARCH_H
 #error must include "stringlib/fastsearch.h" before including this module
 #endif
 
 Py_LOCAL_INLINE(Py_ssize_t)
-stringlib_find(const STRINGLIB_CHAR* str, Py_ssize_t str_len,
+STRINGLIB(find)(const STRINGLIB_CHAR* str, Py_ssize_t str_len,
                const STRINGLIB_CHAR* sub, Py_ssize_t sub_len,
                Py_ssize_t offset)
 {
@@ -19,7 +16,7 @@
     if (sub_len == 0)
         return offset;
 
-    pos = fastsearch(str, str_len, sub, sub_len, -1, FAST_SEARCH);
+    pos = FASTSEARCH(str, str_len, sub, sub_len, -1, FAST_SEARCH);
 
     if (pos >= 0)
         pos += offset;
@@ -28,7 +25,7 @@
 }
 
 Py_LOCAL_INLINE(Py_ssize_t)
-stringlib_rfind(const STRINGLIB_CHAR* str, Py_ssize_t str_len,
+STRINGLIB(rfind)(const STRINGLIB_CHAR* str, Py_ssize_t str_len,
                 const STRINGLIB_CHAR* sub, Py_ssize_t sub_len,
                 Py_ssize_t offset)
 {
@@ -39,7 +36,7 @@
     if (sub_len == 0)
         return str_len + offset;
 
-    pos = fastsearch(str, str_len, sub, sub_len, -1, FAST_RSEARCH);
+    pos = FASTSEARCH(str, str_len, sub, sub_len, -1, FAST_RSEARCH);
 
     if (pos >= 0)
         pos += offset;
@@ -63,29 +60,29 @@
     }
 
 Py_LOCAL_INLINE(Py_ssize_t)
-stringlib_find_slice(const STRINGLIB_CHAR* str, Py_ssize_t str_len,
+STRINGLIB(find_slice)(const STRINGLIB_CHAR* str, Py_ssize_t str_len,
                      const STRINGLIB_CHAR* sub, Py_ssize_t sub_len,
                      Py_ssize_t start, Py_ssize_t end)
 {
     ADJUST_INDICES(start, end, str_len);
-    return stringlib_find(str + start, end - start, sub, sub_len, start);
+    return STRINGLIB(find)(str + start, end - start, sub, sub_len, start);
 }
 
 Py_LOCAL_INLINE(Py_ssize_t)
-stringlib_rfind_slice(const STRINGLIB_CHAR* str, Py_ssize_t str_len,
+STRINGLIB(rfind_slice)(const STRINGLIB_CHAR* str, Py_ssize_t str_len,
                       const STRINGLIB_CHAR* sub, Py_ssize_t sub_len,
                       Py_ssize_t start, Py_ssize_t end)
 {
     ADJUST_INDICES(start, end, str_len);
-    return stringlib_rfind(str + start, end - start, sub, sub_len, start);
+    return STRINGLIB(rfind)(str + start, end - start, sub, sub_len, start);
 }
 
 #ifdef STRINGLIB_WANT_CONTAINS_OBJ
 
 Py_LOCAL_INLINE(int)
-stringlib_contains_obj(PyObject* str, PyObject* sub)
+STRINGLIB(contains_obj)(PyObject* str, PyObject* sub)
 {
-    return stringlib_find(
+    return STRINGLIB(find)(
         STRINGLIB_STR(str), STRINGLIB_LEN(str),
         STRINGLIB_STR(sub), STRINGLIB_LEN(sub), 0
         ) != -1;
@@ -105,7 +102,7 @@
 #define FORMAT_BUFFER_SIZE 50
 
 Py_LOCAL_INLINE(int)
-stringlib_parse_args_finds(const char * function_name, PyObject *args,
+STRINGLIB(parse_args_finds)(const char * function_name, PyObject *args,
                            PyObject **subobj,
                            Py_ssize_t *start, Py_ssize_t *end)
 {
@@ -153,13 +150,13 @@
 */
 
 Py_LOCAL_INLINE(int)
-stringlib_parse_args_finds_unicode(const char * function_name, PyObject *args,
+STRINGLIB(parse_args_finds_unicode)(const char * function_name, PyObject *args,
                                    PyUnicodeObject **substring,
                                    Py_ssize_t *start, Py_ssize_t *end)
 {
     PyObject *tmp_substring;
 
-    if(stringlib_parse_args_finds(function_name, args, &tmp_substring,
+    if(STRINGLIB(parse_args_finds)(function_name, args, &tmp_substring,
                                   start, end)) {
         tmp_substring = PyUnicode_FromObject(tmp_substring);
         if (!tmp_substring)
@@ -171,5 +168,3 @@
 }
 
 #endif /* STRINGLIB_IS_UNICODE */
-
-#endif /* STRINGLIB_FIND_H */
diff --git a/Objects/stringlib/formatter.h b/Objects/stringlib/formatter.h
deleted file mode 100644
--- a/Objects/stringlib/formatter.h
+++ /dev/null
@@ -1,1516 +0,0 @@
-/* implements the string, long, and float formatters.  that is,
-   string.__format__, etc. */
-
-#include <locale.h>
-
-/* Before including this, you must include either:
-   stringlib/unicodedefs.h
-   stringlib/stringdefs.h
-
-   Also, you should define the names:
-   FORMAT_STRING
-   FORMAT_LONG
-   FORMAT_FLOAT
-   FORMAT_COMPLEX
-   to be whatever you want the public names of these functions to
-   be.  These are the only non-static functions defined here.
-*/
-
-/* Raises an exception about an unknown presentation type for this
- * type. */
-
-static void
-unknown_presentation_type(STRINGLIB_CHAR presentation_type,
-                          const char* type_name)
-{
-#if STRINGLIB_IS_UNICODE
-    /* If STRINGLIB_CHAR is Py_UNICODE, %c might be out-of-range,
-       hence the two cases. If it is char, gcc complains that the
-       condition below is always true, hence the ifdef. */
-    if (presentation_type > 32 && presentation_type < 128)
-#endif
-        PyErr_Format(PyExc_ValueError,
-                     "Unknown format code '%c' "
-                     "for object of type '%.200s'",
-                     (char)presentation_type,
-                     type_name);
-#if STRINGLIB_IS_UNICODE
-    else
-        PyErr_Format(PyExc_ValueError,
-                     "Unknown format code '\\x%x' "
-                     "for object of type '%.200s'",
-                     (unsigned int)presentation_type,
-                     type_name);
-#endif
-}
-
-static void
-invalid_comma_type(STRINGLIB_CHAR presentation_type)
-{
-#if STRINGLIB_IS_UNICODE
-    /* See comment in unknown_presentation_type */
-    if (presentation_type > 32 && presentation_type < 128)
-#endif
-        PyErr_Format(PyExc_ValueError,
-                     "Cannot specify ',' with '%c'.",
-                     (char)presentation_type);
-#if STRINGLIB_IS_UNICODE
-    else
-        PyErr_Format(PyExc_ValueError,
-                     "Cannot specify ',' with '\\x%x'.",
-                     (unsigned int)presentation_type);
-#endif
-}
-
-/*
-    get_integer consumes 0 or more decimal digit characters from an
-    input string, updates *result with the corresponding positive
-    integer, and returns the number of digits consumed.
-
-    returns -1 on error.
-*/
-static int
-get_integer(STRINGLIB_CHAR **ptr, STRINGLIB_CHAR *end,
-                  Py_ssize_t *result)
-{
-    Py_ssize_t accumulator, digitval;
-    int numdigits;
-    accumulator = numdigits = 0;
-    for (;;(*ptr)++, numdigits++) {
-        if (*ptr >= end)
-            break;
-        digitval = STRINGLIB_TODECIMAL(**ptr);
-        if (digitval < 0)
-            break;
-        /*
-           Detect possible overflow before it happens:
-
-              accumulator * 10 + digitval > PY_SSIZE_T_MAX if and only if
-              accumulator > (PY_SSIZE_T_MAX - digitval) / 10.
-        */
-        if (accumulator > (PY_SSIZE_T_MAX - digitval) / 10) {
-            PyErr_Format(PyExc_ValueError,
-                         "Too many decimal digits in format string");
-            return -1;
-        }
-        accumulator = accumulator * 10 + digitval;
-    }
-    *result = accumulator;
-    return numdigits;
-}
-
-/************************************************************************/
-/*********** standard format specifier parsing **************************/
-/************************************************************************/
-
-/* returns true if this character is a specifier alignment token */
-Py_LOCAL_INLINE(int)
-is_alignment_token(STRINGLIB_CHAR c)
-{
-    switch (c) {
-    case '<': case '>': case '=': case '^':
-        return 1;
-    default:
-        return 0;
-    }
-}
-
-/* returns true if this character is a sign element */
-Py_LOCAL_INLINE(int)
-is_sign_element(STRINGLIB_CHAR c)
-{
-    switch (c) {
-    case ' ': case '+': case '-':
-        return 1;
-    default:
-        return 0;
-    }
-}
-
-
-typedef struct {
-    STRINGLIB_CHAR fill_char;
-    STRINGLIB_CHAR align;
-    int alternate;
-    STRINGLIB_CHAR sign;
-    Py_ssize_t width;
-    int thousands_separators;
-    Py_ssize_t precision;
-    STRINGLIB_CHAR type;
-} InternalFormatSpec;
-
-
-#if 0
-/* Occassionally useful for debugging. Should normally be commented out. */
-static void
-DEBUG_PRINT_FORMAT_SPEC(InternalFormatSpec *format)
-{
-    printf("internal format spec: fill_char %d\n", format->fill_char);
-    printf("internal format spec: align %d\n", format->align);
-    printf("internal format spec: alternate %d\n", format->alternate);
-    printf("internal format spec: sign %d\n", format->sign);
-    printf("internal format spec: width %zd\n", format->width);
-    printf("internal format spec: thousands_separators %d\n",
-           format->thousands_separators);
-    printf("internal format spec: precision %zd\n", format->precision);
-    printf("internal format spec: type %c\n", format->type);
-    printf("\n");
-}
-#endif
-
-
-/*
-  ptr points to the start of the format_spec, end points just past its end.
-  fills in format with the parsed information.
-  returns 1 on success, 0 on failure.
-  if failure, sets the exception
-*/
-static int
-parse_internal_render_format_spec(STRINGLIB_CHAR *format_spec,
-                                  Py_ssize_t format_spec_len,
-                                  InternalFormatSpec *format,
-                                  char default_type,
-                                  char default_align)
-{
-    STRINGLIB_CHAR *ptr = format_spec;
-    STRINGLIB_CHAR *end = format_spec + format_spec_len;
-
-    /* end-ptr is used throughout this code to specify the length of
-       the input string */
-
-    Py_ssize_t consumed;
-    int align_specified = 0;
-
-    format->fill_char = '\0';
-    format->align = default_align;
-    format->alternate = 0;
-    format->sign = '\0';
-    format->width = -1;
-    format->thousands_separators = 0;
-    format->precision = -1;
-    format->type = default_type;
-
-    /* If the second char is an alignment token,
-       then parse the fill char */
-    if (end-ptr >= 2 && is_alignment_token(ptr[1])) {
-        format->align = ptr[1];
-        format->fill_char = ptr[0];
-        align_specified = 1;
-        ptr += 2;
-    }
-    else if (end-ptr >= 1 && is_alignment_token(ptr[0])) {
-        format->align = ptr[0];
-        align_specified = 1;
-        ++ptr;
-    }
-
-    /* Parse the various sign options */
-    if (end-ptr >= 1 && is_sign_element(ptr[0])) {
-        format->sign = ptr[0];
-        ++ptr;
-    }
-
-    /* If the next character is #, we're in alternate mode.  This only
-       applies to integers. */
-    if (end-ptr >= 1 && ptr[0] == '#') {
-        format->alternate = 1;
-        ++ptr;
-    }
-
-    /* The special case for 0-padding (backwards compat) */
-    if (format->fill_char == '\0' && end-ptr >= 1 && ptr[0] == '0') {
-        format->fill_char = '0';
-        if (!align_specified) {
-            format->align = '=';
-        }
-        ++ptr;
-    }
-
-    consumed = get_integer(&ptr, end, &format->width);
-    if (consumed == -1)
-        /* Overflow error. Exception already set. */
-        return 0;
-
-    /* If consumed is 0, we didn't consume any characters for the
-       width. In that case, reset the width to -1, because
-       get_integer() will have set it to zero. -1 is how we record
-       that the width wasn't specified. */
-    if (consumed == 0)
-        format->width = -1;
-
-    /* Comma signifies add thousands separators */
-    if (end-ptr && ptr[0] == ',') {
-        format->thousands_separators = 1;
-        ++ptr;
-    }
-
-    /* Parse field precision */
-    if (end-ptr && ptr[0] == '.') {
-        ++ptr;
-
-        consumed = get_integer(&ptr, end, &format->precision);
-        if (consumed == -1)
-            /* Overflow error. Exception already set. */
-            return 0;
-
-        /* Not having a precision after a dot is an error. */
-        if (consumed == 0) {
-            PyErr_Format(PyExc_ValueError,
-                         "Format specifier missing precision");
-            return 0;
-        }
-
-    }
-
-    /* Finally, parse the type field. */
-
-    if (end-ptr > 1) {
-        /* More than one char remain, invalid conversion spec. */
-        PyErr_Format(PyExc_ValueError, "Invalid conversion specification");
-        return 0;
-    }
-
-    if (end-ptr == 1) {
-        format->type = ptr[0];
-        ++ptr;
-    }
-
-    /* Do as much validating as we can, just by looking at the format
-       specifier.  Do not take into account what type of formatting
-       we're doing (int, float, string). */
-
-    if (format->thousands_separators) {
-        switch (format->type) {
-        case 'd':
-        case 'e':
-        case 'f':
-        case 'g':
-        case 'E':
-        case 'G':
-        case '%':
-        case 'F':
-        case '\0':
-            /* These are allowed. See PEP 378.*/
-            break;
-        default:
-            invalid_comma_type(format->type);
-            return 0;
-        }
-    }
-
-    return 1;
-}
-
-/* Calculate the padding needed. */
-static void
-calc_padding(Py_ssize_t nchars, Py_ssize_t width, STRINGLIB_CHAR align,
-             Py_ssize_t *n_lpadding, Py_ssize_t *n_rpadding,
-             Py_ssize_t *n_total)
-{
-    if (width >= 0) {
-        if (nchars > width)
-            *n_total = nchars;
-        else
-            *n_total = width;
-    }
-    else {
-        /* not specified, use all of the chars and no more */
-        *n_total = nchars;
-    }
-
-    /* Figure out how much leading space we need, based on the
-       aligning */
-    if (align == '>')
-        *n_lpadding = *n_total - nchars;
-    else if (align == '^')
-        *n_lpadding = (*n_total - nchars) / 2;
-    else if (align == '<' || align == '=')
-        *n_lpadding = 0;
-    else {
-        /* We should never have an unspecified alignment. */
-        *n_lpadding = 0;
-        assert(0);
-    }
-
-    *n_rpadding = *n_total - nchars - *n_lpadding;
-}
-
-/* Do the padding, and return a pointer to where the caller-supplied
-   content goes. */
-static STRINGLIB_CHAR *
-fill_padding(STRINGLIB_CHAR *p, Py_ssize_t nchars, STRINGLIB_CHAR fill_char,
-             Py_ssize_t n_lpadding, Py_ssize_t n_rpadding)
-{
-    /* Pad on left. */
-    if (n_lpadding)
-        STRINGLIB_FILL(p, fill_char, n_lpadding);
-
-    /* Pad on right. */
-    if (n_rpadding)
-        STRINGLIB_FILL(p + nchars + n_lpadding, fill_char, n_rpadding);
-
-    /* Pointer to the user content. */
-    return p + n_lpadding;
-}
-
-#if defined FORMAT_FLOAT || defined FORMAT_LONG || defined FORMAT_COMPLEX
-/************************************************************************/
-/*********** common routines for numeric formatting *********************/
-/************************************************************************/
-
-/* Locale type codes. */
-#define LT_CURRENT_LOCALE 0
-#define LT_DEFAULT_LOCALE 1
-#define LT_NO_LOCALE 2
-
-/* Locale info needed for formatting integers and the part of floats
-   before and including the decimal. Note that locales only support
-   8-bit chars, not unicode. */
-typedef struct {
-    char *decimal_point;
-    char *thousands_sep;
-    char *grouping;
-} LocaleInfo;
-
-/* describes the layout for an integer, see the comment in
-   calc_number_widths() for details */
-typedef struct {
-    Py_ssize_t n_lpadding;
-    Py_ssize_t n_prefix;
-    Py_ssize_t n_spadding;
-    Py_ssize_t n_rpadding;
-    char sign;
-    Py_ssize_t n_sign;      /* number of digits needed for sign (0/1) */
-    Py_ssize_t n_grouped_digits; /* Space taken up by the digits, including
-                                    any grouping chars. */
-    Py_ssize_t n_decimal;   /* 0 if only an integer */
-    Py_ssize_t n_remainder; /* Digits in decimal and/or exponent part,
-                               excluding the decimal itself, if
-                               present. */
-
-    /* These 2 are not the widths of fields, but are needed by
-       STRINGLIB_GROUPING. */
-    Py_ssize_t n_digits;    /* The number of digits before a decimal
-                               or exponent. */
-    Py_ssize_t n_min_width; /* The min_width we used when we computed
-                               the n_grouped_digits width. */
-} NumberFieldWidths;
-
-
-/* Given a number of the form:
-   digits[remainder]
-   where ptr points to the start and end points to the end, find where
-    the integer part ends. This could be a decimal, an exponent, both,
-    or neither.
-   If a decimal point is present, set *has_decimal and increment
-    remainder beyond it.
-   Results are undefined (but shouldn't crash) for improperly
-    formatted strings.
-*/
-static void
-parse_number(STRINGLIB_CHAR *ptr, Py_ssize_t len,
-             Py_ssize_t *n_remainder, int *has_decimal)
-{
-    STRINGLIB_CHAR *end = ptr + len;
-    STRINGLIB_CHAR *remainder;
-
-    while (ptr<end && isdigit(*ptr))
-        ++ptr;
-    remainder = ptr;
-
-    /* Does remainder start with a decimal point? */
-    *has_decimal = ptr<end && *remainder == '.';
-
-    /* Skip the decimal point. */
-    if (*has_decimal)
-        remainder++;
-
-    *n_remainder = end - remainder;
-}
-
-/* not all fields of format are used.  for example, precision is
-   unused.  should this take discrete params in order to be more clear
-   about what it does?  or is passing a single format parameter easier
-   and more efficient enough to justify a little obfuscation? */
-static Py_ssize_t
-calc_number_widths(NumberFieldWidths *spec, Py_ssize_t n_prefix,
-                   STRINGLIB_CHAR sign_char, STRINGLIB_CHAR *number,
-                   Py_ssize_t n_number, Py_ssize_t n_remainder,
-                   int has_decimal, const LocaleInfo *locale,
-                   const InternalFormatSpec *format)
-{
-    Py_ssize_t n_non_digit_non_padding;
-    Py_ssize_t n_padding;
-
-    spec->n_digits = n_number - n_remainder - (has_decimal?1:0);
-    spec->n_lpadding = 0;
-    spec->n_prefix = n_prefix;
-    spec->n_decimal = has_decimal ? strlen(locale->decimal_point) : 0;
-    spec->n_remainder = n_remainder;
-    spec->n_spadding = 0;
-    spec->n_rpadding = 0;
-    spec->sign = '\0';
-    spec->n_sign = 0;
-
-    /* the output will look like:
-       |                                                                                         |
-       | <lpadding> <sign> <prefix> <spadding> <grouped_digits> <decimal> <remainder> <rpadding> |
-       |                                                                                         |
-
-       sign is computed from format->sign and the actual
-       sign of the number
-
-       prefix is given (it's for the '0x' prefix)
-
-       digits is already known
-
-       the total width is either given, or computed from the
-       actual digits
-
-       only one of lpadding, spadding, and rpadding can be non-zero,
-       and it's calculated from the width and other fields
-    */
-
-    /* compute the various parts we're going to write */
-    switch (format->sign) {
-    case '+':
-        /* always put a + or - */
-        spec->n_sign = 1;
-        spec->sign = (sign_char == '-' ? '-' : '+');
-        break;
-    case ' ':
-        spec->n_sign = 1;
-        spec->sign = (sign_char == '-' ? '-' : ' ');
-        break;
-    default:
-        /* Not specified, or the default (-) */
-        if (sign_char == '-') {
-            spec->n_sign = 1;
-            spec->sign = '-';
-        }
-    }
-
-    /* The number of chars used for non-digits and non-padding. */
-    n_non_digit_non_padding = spec->n_sign + spec->n_prefix + spec->n_decimal +
-        spec->n_remainder;
-
-    /* min_width can go negative, that's okay. format->width == -1 means
-       we don't care. */
-    if (format->fill_char == '0' && format->align == '=')
-        spec->n_min_width = format->width - n_non_digit_non_padding;
-    else
-        spec->n_min_width = 0;
-
-    if (spec->n_digits == 0)
-        /* This case only occurs when using 'c' formatting, we need
-           to special case it because the grouping code always wants
-           to have at least one character. */
-        spec->n_grouped_digits = 0;
-    else
-        spec->n_grouped_digits = STRINGLIB_GROUPING(NULL, 0, NULL,
-                                                    spec->n_digits,
-                                                    spec->n_min_width,
-                                                    locale->grouping,
-                                                    locale->thousands_sep);
-
-    /* Given the desired width and the total of digit and non-digit
-       space we consume, see if we need any padding. format->width can
-       be negative (meaning no padding), but this code still works in
-       that case. */
-    n_padding = format->width -
-                        (n_non_digit_non_padding + spec->n_grouped_digits);
-    if (n_padding > 0) {
-        /* Some padding is needed. Determine if it's left, space, or right. */
-        switch (format->align) {
-        case '<':
-            spec->n_rpadding = n_padding;
-            break;
-        case '^':
-            spec->n_lpadding = n_padding / 2;
-            spec->n_rpadding = n_padding - spec->n_lpadding;
-            break;
-        case '=':
-            spec->n_spadding = n_padding;
-            break;
-        case '>':
-            spec->n_lpadding = n_padding;
-            break;
-        default:
-            /* Shouldn't get here, but treat it as '>' */
-            spec->n_lpadding = n_padding;
-            assert(0);
-            break;
-        }
-    }
-    return spec->n_lpadding + spec->n_sign + spec->n_prefix +
-        spec->n_spadding + spec->n_grouped_digits + spec->n_decimal +
-        spec->n_remainder + spec->n_rpadding;
-}
-
-/* Fill in the digit parts of a numbers's string representation,
-   as determined in calc_number_widths().
-   No error checking, since we know the buffer is the correct size. */
-static void
-fill_number(STRINGLIB_CHAR *buf, const NumberFieldWidths *spec,
-            STRINGLIB_CHAR *digits, Py_ssize_t n_digits,
-            STRINGLIB_CHAR *prefix, STRINGLIB_CHAR fill_char,
-            LocaleInfo *locale, int toupper)
-{
-    /* Used to keep track of digits, decimal, and remainder. */
-    STRINGLIB_CHAR *p = digits;
-
-#ifndef NDEBUG
-    Py_ssize_t r;
-#endif
-
-    if (spec->n_lpadding) {
-        STRINGLIB_FILL(buf, fill_char, spec->n_lpadding);
-        buf += spec->n_lpadding;
-    }
-    if (spec->n_sign == 1) {
-        *buf++ = spec->sign;
-    }
-    if (spec->n_prefix) {
-        memmove(buf,
-                prefix,
-                spec->n_prefix * sizeof(STRINGLIB_CHAR));
-        if (toupper) {
-            Py_ssize_t t;
-            for (t = 0; t < spec->n_prefix; ++t)
-                buf[t] = STRINGLIB_TOUPPER(buf[t]);
-        }
-        buf += spec->n_prefix;
-    }
-    if (spec->n_spadding) {
-        STRINGLIB_FILL(buf, fill_char, spec->n_spadding);
-        buf += spec->n_spadding;
-    }
-
-    /* Only for type 'c' special case, it has no digits. */
-    if (spec->n_digits != 0) {
-        /* Fill the digits with InsertThousandsGrouping. */
-#ifndef NDEBUG
-        r =
-#endif
-            STRINGLIB_GROUPING(buf, spec->n_grouped_digits, digits,
-                               spec->n_digits, spec->n_min_width,
-                               locale->grouping, locale->thousands_sep);
-#ifndef NDEBUG
-        assert(r == spec->n_grouped_digits);
-#endif
-        p += spec->n_digits;
-    }
-    if (toupper) {
-        Py_ssize_t t;
-        for (t = 0; t < spec->n_grouped_digits; ++t)
-            buf[t] = STRINGLIB_TOUPPER(buf[t]);
-    }
-    buf += spec->n_grouped_digits;
-
-    if (spec->n_decimal) {
-        Py_ssize_t t;
-        for (t = 0; t < spec->n_decimal; ++t)
-            buf[t] = locale->decimal_point[t];
-        buf += spec->n_decimal;
-        p += 1;
-    }
-
-    if (spec->n_remainder) {
-        memcpy(buf, p, spec->n_remainder * sizeof(STRINGLIB_CHAR));
-        buf += spec->n_remainder;
-        p += spec->n_remainder;
-    }
-
-    if (spec->n_rpadding) {
-        STRINGLIB_FILL(buf, fill_char, spec->n_rpadding);
-        buf += spec->n_rpadding;
-    }
-}
-
-static char no_grouping[1] = {CHAR_MAX};
-
-/* Find the decimal point character(s?), thousands_separator(s?), and
-   grouping description, either for the current locale if type is
-   LT_CURRENT_LOCALE, a hard-coded locale if LT_DEFAULT_LOCALE, or
-   none if LT_NO_LOCALE. */
-static void
-get_locale_info(int type, LocaleInfo *locale_info)
-{
-    switch (type) {
-    case LT_CURRENT_LOCALE: {
-        struct lconv *locale_data = localeconv();
-        locale_info->decimal_point = locale_data->decimal_point;
-        locale_info->thousands_sep = locale_data->thousands_sep;
-        locale_info->grouping = locale_data->grouping;
-        break;
-    }
-    case LT_DEFAULT_LOCALE:
-        locale_info->decimal_point = ".";
-        locale_info->thousands_sep = ",";
-        locale_info->grouping = "\3"; /* Group every 3 characters.  The
-                                         (implicit) trailing 0 means repeat
-                                         infinitely. */
-        break;
-    case LT_NO_LOCALE:
-        locale_info->decimal_point = ".";
-        locale_info->thousands_sep = "";
-        locale_info->grouping = no_grouping;
-        break;
-    default:
-        assert(0);
-    }
-}
-
-#endif /* FORMAT_FLOAT || FORMAT_LONG || FORMAT_COMPLEX */
-
-/************************************************************************/
-/*********** string formatting ******************************************/
-/************************************************************************/
-
-static PyObject *
-format_string_internal(PyObject *value, const InternalFormatSpec *format)
-{
-    Py_ssize_t lpad;
-    Py_ssize_t rpad;
-    Py_ssize_t total;
-    STRINGLIB_CHAR *p;
-    Py_ssize_t len = STRINGLIB_LEN(value);
-    PyObject *result = NULL;
-
-    /* sign is not allowed on strings */
-    if (format->sign != '\0') {
-        PyErr_SetString(PyExc_ValueError,
-                        "Sign not allowed in string format specifier");
-        goto done;
-    }
-
-    /* alternate is not allowed on strings */
-    if (format->alternate) {
-        PyErr_SetString(PyExc_ValueError,
-                        "Alternate form (#) not allowed in string format "
-                        "specifier");
-        goto done;
-    }
-
-    /* '=' alignment not allowed on strings */
-    if (format->align == '=') {
-        PyErr_SetString(PyExc_ValueError,
-                        "'=' alignment not allowed "
-                        "in string format specifier");
-        goto done;
-    }
-
-    /* if precision is specified, output no more that format.precision
-       characters */
-    if (format->precision >= 0 && len >= format->precision) {
-        len = format->precision;
-    }
-
-    calc_padding(len, format->width, format->align, &lpad, &rpad, &total);
-
-    /* allocate the resulting string */
-    result = STRINGLIB_NEW(NULL, total);
-    if (result == NULL)
-        goto done;
-
-    /* Write into that space. First the padding. */
-    p = fill_padding(STRINGLIB_STR(result), len,
-                     format->fill_char=='\0'?' ':format->fill_char,
-                     lpad, rpad);
-
-    /* Then the source string. */
-    memcpy(p, STRINGLIB_STR(value), len * sizeof(STRINGLIB_CHAR));
-
-done:
-    return result;
-}
-
-
-/************************************************************************/
-/*********** long formatting ********************************************/
-/************************************************************************/
-
-#if defined FORMAT_LONG || defined FORMAT_INT
-typedef PyObject*
-(*IntOrLongToString)(PyObject *value, int base);
-
-static PyObject *
-format_int_or_long_internal(PyObject *value, const InternalFormatSpec *format,
-                            IntOrLongToString tostring)
-{
-    PyObject *result = NULL;
-    PyObject *tmp = NULL;
-    STRINGLIB_CHAR *pnumeric_chars;
-    STRINGLIB_CHAR numeric_char;
-    STRINGLIB_CHAR sign_char = '\0';
-    Py_ssize_t n_digits;       /* count of digits need from the computed
-                                  string */
-    Py_ssize_t n_remainder = 0; /* Used only for 'c' formatting, which
-                                   produces non-digits */
-    Py_ssize_t n_prefix = 0;   /* Count of prefix chars, (e.g., '0x') */
-    Py_ssize_t n_total;
-    STRINGLIB_CHAR *prefix = NULL;
-    NumberFieldWidths spec;
-    long x;
-
-    /* Locale settings, either from the actual locale or
-       from a hard-code pseudo-locale */
-    LocaleInfo locale;
-
-    /* no precision allowed on integers */
-    if (format->precision != -1) {
-        PyErr_SetString(PyExc_ValueError,
-                        "Precision not allowed in integer format specifier");
-        goto done;
-    }
-
-    /* special case for character formatting */
-    if (format->type == 'c') {
-        /* error to specify a sign */
-        if (format->sign != '\0') {
-            PyErr_SetString(PyExc_ValueError,
-                            "Sign not allowed with integer"
-                            " format specifier 'c'");
-            goto done;
-        }
-
-        /* taken from unicodeobject.c formatchar() */
-        /* Integer input truncated to a character */
-/* XXX: won't work for int */
-        x = PyLong_AsLong(value);
-        if (x == -1 && PyErr_Occurred())
-            goto done;
-#ifdef Py_UNICODE_WIDE
-        if (x < 0 || x > 0x10ffff) {
-            PyErr_SetString(PyExc_OverflowError,
-                            "%c arg not in range(0x110000) "
-                            "(wide Python build)");
-            goto done;
-        }
-#else
-        if (x < 0 || x > 0xffff) {
-            PyErr_SetString(PyExc_OverflowError,
-                            "%c arg not in range(0x10000) "
-                            "(narrow Python build)");
-            goto done;
-        }
-#endif
-        numeric_char = (STRINGLIB_CHAR)x;
-        pnumeric_chars = &numeric_char;
-        n_digits = 1;
-
-        /* As a sort-of hack, we tell calc_number_widths that we only
-           have "remainder" characters. calc_number_widths thinks
-           these are characters that don't get formatted, only copied
-           into the output string. We do this for 'c' formatting,
-           because the characters are likely to be non-digits. */
-        n_remainder = 1;
-    }
-    else {
-        int base;
-        int leading_chars_to_skip = 0;  /* Number of characters added by
-                                           PyNumber_ToBase that we want to
-                                           skip over. */
-
-        /* Compute the base and how many characters will be added by
-           PyNumber_ToBase */
-        switch (format->type) {
-        case 'b':
-            base = 2;
-            leading_chars_to_skip = 2; /* 0b */
-            break;
-        case 'o':
-            base = 8;
-            leading_chars_to_skip = 2; /* 0o */
-            break;
-        case 'x':
-        case 'X':
-            base = 16;
-            leading_chars_to_skip = 2; /* 0x */
-            break;
-        default:  /* shouldn't be needed, but stops a compiler warning */
-        case 'd':
-        case 'n':
-            base = 10;
-            break;
-        }
-
-        /* The number of prefix chars is the same as the leading
-           chars to skip */
-        if (format->alternate)
-            n_prefix = leading_chars_to_skip;
-
-        /* Do the hard part, converting to a string in a given base */
-        tmp = tostring(value, base);
-        if (tmp == NULL)
-            goto done;
-
-        pnumeric_chars = STRINGLIB_STR(tmp);
-        n_digits = STRINGLIB_LEN(tmp);
-
-        prefix = pnumeric_chars;
-
-        /* Remember not to modify what pnumeric_chars points to.  it
-           might be interned.  Only modify it after we copy it into a
-           newly allocated output buffer. */
-
-        /* Is a sign character present in the output?  If so, remember it
-           and skip it */
-        if (pnumeric_chars[0] == '-') {
-            sign_char = pnumeric_chars[0];
-            ++prefix;
-            ++leading_chars_to_skip;
-        }
-
-        /* Skip over the leading chars (0x, 0b, etc.) */
-        n_digits -= leading_chars_to_skip;
-        pnumeric_chars += leading_chars_to_skip;
-    }
-
-    /* Determine the grouping, separator, and decimal point, if any. */
-    get_locale_info(format->type == 'n' ? LT_CURRENT_LOCALE :
-                    (format->thousands_separators ?
-                     LT_DEFAULT_LOCALE :
-                     LT_NO_LOCALE),
-                    &locale);
-
-    /* Calculate how much memory we'll need. */
-    n_total = calc_number_widths(&spec, n_prefix, sign_char, pnumeric_chars,
-                       n_digits, n_remainder, 0, &locale, format);
-
-    /* Allocate the memory. */
-    result = STRINGLIB_NEW(NULL, n_total);
-    if (!result)
-        goto done;
-
-    /* Populate the memory. */
-    fill_number(STRINGLIB_STR(result), &spec, pnumeric_chars, n_digits,
-                prefix, format->fill_char == '\0' ? ' ' : format->fill_char,
-                &locale, format->type == 'X');
-
-done:
-    Py_XDECREF(tmp);
-    return result;
-}
-#endif /* defined FORMAT_LONG || defined FORMAT_INT */
-
-/************************************************************************/
-/*********** float formatting *******************************************/
-/************************************************************************/
-
-#ifdef FORMAT_FLOAT
-#if STRINGLIB_IS_UNICODE
-static void
-strtounicode(Py_UNICODE *buffer, const char *charbuffer, Py_ssize_t len)
-{
-    Py_ssize_t i;
-    for (i = 0; i < len; ++i)
-        buffer[i] = (Py_UNICODE)charbuffer[i];
-}
-#endif
-
-/* much of this is taken from unicodeobject.c */
-static PyObject *
-format_float_internal(PyObject *value,
-                      const InternalFormatSpec *format)
-{
-    char *buf = NULL;       /* buffer returned from PyOS_double_to_string */
-    Py_ssize_t n_digits;
-    Py_ssize_t n_remainder;
-    Py_ssize_t n_total;
-    int has_decimal;
-    double val;
-    Py_ssize_t precision = format->precision;
-    Py_ssize_t default_precision = 6;
-    STRINGLIB_CHAR type = format->type;
-    int add_pct = 0;
-    STRINGLIB_CHAR *p;
-    NumberFieldWidths spec;
-    int flags = 0;
-    PyObject *result = NULL;
-    STRINGLIB_CHAR sign_char = '\0';
-    int float_type; /* Used to see if we have a nan, inf, or regular float. */
-
-#if STRINGLIB_IS_UNICODE
-    Py_UNICODE *unicode_tmp = NULL;
-#endif
-
-    /* Locale settings, either from the actual locale or
-       from a hard-code pseudo-locale */
-    LocaleInfo locale;
-
-    if (format->alternate)
-        flags |= Py_DTSF_ALT;
-
-    if (type == '\0') {
-        /* Omitted type specifier.  Behaves in the same way as repr(x)
-           and str(x) if no precision is given, else like 'g', but with
-           at least one digit after the decimal point. */
-        flags |= Py_DTSF_ADD_DOT_0;
-        type = 'r';
-        default_precision = 0;
-    }
-
-    if (type == 'n')
-        /* 'n' is the same as 'g', except for the locale used to
-           format the result. We take care of that later. */
-        type = 'g';
-
-    val = PyFloat_AsDouble(value);
-    if (val == -1.0 && PyErr_Occurred())
-        goto done;
-
-    if (type == '%') {
-        type = 'f';
-        val *= 100;
-        add_pct = 1;
-    }
-
-    if (precision < 0)
-        precision = default_precision;
-    else if (type == 'r')
-        type = 'g';
-
-    /* Cast "type", because if we're in unicode we need to pass a
-       8-bit char. This is safe, because we've restricted what "type"
-       can be. */
-    buf = PyOS_double_to_string(val, (char)type, precision, flags,
-                                &float_type);
-    if (buf == NULL)
-        goto done;
-    n_digits = strlen(buf);
-
-    if (add_pct) {
-        /* We know that buf has a trailing zero (since we just called
-           strlen() on it), and we don't use that fact any more. So we
-           can just write over the trailing zero. */
-        buf[n_digits] = '%';
-        n_digits += 1;
-    }
-
-    /* Since there is no unicode version of PyOS_double_to_string,
-       just use the 8 bit version and then convert to unicode. */
-#if STRINGLIB_IS_UNICODE
-    unicode_tmp = (Py_UNICODE*)PyMem_Malloc((n_digits)*sizeof(Py_UNICODE));
-    if (unicode_tmp == NULL) {
-        PyErr_NoMemory();
-        goto done;
-    }
-    strtounicode(unicode_tmp, buf, n_digits);
-    p = unicode_tmp;
-#else
-    p = buf;
-#endif
-
-    /* Is a sign character present in the output?  If so, remember it
-       and skip it */
-    if (*p == '-') {
-        sign_char = *p;
-        ++p;
-        --n_digits;
-    }
-
-    /* Determine if we have any "remainder" (after the digits, might include
-       decimal or exponent or both (or neither)) */
-    parse_number(p, n_digits, &n_remainder, &has_decimal);
-
-    /* Determine the grouping, separator, and decimal point, if any. */
-    get_locale_info(format->type == 'n' ? LT_CURRENT_LOCALE :
-                    (format->thousands_separators ?
-                     LT_DEFAULT_LOCALE :
-                     LT_NO_LOCALE),
-                    &locale);
-
-    /* Calculate how much memory we'll need. */
-    n_total = calc_number_widths(&spec, 0, sign_char, p, n_digits,
-                                 n_remainder, has_decimal, &locale, format);
-
-    /* Allocate the memory. */
-    result = STRINGLIB_NEW(NULL, n_total);
-    if (result == NULL)
-        goto done;
-
-    /* Populate the memory. */
-    fill_number(STRINGLIB_STR(result), &spec, p, n_digits, NULL,
-                format->fill_char == '\0' ? ' ' : format->fill_char, &locale,
-                0);
-
-done:
-    PyMem_Free(buf);
-#if STRINGLIB_IS_UNICODE
-    PyMem_Free(unicode_tmp);
-#endif
-    return result;
-}
-#endif /* FORMAT_FLOAT */
-
-/************************************************************************/
-/*********** complex formatting *****************************************/
-/************************************************************************/
-
-#ifdef FORMAT_COMPLEX
-
-static PyObject *
-format_complex_internal(PyObject *value,
-                        const InternalFormatSpec *format)
-{
-    double re;
-    double im;
-    char *re_buf = NULL;       /* buffer returned from PyOS_double_to_string */
-    char *im_buf = NULL;       /* buffer returned from PyOS_double_to_string */
-
-    InternalFormatSpec tmp_format = *format;
-    Py_ssize_t n_re_digits;
-    Py_ssize_t n_im_digits;
-    Py_ssize_t n_re_remainder;
-    Py_ssize_t n_im_remainder;
-    Py_ssize_t n_re_total;
-    Py_ssize_t n_im_total;
-    int re_has_decimal;
-    int im_has_decimal;
-    Py_ssize_t precision = format->precision;
-    Py_ssize_t default_precision = 6;
-    STRINGLIB_CHAR type = format->type;
-    STRINGLIB_CHAR *p_re;
-    STRINGLIB_CHAR *p_im;
-    NumberFieldWidths re_spec;
-    NumberFieldWidths im_spec;
-    int flags = 0;
-    PyObject *result = NULL;
-    STRINGLIB_CHAR *p;
-    STRINGLIB_CHAR re_sign_char = '\0';
-    STRINGLIB_CHAR im_sign_char = '\0';
-    int re_float_type; /* Used to see if we have a nan, inf, or regular float. */
-    int im_float_type;
-    int add_parens = 0;
-    int skip_re = 0;
-    Py_ssize_t lpad;
-    Py_ssize_t rpad;
-    Py_ssize_t total;
-
-#if STRINGLIB_IS_UNICODE
-    Py_UNICODE *re_unicode_tmp = NULL;
-    Py_UNICODE *im_unicode_tmp = NULL;
-#endif
-
-    /* Locale settings, either from the actual locale or
-       from a hard-code pseudo-locale */
-    LocaleInfo locale;
-
-    /* Zero padding is not allowed. */
-    if (format->fill_char == '0') {
-        PyErr_SetString(PyExc_ValueError,
-                        "Zero padding is not allowed in complex format "
-                        "specifier");
-        goto done;
-    }
-
-    /* Neither is '=' alignment . */
-    if (format->align == '=') {
-        PyErr_SetString(PyExc_ValueError,
-                        "'=' alignment flag is not allowed in complex format "
-                        "specifier");
-        goto done;
-    }
-
-    re = PyComplex_RealAsDouble(value);
-    if (re == -1.0 && PyErr_Occurred())
-        goto done;
-    im = PyComplex_ImagAsDouble(value);
-    if (im == -1.0 && PyErr_Occurred())
-        goto done;
-
-    if (format->alternate)
-        flags |= Py_DTSF_ALT;
-
-    if (type == '\0') {
-        /* Omitted type specifier. Should be like str(self). */
-        type = 'r';
-        default_precision = 0;
-        if (re == 0.0 && copysign(1.0, re) == 1.0)
-            skip_re = 1;
-        else
-            add_parens = 1;
-    }
-
-    if (type == 'n')
-        /* 'n' is the same as 'g', except for the locale used to
-           format the result. We take care of that later. */
-        type = 'g';
-
-    if (precision < 0)
-        precision = default_precision;
-    else if (type == 'r')
-        type = 'g';
-
-    /* Cast "type", because if we're in unicode we need to pass a
-       8-bit char. This is safe, because we've restricted what "type"
-       can be. */
-    re_buf = PyOS_double_to_string(re, (char)type, precision, flags,
-                                   &re_float_type);
-    if (re_buf == NULL)
-        goto done;
-    im_buf = PyOS_double_to_string(im, (char)type, precision, flags,
-                                   &im_float_type);
-    if (im_buf == NULL)
-        goto done;
-
-    n_re_digits = strlen(re_buf);
-    n_im_digits = strlen(im_buf);
-
-    /* Since there is no unicode version of PyOS_double_to_string,
-       just use the 8 bit version and then convert to unicode. */
-#if STRINGLIB_IS_UNICODE
-    re_unicode_tmp = (Py_UNICODE*)PyMem_Malloc((n_re_digits)*sizeof(Py_UNICODE));
-    if (re_unicode_tmp == NULL) {
-        PyErr_NoMemory();
-        goto done;
-    }
-    strtounicode(re_unicode_tmp, re_buf, n_re_digits);
-    p_re = re_unicode_tmp;
-
-    im_unicode_tmp = (Py_UNICODE*)PyMem_Malloc((n_im_digits)*sizeof(Py_UNICODE));
-    if (im_unicode_tmp == NULL) {
-        PyErr_NoMemory();
-        goto done;
-    }
-    strtounicode(im_unicode_tmp, im_buf, n_im_digits);
-    p_im = im_unicode_tmp;
-#else
-    p_re = re_buf;
-    p_im = im_buf;
-#endif
-
-    /* Is a sign character present in the output?  If so, remember it
-       and skip it */
-    if (*p_re == '-') {
-        re_sign_char = *p_re;
-        ++p_re;
-        --n_re_digits;
-    }
-    if (*p_im == '-') {
-        im_sign_char = *p_im;
-        ++p_im;
-        --n_im_digits;
-    }
-
-    /* Determine if we have any "remainder" (after the digits, might include
-       decimal or exponent or both (or neither)) */
-    parse_number(p_re, n_re_digits, &n_re_remainder, &re_has_decimal);
-    parse_number(p_im, n_im_digits, &n_im_remainder, &im_has_decimal);
-
-    /* Determine the grouping, separator, and decimal point, if any. */
-    get_locale_info(format->type == 'n' ? LT_CURRENT_LOCALE :
-                    (format->thousands_separators ?
-                     LT_DEFAULT_LOCALE :
-                     LT_NO_LOCALE),
-                    &locale);
-
-    /* Turn off any padding. We'll do it later after we've composed
-       the numbers without padding. */
-    tmp_format.fill_char = '\0';
-    tmp_format.align = '<';
-    tmp_format.width = -1;
-
-    /* Calculate how much memory we'll need. */
-    n_re_total = calc_number_widths(&re_spec, 0, re_sign_char, p_re,
-                                    n_re_digits, n_re_remainder,
-                                    re_has_decimal, &locale, &tmp_format);
-
-    /* Same formatting, but always include a sign, unless the real part is
-     * going to be omitted, in which case we use whatever sign convention was
-     * requested by the original format. */
-    if (!skip_re)
-        tmp_format.sign = '+';
-    n_im_total = calc_number_widths(&im_spec, 0, im_sign_char, p_im,
-                                    n_im_digits, n_im_remainder,
-                                    im_has_decimal, &locale, &tmp_format);
-
-    if (skip_re)
-        n_re_total = 0;
-
-    /* Add 1 for the 'j', and optionally 2 for parens. */
-    calc_padding(n_re_total + n_im_total + 1 + add_parens * 2,
-                 format->width, format->align, &lpad, &rpad, &total);
-
-    result = STRINGLIB_NEW(NULL, total);
-    if (result == NULL)
-        goto done;
-
-    /* Populate the memory. First, the padding. */
-    p = fill_padding(STRINGLIB_STR(result),
-                     n_re_total + n_im_total + 1 + add_parens * 2,
-                     format->fill_char=='\0' ? ' ' : format->fill_char,
-                     lpad, rpad);
-
-    if (add_parens)
-        *p++ = '(';
-
-    if (!skip_re) {
-        fill_number(p, &re_spec, p_re, n_re_digits, NULL, 0, &locale, 0);
-        p += n_re_total;
-    }
-    fill_number(p, &im_spec, p_im, n_im_digits, NULL, 0, &locale, 0);
-    p += n_im_total;
-    *p++ = 'j';
-
-    if (add_parens)
-        *p++ = ')';
-
-done:
-    PyMem_Free(re_buf);
-    PyMem_Free(im_buf);
-#if STRINGLIB_IS_UNICODE
-    PyMem_Free(re_unicode_tmp);
-    PyMem_Free(im_unicode_tmp);
-#endif
-    return result;
-}
-#endif /* FORMAT_COMPLEX */
-
-/************************************************************************/
-/*********** built in formatters ****************************************/
-/************************************************************************/
-PyObject *
-FORMAT_STRING(PyObject *obj,
-              STRINGLIB_CHAR *format_spec,
-              Py_ssize_t format_spec_len)
-{
-    InternalFormatSpec format;
-    PyObject *result = NULL;
-
-    /* check for the special case of zero length format spec, make
-       it equivalent to str(obj) */
-    if (format_spec_len == 0) {
-        result = STRINGLIB_TOSTR(obj);
-        goto done;
-    }
-
-    /* parse the format_spec */
-    if (!parse_internal_render_format_spec(format_spec, format_spec_len,
-                                           &format, 's', '<'))
-        goto done;
-
-    /* type conversion? */
-    switch (format.type) {
-    case 's':
-        /* no type conversion needed, already a string.  do the formatting */
-        result = format_string_internal(obj, &format);
-        break;
-    default:
-        /* unknown */
-        unknown_presentation_type(format.type, obj->ob_type->tp_name);
-        goto done;
-    }
-
-done:
-    return result;
-}
-
-#if defined FORMAT_LONG || defined FORMAT_INT
-static PyObject*
-format_int_or_long(PyObject* obj,
-                   STRINGLIB_CHAR *format_spec,
-                   Py_ssize_t format_spec_len,
-                   IntOrLongToString tostring)
-{
-    PyObject *result = NULL;
-    PyObject *tmp = NULL;
-    InternalFormatSpec format;
-
-    /* check for the special case of zero length format spec, make
-       it equivalent to str(obj) */
-    if (format_spec_len == 0) {
-        result = STRINGLIB_TOSTR(obj);
-        goto done;
-    }
-
-    /* parse the format_spec */
-    if (!parse_internal_render_format_spec(format_spec,
-                                           format_spec_len,
-                                           &format, 'd', '>'))
-        goto done;
-
-    /* type conversion? */
-    switch (format.type) {
-    case 'b':
-    case 'c':
-    case 'd':
-    case 'o':
-    case 'x':
-    case 'X':
-    case 'n':
-        /* no type conversion needed, already an int (or long).  do
-           the formatting */
-            result = format_int_or_long_internal(obj, &format, tostring);
-        break;
-
-    case 'e':
-    case 'E':
-    case 'f':
-    case 'F':
-    case 'g':
-    case 'G':
-    case '%':
-        /* convert to float */
-        tmp = PyNumber_Float(obj);
-        if (tmp == NULL)
-            goto done;
-        result = format_float_internal(tmp, &format);
-        break;
-
-    default:
-        /* unknown */
-        unknown_presentation_type(format.type, obj->ob_type->tp_name);
-        goto done;
-    }
-
-done:
-    Py_XDECREF(tmp);
-    return result;
-}
-#endif /* FORMAT_LONG || defined FORMAT_INT */
-
-#ifdef FORMAT_LONG
-/* Need to define long_format as a function that will convert a long
-   to a string.  In 3.0, _PyLong_Format has the correct signature.  In
-   2.x, we need to fudge a few parameters */
-#if PY_VERSION_HEX >= 0x03000000
-#define long_format _PyLong_Format
-#else
-static PyObject*
-long_format(PyObject* value, int base)
-{
-    /* Convert to base, don't add trailing 'L', and use the new octal
-       format. We already know this is a long object */
-    assert(PyLong_Check(value));
-    /* convert to base, don't add 'L', and use the new octal format */
-    return _PyLong_Format(value, base, 0, 1);
-}
-#endif
-
-PyObject *
-FORMAT_LONG(PyObject *obj,
-            STRINGLIB_CHAR *format_spec,
-            Py_ssize_t format_spec_len)
-{
-    return format_int_or_long(obj, format_spec, format_spec_len,
-                              long_format);
-}
-#endif /* FORMAT_LONG */
-
-#ifdef FORMAT_INT
-/* this is only used for 2.x, not 3.0 */
-static PyObject*
-int_format(PyObject* value, int base)
-{
-    /* Convert to base, and use the new octal format. We already
-       know this is an int object */
-    assert(PyInt_Check(value));
-    return _PyInt_Format((PyIntObject*)value, base, 1);
-}
-
-PyObject *
-FORMAT_INT(PyObject *obj,
-           STRINGLIB_CHAR *format_spec,
-           Py_ssize_t format_spec_len)
-{
-    return format_int_or_long(obj, format_spec, format_spec_len,
-                              int_format);
-}
-#endif /* FORMAT_INT */
-
-#ifdef FORMAT_FLOAT
-PyObject *
-FORMAT_FLOAT(PyObject *obj,
-             STRINGLIB_CHAR *format_spec,
-             Py_ssize_t format_spec_len)
-{
-    PyObject *result = NULL;
-    InternalFormatSpec format;
-
-    /* check for the special case of zero length format spec, make
-       it equivalent to str(obj) */
-    if (format_spec_len == 0) {
-        result = STRINGLIB_TOSTR(obj);
-        goto done;
-    }
-
-    /* parse the format_spec */
-    if (!parse_internal_render_format_spec(format_spec,
-                                           format_spec_len,
-                                           &format, '\0', '>'))
-        goto done;
-
-    /* type conversion? */
-    switch (format.type) {
-    case '\0': /* No format code: like 'g', but with at least one decimal. */
-    case 'e':
-    case 'E':
-    case 'f':
-    case 'F':
-    case 'g':
-    case 'G':
-    case 'n':
-    case '%':
-        /* no conversion, already a float.  do the formatting */
-        result = format_float_internal(obj, &format);
-        break;
-
-    default:
-        /* unknown */
-        unknown_presentation_type(format.type, obj->ob_type->tp_name);
-        goto done;
-    }
-
-done:
-    return result;
-}
-#endif /* FORMAT_FLOAT */
-
-#ifdef FORMAT_COMPLEX
-PyObject *
-FORMAT_COMPLEX(PyObject *obj,
-               STRINGLIB_CHAR *format_spec,
-               Py_ssize_t format_spec_len)
-{
-    PyObject *result = NULL;
-    InternalFormatSpec format;
-
-    /* check for the special case of zero length format spec, make
-       it equivalent to str(obj) */
-    if (format_spec_len == 0) {
-        result = STRINGLIB_TOSTR(obj);
-        goto done;
-    }
-
-    /* parse the format_spec */
-    if (!parse_internal_render_format_spec(format_spec,
-                                           format_spec_len,
-                                           &format, '\0', '>'))
-        goto done;
-
-    /* type conversion? */
-    switch (format.type) {
-    case '\0': /* No format code: like 'g', but with at least one decimal. */
-    case 'e':
-    case 'E':
-    case 'f':
-    case 'F':
-    case 'g':
-    case 'G':
-    case 'n':
-        /* no conversion, already a complex.  do the formatting */
-        result = format_complex_internal(obj, &format);
-        break;
-
-    default:
-        /* unknown */
-        unknown_presentation_type(format.type, obj->ob_type->tp_name);
-        goto done;
-    }
-
-done:
-    return result;
-}
-#endif /* FORMAT_COMPLEX */
diff --git a/Objects/stringlib/localeutil.h b/Objects/stringlib/localeutil.h
--- a/Objects/stringlib/localeutil.h
+++ b/Objects/stringlib/localeutil.h
@@ -1,8 +1,5 @@
 /* stringlib: locale related helpers implementation */
 
-#ifndef STRINGLIB_LOCALEUTIL_H
-#define STRINGLIB_LOCALEUTIL_H
-
 #include <locale.h>
 
 #define MAX(x, y) ((x) < (y) ? (y) : (x))
@@ -12,10 +9,10 @@
     const char *grouping;
     char previous;
     Py_ssize_t i; /* Where we're currently pointing in grouping. */
-} GroupGenerator;
+} STRINGLIB(GroupGenerator);
 
 static void
-_GroupGenerator_init(GroupGenerator *self, const char *grouping)
+STRINGLIB(GroupGenerator_init)(STRINGLIB(GroupGenerator) *self, const char *grouping)
 {
     self->grouping = grouping;
     self->i = 0;
@@ -24,7 +21,7 @@
 
 /* Returns the next grouping, or 0 to signify end. */
 static Py_ssize_t
-_GroupGenerator_next(GroupGenerator *self)
+STRINGLIB(GroupGenerator_next)(STRINGLIB(GroupGenerator) *self)
 {
     /* Note that we don't really do much error checking here. If a
        grouping string contains just CHAR_MAX, for example, then just
@@ -48,13 +45,11 @@
 /* Fill in some digits, leading zeros, and thousands separator. All
    are optional, depending on when we're called. */
 static void
-fill(STRINGLIB_CHAR **digits_end, STRINGLIB_CHAR **buffer_end,
+STRINGLIB(fill)(STRINGLIB_CHAR **digits_end, STRINGLIB_CHAR **buffer_end,
      Py_ssize_t n_chars, Py_ssize_t n_zeros, const char* thousands_sep,
      Py_ssize_t thousands_sep_len)
 {
-#if STRINGLIB_IS_UNICODE
     Py_ssize_t i;
-#endif
 
     if (thousands_sep) {
         *buffer_end -= thousands_sep_len;
@@ -76,7 +71,8 @@
     memcpy(*buffer_end, *digits_end, n_chars * sizeof(STRINGLIB_CHAR));
 
     *buffer_end -= n_zeros;
-    STRINGLIB_FILL(*buffer_end, '0', n_zeros);
+    for (i = 0; i < n_zeros; i++)
+        (*buffer_end)[i] = '0';
 }
 
 /**
@@ -133,15 +129,15 @@
                                         be looked at */
     /* A generator that returns all of the grouping widths, until it
        returns 0. */
-    GroupGenerator groupgen;
-    _GroupGenerator_init(&groupgen, grouping);
+    STRINGLIB(GroupGenerator) groupgen;
+    STRINGLIB(GroupGenerator_init)(&groupgen, grouping);
 
     if (buffer) {
         buffer_end = buffer + n_buffer;
         digits_end = digits + n_digits;
     }
 
-    while ((l = _GroupGenerator_next(&groupgen)) > 0) {
+    while ((l = STRINGLIB(GroupGenerator_next)(&groupgen)) > 0) {
         l = MIN(l, MAX(MAX(remaining, min_width), 1));
         n_zeros = MAX(0, l - remaining);
         n_chars = MAX(0, MIN(remaining, l));
@@ -153,7 +149,7 @@
 
         if (buffer) {
             /* Copy into the output buffer. */
-            fill(&digits_end, &buffer_end, n_chars, n_zeros,
+            STRINGLIB(fill)(&digits_end, &buffer_end, n_chars, n_zeros,
                  use_separator ? thousands_sep : NULL, thousands_sep_len);
         }
 
@@ -180,7 +176,7 @@
         count += (use_separator ? thousands_sep_len : 0) + n_zeros + n_chars;
         if (buffer) {
             /* Copy into the output buffer. */
-            fill(&digits_end, &buffer_end, n_chars, n_zeros,
+            STRINGLIB(fill)(&digits_end, &buffer_end, n_chars, n_zeros,
                  use_separator ? thousands_sep : NULL, thousands_sep_len);
         }
     }
@@ -209,4 +205,3 @@
         return _Py_InsertThousandsGrouping(buffer, n_buffer, digits, n_digits,
                                            min_width, grouping, thousands_sep);
 }
-#endif /* STRINGLIB_LOCALEUTIL_H */
diff --git a/Objects/stringlib/partition.h b/Objects/stringlib/partition.h
--- a/Objects/stringlib/partition.h
+++ b/Objects/stringlib/partition.h
@@ -1,14 +1,11 @@
 /* stringlib: partition implementation */
 
-#ifndef STRINGLIB_PARTITION_H
-#define STRINGLIB_PARTITION_H
-
 #ifndef STRINGLIB_FASTSEARCH_H
 #error must include "stringlib/fastsearch.h" before including this module
 #endif
 
 Py_LOCAL_INLINE(PyObject*)
-stringlib_partition(PyObject* str_obj,
+STRINGLIB(partition)(PyObject* str_obj,
                     const STRINGLIB_CHAR* str, Py_ssize_t str_len,
                     PyObject* sep_obj,
                     const STRINGLIB_CHAR* sep, Py_ssize_t sep_len)
@@ -25,7 +22,7 @@
     if (!out)
         return NULL;
 
-    pos = fastsearch(str, str_len, sep, sep_len, -1, FAST_SEARCH);
+    pos = FASTSEARCH(str, str_len, sep, sep_len, -1, FAST_SEARCH);
 
     if (pos < 0) {
 #if STRINGLIB_MUTABLE
@@ -58,7 +55,7 @@
 }
 
 Py_LOCAL_INLINE(PyObject*)
-stringlib_rpartition(PyObject* str_obj,
+STRINGLIB(rpartition)(PyObject* str_obj,
                      const STRINGLIB_CHAR* str, Py_ssize_t str_len,
                      PyObject* sep_obj,
                      const STRINGLIB_CHAR* sep, Py_ssize_t sep_len)
@@ -75,7 +72,7 @@
     if (!out)
         return NULL;
 
-    pos = fastsearch(str, str_len, sep, sep_len, -1, FAST_RSEARCH);
+    pos = FASTSEARCH(str, str_len, sep, sep_len, -1, FAST_RSEARCH);
 
     if (pos < 0) {
 #if STRINGLIB_MUTABLE
@@ -107,4 +104,3 @@
     return out;
 }
 
-#endif
diff --git a/Objects/stringlib/split.h b/Objects/stringlib/split.h
--- a/Objects/stringlib/split.h
+++ b/Objects/stringlib/split.h
@@ -1,8 +1,5 @@
 /* stringlib: split implementation */
 
-#ifndef STRINGLIB_SPLIT_H
-#define STRINGLIB_SPLIT_H
-
 #ifndef STRINGLIB_FASTSEARCH_H
 #error must include "stringlib/fastsearch.h" before including this module
 #endif
@@ -54,7 +51,7 @@
 #define FIX_PREALLOC_SIZE(list) Py_SIZE(list) = count
 
 Py_LOCAL_INLINE(PyObject *)
-stringlib_split_whitespace(PyObject* str_obj,
+STRINGLIB(split_whitespace)(PyObject* str_obj,
                            const STRINGLIB_CHAR* str, Py_ssize_t str_len,
                            Py_ssize_t maxcount)
 {
@@ -102,7 +99,7 @@
 }
 
 Py_LOCAL_INLINE(PyObject *)
-stringlib_split_char(PyObject* str_obj,
+STRINGLIB(split_char)(PyObject* str_obj,
                      const STRINGLIB_CHAR* str, Py_ssize_t str_len,
                      const STRINGLIB_CHAR ch,
                      Py_ssize_t maxcount)
@@ -145,7 +142,7 @@
 }
 
 Py_LOCAL_INLINE(PyObject *)
-stringlib_split(PyObject* str_obj,
+STRINGLIB(split)(PyObject* str_obj,
                 const STRINGLIB_CHAR* str, Py_ssize_t str_len,
                 const STRINGLIB_CHAR* sep, Py_ssize_t sep_len,
                 Py_ssize_t maxcount)
@@ -158,7 +155,7 @@
         return NULL;
     }
     else if (sep_len == 1)
-        return stringlib_split_char(str_obj, str, str_len, sep[0], maxcount);
+        return STRINGLIB(split_char)(str_obj, str, str_len, sep[0], maxcount);
 
     list = PyList_New(PREALLOC_SIZE(maxcount));
     if (list == NULL)
@@ -166,7 +163,7 @@
 
     i = j = 0;
     while (maxcount-- > 0) {
-        pos = fastsearch(str+i, str_len-i, sep, sep_len, -1, FAST_SEARCH);
+        pos = FASTSEARCH(str+i, str_len-i, sep, sep_len, -1, FAST_SEARCH);
         if (pos < 0)
             break;
         j = i + pos;
@@ -193,7 +190,7 @@
 }
 
 Py_LOCAL_INLINE(PyObject *)
-stringlib_rsplit_whitespace(PyObject* str_obj,
+STRINGLIB(rsplit_whitespace)(PyObject* str_obj,
                             const STRINGLIB_CHAR* str, Py_ssize_t str_len,
                             Py_ssize_t maxcount)
 {
@@ -243,7 +240,7 @@
 }
 
 Py_LOCAL_INLINE(PyObject *)
-stringlib_rsplit_char(PyObject* str_obj,
+STRINGLIB(rsplit_char)(PyObject* str_obj,
                       const STRINGLIB_CHAR* str, Py_ssize_t str_len,
                       const STRINGLIB_CHAR ch,
                       Py_ssize_t maxcount)
@@ -287,7 +284,7 @@
 }
 
 Py_LOCAL_INLINE(PyObject *)
-stringlib_rsplit(PyObject* str_obj,
+STRINGLIB(rsplit)(PyObject* str_obj,
                  const STRINGLIB_CHAR* str, Py_ssize_t str_len,
                  const STRINGLIB_CHAR* sep, Py_ssize_t sep_len,
                  Py_ssize_t maxcount)
@@ -300,7 +297,7 @@
         return NULL;
     }
     else if (sep_len == 1)
-        return stringlib_rsplit_char(str_obj, str, str_len, sep[0], maxcount);
+        return STRINGLIB(rsplit_char)(str_obj, str, str_len, sep[0], maxcount);
 
     list = PyList_New(PREALLOC_SIZE(maxcount));
     if (list == NULL)
@@ -308,7 +305,7 @@
 
     j = str_len;
     while (maxcount-- > 0) {
-        pos = fastsearch(str, j, sep, sep_len, -1, FAST_RSEARCH);
+        pos = FASTSEARCH(str, j, sep, sep_len, -1, FAST_RSEARCH);
         if (pos < 0)
             break;
         SPLIT_ADD(str, pos + sep_len, j);
@@ -336,7 +333,7 @@
 }
 
 Py_LOCAL_INLINE(PyObject *)
-stringlib_splitlines(PyObject* str_obj,
+STRINGLIB(splitlines)(PyObject* str_obj,
                      const STRINGLIB_CHAR* str, Py_ssize_t str_len,
                      int keepends)
 {
@@ -391,4 +388,3 @@
     return NULL;
 }
 
-#endif
diff --git a/Objects/stringlib/string_format.h b/Objects/stringlib/string_format.h
deleted file mode 100644
--- a/Objects/stringlib/string_format.h
+++ /dev/null
@@ -1,1385 +0,0 @@
-/*
-    string_format.h -- implementation of string.format().
-
-    It uses the Objects/stringlib conventions, so that it can be
-    compiled for both unicode and string objects.
-*/
-
-
-/* Defines for Python 2.6 compatibility */
-#if PY_VERSION_HEX < 0x03000000
-#define PyLong_FromSsize_t _PyLong_FromSsize_t
-#endif
-
-/* Defines for more efficiently reallocating the string buffer */
-#define INITIAL_SIZE_INCREMENT 100
-#define SIZE_MULTIPLIER 2
-#define MAX_SIZE_INCREMENT  3200
-
-
-/************************************************************************/
-/***********   Global data structures and forward declarations  *********/
-/************************************************************************/
-
-/*
-   A SubString consists of the characters between two string or
-   unicode pointers.
-*/
-typedef struct {
-    STRINGLIB_CHAR *ptr;
-    STRINGLIB_CHAR *end;
-} SubString;
-
-
-typedef enum {
-    ANS_INIT,
-    ANS_AUTO,
-    ANS_MANUAL
-} AutoNumberState;   /* Keep track if we're auto-numbering fields */
-
-/* Keeps track of our auto-numbering state, and which number field we're on */
-typedef struct {
-    AutoNumberState an_state;
-    int an_field_number;
-} AutoNumber;
-
-
-/* forward declaration for recursion */
-static PyObject *
-build_string(SubString *input, PyObject *args, PyObject *kwargs,
-             int recursion_depth, AutoNumber *auto_number);
-
-
-
-/************************************************************************/
-/**************************  Utility  functions  ************************/
-/************************************************************************/
-
-static void
-AutoNumber_Init(AutoNumber *auto_number)
-{
-    auto_number->an_state = ANS_INIT;
-    auto_number->an_field_number = 0;
-}
-
-/* fill in a SubString from a pointer and length */
-Py_LOCAL_INLINE(void)
-SubString_init(SubString *str, STRINGLIB_CHAR *p, Py_ssize_t len)
-{
-    str->ptr = p;
-    if (p == NULL)
-        str->end = NULL;
-    else
-        str->end = str->ptr + len;
-}
-
-/* return a new string.  if str->ptr is NULL, return None */
-Py_LOCAL_INLINE(PyObject *)
-SubString_new_object(SubString *str)
-{
-    if (str->ptr == NULL) {
-        Py_INCREF(Py_None);
-        return Py_None;
-    }
-    return STRINGLIB_NEW(str->ptr, str->end - str->ptr);
-}
-
-/* return a new string.  if str->ptr is NULL, return None */
-Py_LOCAL_INLINE(PyObject *)
-SubString_new_object_or_empty(SubString *str)
-{
-    if (str->ptr == NULL) {
-        return STRINGLIB_NEW(NULL, 0);
-    }
-    return STRINGLIB_NEW(str->ptr, str->end - str->ptr);
-}
-
-/* Return 1 if an error has been detected switching between automatic
-   field numbering and manual field specification, else return 0. Set
-   ValueError on error. */
-static int
-autonumber_state_error(AutoNumberState state, int field_name_is_empty)
-{
-    if (state == ANS_MANUAL) {
-        if (field_name_is_empty) {
-            PyErr_SetString(PyExc_ValueError, "cannot switch from "
-                            "manual field specification to "
-                            "automatic field numbering");
-            return 1;
-        }
-    }
-    else {
-        if (!field_name_is_empty) {
-            PyErr_SetString(PyExc_ValueError, "cannot switch from "
-                            "automatic field numbering to "
-                            "manual field specification");
-            return 1;
-        }
-    }
-    return 0;
-}
-
-
-/************************************************************************/
-/***********    Output string management functions       ****************/
-/************************************************************************/
-
-typedef struct {
-    STRINGLIB_CHAR *ptr;
-    STRINGLIB_CHAR *end;
-    PyObject *obj;
-    Py_ssize_t size_increment;
-} OutputString;
-
-/* initialize an OutputString object, reserving size characters */
-static int
-output_initialize(OutputString *output, Py_ssize_t size)
-{
-    output->obj = STRINGLIB_NEW(NULL, size);
-    if (output->obj == NULL)
-        return 0;
-
-    output->ptr = STRINGLIB_STR(output->obj);
-    output->end = STRINGLIB_LEN(output->obj) + output->ptr;
-    output->size_increment = INITIAL_SIZE_INCREMENT;
-
-    return 1;
-}
-
-/*
-    output_extend reallocates the output string buffer.
-    It returns a status:  0 for a failed reallocation,
-    1 for success.
-*/
-
-static int
-output_extend(OutputString *output, Py_ssize_t count)
-{
-    STRINGLIB_CHAR *startptr = STRINGLIB_STR(output->obj);
-    Py_ssize_t curlen = output->ptr - startptr;
-    Py_ssize_t maxlen = curlen + count + output->size_increment;
-
-    if (STRINGLIB_RESIZE(&output->obj, maxlen) < 0)
-        return 0;
-    startptr = STRINGLIB_STR(output->obj);
-    output->ptr = startptr + curlen;
-    output->end = startptr + maxlen;
-    if (output->size_increment < MAX_SIZE_INCREMENT)
-        output->size_increment *= SIZE_MULTIPLIER;
-    return 1;
-}
-
-/*
-    output_data dumps characters into our output string
-    buffer.
-
-    In some cases, it has to reallocate the string.
-
-    It returns a status:  0 for a failed reallocation,
-    1 for success.
-*/
-static int
-output_data(OutputString *output, const STRINGLIB_CHAR *s, Py_ssize_t count)
-{
-    if ((count > output->end - output->ptr) && !output_extend(output, count))
-        return 0;
-    memcpy(output->ptr, s, count * sizeof(STRINGLIB_CHAR));
-    output->ptr += count;
-    return 1;
-}
-
-/************************************************************************/
-/***********  Format string parsing -- integers and identifiers *********/
-/************************************************************************/
-
-static Py_ssize_t
-get_integer(const SubString *str)
-{
-    Py_ssize_t accumulator = 0;
-    Py_ssize_t digitval;
-    Py_ssize_t oldaccumulator;
-    STRINGLIB_CHAR *p;
-
-    /* empty string is an error */
-    if (str->ptr >= str->end)
-        return -1;
-
-    for (p = str->ptr; p < str->end; p++) {
-        digitval = STRINGLIB_TODECIMAL(*p);
-        if (digitval < 0)
-            return -1;
-        /*
-           Detect possible overflow before it happens:
-
-              accumulator * 10 + digitval > PY_SSIZE_T_MAX if and only if
-              accumulator > (PY_SSIZE_T_MAX - digitval) / 10.
-        */
-        if (accumulator > (PY_SSIZE_T_MAX - digitval) / 10) {
-            PyErr_Format(PyExc_ValueError,
-                         "Too many decimal digits in format string");
-            return -1;
-        }
-        accumulator = accumulator * 10 + digitval;
-    }
-    return accumulator;
-}
-
-/************************************************************************/
-/******** Functions to get field objects and specification strings ******/
-/************************************************************************/
-
-/* do the equivalent of obj.name */
-static PyObject *
-getattr(PyObject *obj, SubString *name)
-{
-    PyObject *newobj;
-    PyObject *str = SubString_new_object(name);
-    if (str == NULL)
-        return NULL;
-    newobj = PyObject_GetAttr(obj, str);
-    Py_DECREF(str);
-    return newobj;
-}
-
-/* do the equivalent of obj[idx], where obj is a sequence */
-static PyObject *
-getitem_sequence(PyObject *obj, Py_ssize_t idx)
-{
-    return PySequence_GetItem(obj, idx);
-}
-
-/* do the equivalent of obj[idx], where obj is not a sequence */
-static PyObject *
-getitem_idx(PyObject *obj, Py_ssize_t idx)
-{
-    PyObject *newobj;
-    PyObject *idx_obj = PyLong_FromSsize_t(idx);
-    if (idx_obj == NULL)
-        return NULL;
-    newobj = PyObject_GetItem(obj, idx_obj);
-    Py_DECREF(idx_obj);
-    return newobj;
-}
-
-/* do the equivalent of obj[name] */
-static PyObject *
-getitem_str(PyObject *obj, SubString *name)
-{
-    PyObject *newobj;
-    PyObject *str = SubString_new_object(name);
-    if (str == NULL)
-        return NULL;
-    newobj = PyObject_GetItem(obj, str);
-    Py_DECREF(str);
-    return newobj;
-}
-
-typedef struct {
-    /* the entire string we're parsing.  we assume that someone else
-       is managing its lifetime, and that it will exist for the
-       lifetime of the iterator.  can be empty */
-    SubString str;
-
-    /* pointer to where we are inside field_name */
-    STRINGLIB_CHAR *ptr;
-} FieldNameIterator;
-
-
-static int
-FieldNameIterator_init(FieldNameIterator *self, STRINGLIB_CHAR *ptr,
-                       Py_ssize_t len)
-{
-    SubString_init(&self->str, ptr, len);
-    self->ptr = self->str.ptr;
-    return 1;
-}
-
-static int
-_FieldNameIterator_attr(FieldNameIterator *self, SubString *name)
-{
-    STRINGLIB_CHAR c;
-
-    name->ptr = self->ptr;
-
-    /* return everything until '.' or '[' */
-    while (self->ptr < self->str.end) {
-        switch (c = *self->ptr++) {
-        case '[':
-        case '.':
-            /* backup so that we this character will be seen next time */
-            self->ptr--;
-            break;
-        default:
-            continue;
-        }
-        break;
-    }
-    /* end of string is okay */
-    name->end = self->ptr;
-    return 1;
-}
-
-static int
-_FieldNameIterator_item(FieldNameIterator *self, SubString *name)
-{
-    int bracket_seen = 0;
-    STRINGLIB_CHAR c;
-
-    name->ptr = self->ptr;
-
-    /* return everything until ']' */
-    while (self->ptr < self->str.end) {
-        switch (c = *self->ptr++) {
-        case ']':
-            bracket_seen = 1;
-            break;
-        default:
-            continue;
-        }
-        break;
-    }
-    /* make sure we ended with a ']' */
-    if (!bracket_seen) {
-        PyErr_SetString(PyExc_ValueError, "Missing ']' in format string");
-        return 0;
-    }
-
-    /* end of string is okay */
-    /* don't include the ']' */
-    name->end = self->ptr-1;
-    return 1;
-}
-
-/* returns 0 on error, 1 on non-error termination, and 2 if it returns a value */
-static int
-FieldNameIterator_next(FieldNameIterator *self, int *is_attribute,
-                       Py_ssize_t *name_idx, SubString *name)
-{
-    /* check at end of input */
-    if (self->ptr >= self->str.end)
-        return 1;
-
-    switch (*self->ptr++) {
-    case '.':
-        *is_attribute = 1;
-        if (_FieldNameIterator_attr(self, name) == 0)
-            return 0;
-        *name_idx = -1;
-        break;
-    case '[':
-        *is_attribute = 0;
-        if (_FieldNameIterator_item(self, name) == 0)
-            return 0;
-        *name_idx = get_integer(name);
-        if (*name_idx == -1 && PyErr_Occurred())
-            return 0;
-        break;
-    default:
-        /* Invalid character follows ']' */
-        PyErr_SetString(PyExc_ValueError, "Only '.' or '[' may "
-                        "follow ']' in format field specifier");
-        return 0;
-    }
-
-    /* empty string is an error */
-    if (name->ptr == name->end) {
-        PyErr_SetString(PyExc_ValueError, "Empty attribute in format string");
-        return 0;
-    }
-
-    return 2;
-}
-
-
-/* input: field_name
-   output: 'first' points to the part before the first '[' or '.'
-           'first_idx' is -1 if 'first' is not an integer, otherwise
-                       it's the value of first converted to an integer
-           'rest' is an iterator to return the rest
-*/
-static int
-field_name_split(STRINGLIB_CHAR *ptr, Py_ssize_t len, SubString *first,
-                 Py_ssize_t *first_idx, FieldNameIterator *rest,
-                 AutoNumber *auto_number)
-{
-    STRINGLIB_CHAR c;
-    STRINGLIB_CHAR *p = ptr;
-    STRINGLIB_CHAR *end = ptr + len;
-    int field_name_is_empty;
-    int using_numeric_index;
-
-    /* find the part up until the first '.' or '[' */
-    while (p < end) {
-        switch (c = *p++) {
-        case '[':
-        case '.':
-            /* backup so that we this character is available to the
-               "rest" iterator */
-            p--;
-            break;
-        default:
-            continue;
-        }
-        break;
-    }
-
-    /* set up the return values */
-    SubString_init(first, ptr, p - ptr);
-    FieldNameIterator_init(rest, p, end - p);
-
-    /* see if "first" is an integer, in which case it's used as an index */
-    *first_idx = get_integer(first);
-    if (*first_idx == -1 && PyErr_Occurred())
-        return 0;
-
-    field_name_is_empty = first->ptr >= first->end;
-
-    /* If the field name is omitted or if we have a numeric index
-       specified, then we're doing numeric indexing into args. */
-    using_numeric_index = field_name_is_empty || *first_idx != -1;
-
-    /* We always get here exactly one time for each field we're
-       processing. And we get here in field order (counting by left
-       braces). So this is the perfect place to handle automatic field
-       numbering if the field name is omitted. */
-
-    /* Check if we need to do the auto-numbering. It's not needed if
-       we're called from string.Format routines, because it's handled
-       in that class by itself. */
-    if (auto_number) {
-        /* Initialize our auto numbering state if this is the first
-           time we're either auto-numbering or manually numbering. */
-        if (auto_number->an_state == ANS_INIT && using_numeric_index)
-            auto_number->an_state = field_name_is_empty ?
-                ANS_AUTO : ANS_MANUAL;
-
-        /* Make sure our state is consistent with what we're doing
-           this time through. Only check if we're using a numeric
-           index. */
-        if (using_numeric_index)
-            if (autonumber_state_error(auto_number->an_state,
-                                       field_name_is_empty))
-                return 0;
-        /* Zero length field means we want to do auto-numbering of the
-           fields. */
-        if (field_name_is_empty)
-            *first_idx = (auto_number->an_field_number)++;
-    }
-
-    return 1;
-}
-
-
-/*
-    get_field_object returns the object inside {}, before the
-    format_spec.  It handles getindex and getattr lookups and consumes
-    the entire input string.
-*/
-static PyObject *
-get_field_object(SubString *input, PyObject *args, PyObject *kwargs,
-                 AutoNumber *auto_number)
-{
-    PyObject *obj = NULL;
-    int ok;
-    int is_attribute;
-    SubString name;
-    SubString first;
-    Py_ssize_t index;
-    FieldNameIterator rest;
-
-    if (!field_name_split(input->ptr, input->end - input->ptr, &first,
-                          &index, &rest, auto_number)) {
-        goto error;
-    }
-
-    if (index == -1) {
-        /* look up in kwargs */
-        PyObject *key = SubString_new_object(&first);
-        if (key == NULL)
-            goto error;
-
-        /* Use PyObject_GetItem instead of PyDict_GetItem because this
-           code is no longer just used with kwargs. It might be passed
-           a non-dict when called through format_map. */
-        if ((kwargs == NULL) || (obj = PyObject_GetItem(kwargs, key)) == NULL) {
-            PyErr_SetObject(PyExc_KeyError, key);
-            Py_DECREF(key);
-            goto error;
-        }
-        Py_DECREF(key);
-    }
-    else {
-        /* If args is NULL, we have a format string with a positional field
-           with only kwargs to retrieve it from. This can only happen when
-           used with format_map(), where positional arguments are not
-           allowed. */
-        if (args == NULL) {
-            PyErr_SetString(PyExc_ValueError, "Format string contains "
-                            "positional fields");
-            goto error;
-        }
-
-        /* look up in args */
-        obj = PySequence_GetItem(args, index);
-        if (obj == NULL)
-            goto error;
-    }
-
-    /* iterate over the rest of the field_name */
-    while ((ok = FieldNameIterator_next(&rest, &is_attribute, &index,
-                                        &name)) == 2) {
-        PyObject *tmp;
-
-        if (is_attribute)
-            /* getattr lookup "." */
-            tmp = getattr(obj, &name);
-        else
-            /* getitem lookup "[]" */
-            if (index == -1)
-                tmp = getitem_str(obj, &name);
-            else
-                if (PySequence_Check(obj))
-                    tmp = getitem_sequence(obj, index);
-                else
-                    /* not a sequence */
-                    tmp = getitem_idx(obj, index);
-        if (tmp == NULL)
-            goto error;
-
-        /* assign to obj */
-        Py_DECREF(obj);
-        obj = tmp;
-    }
-    /* end of iterator, this is the non-error case */
-    if (ok == 1)
-        return obj;
-error:
-    Py_XDECREF(obj);
-    return NULL;
-}
-
-/************************************************************************/
-/*****************  Field rendering functions  **************************/
-/************************************************************************/
-
-/*
-    render_field() is the main function in this section.  It takes the
-    field object and field specification string generated by
-    get_field_and_spec, and renders the field into the output string.
-
-    render_field calls fieldobj.__format__(format_spec) method, and
-    appends to the output.
-*/
-static int
-render_field(PyObject *fieldobj, SubString *format_spec, OutputString *output)
-{
-    int ok = 0;
-    PyObject *result = NULL;
-    PyObject *format_spec_object = NULL;
-    PyObject *(*formatter)(PyObject *, STRINGLIB_CHAR *, Py_ssize_t) = NULL;
-    STRINGLIB_CHAR* format_spec_start = format_spec->ptr ?
-            format_spec->ptr : NULL;
-    Py_ssize_t format_spec_len = format_spec->ptr ?
-            format_spec->end - format_spec->ptr : 0;
-
-    /* If we know the type exactly, skip the lookup of __format__ and just
-       call the formatter directly. */
-    if (PyUnicode_CheckExact(fieldobj))
-        formatter = _PyUnicode_FormatAdvanced;
-    else if (PyLong_CheckExact(fieldobj))
-        formatter =_PyLong_FormatAdvanced;
-    else if (PyFloat_CheckExact(fieldobj))
-        formatter = _PyFloat_FormatAdvanced;
-
-    /* XXX: for 2.6, convert format_spec to the appropriate type
-       (unicode, str) */
-
-    if (formatter) {
-        /* we know exactly which formatter will be called when __format__ is
-           looked up, so call it directly, instead. */
-        result = formatter(fieldobj, format_spec_start, format_spec_len);
-    }
-    else {
-        /* We need to create an object out of the pointers we have, because
-           __format__ takes a string/unicode object for format_spec. */
-        format_spec_object = STRINGLIB_NEW(format_spec_start,
-                                           format_spec_len);
-        if (format_spec_object == NULL)
-            goto done;
-
-        result = PyObject_Format(fieldobj, format_spec_object);
-    }
-    if (result == NULL)
-        goto done;
-
-#if PY_VERSION_HEX >= 0x03000000
-    assert(PyUnicode_Check(result));
-#else
-    assert(PyBytes_Check(result) || PyUnicode_Check(result));
-
-    /* Convert result to our type.  We could be str, and result could
-       be unicode */
-    {
-        PyObject *tmp = STRINGLIB_TOSTR(result);
-        if (tmp == NULL)
-            goto done;
-        Py_DECREF(result);
-        result = tmp;
-    }
-#endif
-
-    ok = output_data(output,
-                     STRINGLIB_STR(result), STRINGLIB_LEN(result));
-done:
-    Py_XDECREF(format_spec_object);
-    Py_XDECREF(result);
-    return ok;
-}
-
-static int
-parse_field(SubString *str, SubString *field_name, SubString *format_spec,
-            STRINGLIB_CHAR *conversion)
-{
-    /* Note this function works if the field name is zero length,
-       which is good.  Zero length field names are handled later, in
-       field_name_split. */
-
-    STRINGLIB_CHAR c = 0;
-
-    /* initialize these, as they may be empty */
-    *conversion = '\0';
-    SubString_init(format_spec, NULL, 0);
-
-    /* Search for the field name.  it's terminated by the end of
-       the string, or a ':' or '!' */
-    field_name->ptr = str->ptr;
-    while (str->ptr < str->end) {
-        switch (c = *(str->ptr++)) {
-        case ':':
-        case '!':
-            break;
-        default:
-            continue;
-        }
-        break;
-    }
-
-    if (c == '!' || c == ':') {
-        /* we have a format specifier and/or a conversion */
-        /* don't include the last character */
-        field_name->end = str->ptr-1;
-
-        /* the format specifier is the rest of the string */
-        format_spec->ptr = str->ptr;
-        format_spec->end = str->end;
-
-        /* see if there's a conversion specifier */
-        if (c == '!') {
-            /* there must be another character present */
-            if (format_spec->ptr >= format_spec->end) {
-                PyErr_SetString(PyExc_ValueError,
-                                "end of format while looking for conversion "
-                                "specifier");
-                return 0;
-            }
-            *conversion = *(format_spec->ptr++);
-
-            /* if there is another character, it must be a colon */
-            if (format_spec->ptr < format_spec->end) {
-                c = *(format_spec->ptr++);
-                if (c != ':') {
-                    PyErr_SetString(PyExc_ValueError,
-                                    "expected ':' after format specifier");
-                    return 0;
-                }
-            }
-        }
-    }
-    else
-        /* end of string, there's no format_spec or conversion */
-        field_name->end = str->ptr;
-
-    return 1;
-}
-
-/************************************************************************/
-/******* Output string allocation and escape-to-markup processing  ******/
-/************************************************************************/
-
-/* MarkupIterator breaks the string into pieces of either literal
-   text, or things inside {} that need to be marked up.  it is
-   designed to make it easy to wrap a Python iterator around it, for
-   use with the Formatter class */
-
-typedef struct {
-    SubString str;
-} MarkupIterator;
-
-static int
-MarkupIterator_init(MarkupIterator *self, STRINGLIB_CHAR *ptr, Py_ssize_t len)
-{
-    SubString_init(&self->str, ptr, len);
-    return 1;
-}
-
-/* returns 0 on error, 1 on non-error termination, and 2 if it got a
-   string (or something to be expanded) */
-static int
-MarkupIterator_next(MarkupIterator *self, SubString *literal,
-                    int *field_present, SubString *field_name,
-                    SubString *format_spec, STRINGLIB_CHAR *conversion,
-                    int *format_spec_needs_expanding)
-{
-    int at_end;
-    STRINGLIB_CHAR c = 0;
-    STRINGLIB_CHAR *start;
-    int count;
-    Py_ssize_t len;
-    int markup_follows = 0;
-
-    /* initialize all of the output variables */
-    SubString_init(literal, NULL, 0);
-    SubString_init(field_name, NULL, 0);
-    SubString_init(format_spec, NULL, 0);
-    *conversion = '\0';
-    *format_spec_needs_expanding = 0;
-    *field_present = 0;
-
-    /* No more input, end of iterator.  This is the normal exit
-       path. */
-    if (self->str.ptr >= self->str.end)
-        return 1;
-
-    start = self->str.ptr;
-
-    /* First read any literal text. Read until the end of string, an
-       escaped '{' or '}', or an unescaped '{'.  In order to never
-       allocate memory and so I can just pass pointers around, if
-       there's an escaped '{' or '}' then we'll return the literal
-       including the brace, but no format object.  The next time
-       through, we'll return the rest of the literal, skipping past
-       the second consecutive brace. */
-    while (self->str.ptr < self->str.end) {
-        switch (c = *(self->str.ptr++)) {
-        case '{':
-        case '}':
-            markup_follows = 1;
-            break;
-        default:
-            continue;
-        }
-        break;
-    }
-
-    at_end = self->str.ptr >= self->str.end;
-    len = self->str.ptr - start;
-
-    if ((c == '}') && (at_end || (c != *self->str.ptr))) {
-        PyErr_SetString(PyExc_ValueError, "Single '}' encountered "
-                        "in format string");
-        return 0;
-    }
-    if (at_end && c == '{') {
-        PyErr_SetString(PyExc_ValueError, "Single '{' encountered "
-                        "in format string");
-        return 0;
-    }
-    if (!at_end) {
-        if (c == *self->str.ptr) {
-            /* escaped } or {, skip it in the input.  there is no
-               markup object following us, just this literal text */
-            self->str.ptr++;
-            markup_follows = 0;
-        }
-        else
-            len--;
-    }
-
-    /* record the literal text */
-    literal->ptr = start;
-    literal->end = start + len;
-
-    if (!markup_follows)
-        return 2;
-
-    /* this is markup, find the end of the string by counting nested
-       braces.  note that this prohibits escaped braces, so that
-       format_specs cannot have braces in them. */
-    *field_present = 1;
-    count = 1;
-
-    start = self->str.ptr;
-
-    /* we know we can't have a zero length string, so don't worry
-       about that case */
-    while (self->str.ptr < self->str.end) {
-        switch (c = *(self->str.ptr++)) {
-        case '{':
-            /* the format spec needs to be recursively expanded.
-               this is an optimization, and not strictly needed */
-            *format_spec_needs_expanding = 1;
-            count++;
-            break;
-        case '}':
-            count--;
-            if (count <= 0) {
-                /* we're done.  parse and get out */
-                SubString s;
-
-                SubString_init(&s, start, self->str.ptr - 1 - start);
-                if (parse_field(&s, field_name, format_spec, conversion) == 0)
-                    return 0;
-
-                /* success */
-                return 2;
-            }
-            break;
-        }
-    }
-
-    /* end of string while searching for matching '}' */
-    PyErr_SetString(PyExc_ValueError, "unmatched '{' in format");
-    return 0;
-}
-
-
-/* do the !r or !s conversion on obj */
-static PyObject *
-do_conversion(PyObject *obj, STRINGLIB_CHAR conversion)
-{
-    /* XXX in pre-3.0, do we need to convert this to unicode, since it
-       might have returned a string? */
-    switch (conversion) {
-    case 'r':
-        return PyObject_Repr(obj);
-    case 's':
-        return STRINGLIB_TOSTR(obj);
-#if PY_VERSION_HEX >= 0x03000000
-    case 'a':
-        return STRINGLIB_TOASCII(obj);
-#endif
-    default:
-        if (conversion > 32 && conversion < 127) {
-                /* It's the ASCII subrange; casting to char is safe
-                   (assuming the execution character set is an ASCII
-                   superset). */
-                PyErr_Format(PyExc_ValueError,
-                     "Unknown conversion specifier %c",
-                     (char)conversion);
-        } else
-                PyErr_Format(PyExc_ValueError,
-                     "Unknown conversion specifier \\x%x",
-                     (unsigned int)conversion);
-        return NULL;
-    }
-}
-
-/* given:
-
-   {field_name!conversion:format_spec}
-
-   compute the result and write it to output.
-   format_spec_needs_expanding is an optimization.  if it's false,
-   just output the string directly, otherwise recursively expand the
-   format_spec string.
-
-   field_name is allowed to be zero length, in which case we
-   are doing auto field numbering.
-*/
-
-static int
-output_markup(SubString *field_name, SubString *format_spec,
-              int format_spec_needs_expanding, STRINGLIB_CHAR conversion,
-              OutputString *output, PyObject *args, PyObject *kwargs,
-              int recursion_depth, AutoNumber *auto_number)
-{
-    PyObject *tmp = NULL;
-    PyObject *fieldobj = NULL;
-    SubString expanded_format_spec;
-    SubString *actual_format_spec;
-    int result = 0;
-
-    /* convert field_name to an object */
-    fieldobj = get_field_object(field_name, args, kwargs, auto_number);
-    if (fieldobj == NULL)
-        goto done;
-
-    if (conversion != '\0') {
-        tmp = do_conversion(fieldobj, conversion);
-        if (tmp == NULL)
-            goto done;
-
-        /* do the assignment, transferring ownership: fieldobj = tmp */
-        Py_DECREF(fieldobj);
-        fieldobj = tmp;
-        tmp = NULL;
-    }
-
-    /* if needed, recurively compute the format_spec */
-    if (format_spec_needs_expanding) {
-        tmp = build_string(format_spec, args, kwargs, recursion_depth-1,
-                           auto_number);
-        if (tmp == NULL)
-            goto done;
-
-        /* note that in the case we're expanding the format string,
-           tmp must be kept around until after the call to
-           render_field. */
-        SubString_init(&expanded_format_spec,
-                       STRINGLIB_STR(tmp), STRINGLIB_LEN(tmp));
-        actual_format_spec = &expanded_format_spec;
-    }
-    else
-        actual_format_spec = format_spec;
-
-    if (render_field(fieldobj, actual_format_spec, output) == 0)
-        goto done;
-
-    result = 1;
-
-done:
-    Py_XDECREF(fieldobj);
-    Py_XDECREF(tmp);
-
-    return result;
-}
-
-/*
-    do_markup is the top-level loop for the format() method.  It
-    searches through the format string for escapes to markup codes, and
-    calls other functions to move non-markup text to the output,
-    and to perform the markup to the output.
-*/
-static int
-do_markup(SubString *input, PyObject *args, PyObject *kwargs,
-          OutputString *output, int recursion_depth, AutoNumber *auto_number)
-{
-    MarkupIterator iter;
-    int format_spec_needs_expanding;
-    int result;
-    int field_present;
-    SubString literal;
-    SubString field_name;
-    SubString format_spec;
-    STRINGLIB_CHAR conversion;
-
-    MarkupIterator_init(&iter, input->ptr, input->end - input->ptr);
-    while ((result = MarkupIterator_next(&iter, &literal, &field_present,
-                                         &field_name, &format_spec,
-                                         &conversion,
-                                         &format_spec_needs_expanding)) == 2) {
-        if (!output_data(output, literal.ptr, literal.end - literal.ptr))
-            return 0;
-        if (field_present)
-            if (!output_markup(&field_name, &format_spec,
-                               format_spec_needs_expanding, conversion, output,
-                               args, kwargs, recursion_depth, auto_number))
-                return 0;
-    }
-    return result;
-}
-
-
-/*
-    build_string allocates the output string and then
-    calls do_markup to do the heavy lifting.
-*/
-static PyObject *
-build_string(SubString *input, PyObject *args, PyObject *kwargs,
-             int recursion_depth, AutoNumber *auto_number)
-{
-    OutputString output;
-    PyObject *result = NULL;
-    Py_ssize_t count;
-
-    output.obj = NULL; /* needed so cleanup code always works */
-
-    /* check the recursion level */
-    if (recursion_depth <= 0) {
-        PyErr_SetString(PyExc_ValueError,
-                        "Max string recursion exceeded");
-        goto done;
-    }
-
-    /* initial size is the length of the format string, plus the size
-       increment.  seems like a reasonable default */
-    if (!output_initialize(&output,
-                           input->end - input->ptr +
-                           INITIAL_SIZE_INCREMENT))
-        goto done;
-
-    if (!do_markup(input, args, kwargs, &output, recursion_depth,
-                   auto_number)) {
-        goto done;
-    }
-
-    count = output.ptr - STRINGLIB_STR(output.obj);
-    if (STRINGLIB_RESIZE(&output.obj, count) < 0) {
-        goto done;
-    }
-
-    /* transfer ownership to result */
-    result = output.obj;
-    output.obj = NULL;
-
-done:
-    Py_XDECREF(output.obj);
-    return result;
-}
-
-/************************************************************************/
-/*********** main routine ***********************************************/
-/************************************************************************/
-
-/* this is the main entry point */
-static PyObject *
-do_string_format(PyObject *self, PyObject *args, PyObject *kwargs)
-{
-    SubString input;
-
-    /* PEP 3101 says only 2 levels, so that
-       "{0:{1}}".format('abc', 's')            # works
-       "{0:{1:{2}}}".format('abc', 's', '')    # fails
-    */
-    int recursion_depth = 2;
-
-    AutoNumber auto_number;
-
-    AutoNumber_Init(&auto_number);
-    SubString_init(&input, STRINGLIB_STR(self), STRINGLIB_LEN(self));
-    return build_string(&input, args, kwargs, recursion_depth, &auto_number);
-}
-
-static PyObject *
-do_string_format_map(PyObject *self, PyObject *obj)
-{
-    return do_string_format(self, NULL, obj);
-}
-
-
-/************************************************************************/
-/*********** formatteriterator ******************************************/
-/************************************************************************/
-
-/* This is used to implement string.Formatter.vparse().  It exists so
-   Formatter can share code with the built in unicode.format() method.
-   It's really just a wrapper around MarkupIterator that is callable
-   from Python. */
-
-typedef struct {
-    PyObject_HEAD
-
-    STRINGLIB_OBJECT *str;
-
-    MarkupIterator it_markup;
-} formatteriterobject;
-
-static void
-formatteriter_dealloc(formatteriterobject *it)
-{
-    Py_XDECREF(it->str);
-    PyObject_FREE(it);
-}
-
-/* returns a tuple:
-   (literal, field_name, format_spec, conversion)
-
-   literal is any literal text to output.  might be zero length
-   field_name is the string before the ':'.  might be None
-   format_spec is the string after the ':'.  mibht be None
-   conversion is either None, or the string after the '!'
-*/
-static PyObject *
-formatteriter_next(formatteriterobject *it)
-{
-    SubString literal;
-    SubString field_name;
-    SubString format_spec;
-    STRINGLIB_CHAR conversion;
-    int format_spec_needs_expanding;
-    int field_present;
-    int result = MarkupIterator_next(&it->it_markup, &literal, &field_present,
-                                     &field_name, &format_spec, &conversion,
-                                     &format_spec_needs_expanding);
-
-    /* all of the SubString objects point into it->str, so no
-       memory management needs to be done on them */
-    assert(0 <= result && result <= 2);
-    if (result == 0 || result == 1)
-        /* if 0, error has already been set, if 1, iterator is empty */
-        return NULL;
-    else {
-        PyObject *literal_str = NULL;
-        PyObject *field_name_str = NULL;
-        PyObject *format_spec_str = NULL;
-        PyObject *conversion_str = NULL;
-        PyObject *tuple = NULL;
-
-        literal_str = SubString_new_object(&literal);
-        if (literal_str == NULL)
-            goto done;
-
-        field_name_str = SubString_new_object(&field_name);
-        if (field_name_str == NULL)
-            goto done;
-
-        /* if field_name is non-zero length, return a string for
-           format_spec (even if zero length), else return None */
-        format_spec_str = (field_present ?
-                           SubString_new_object_or_empty :
-                           SubString_new_object)(&format_spec);
-        if (format_spec_str == NULL)
-            goto done;
-
-        /* if the conversion is not specified, return a None,
-           otherwise create a one length string with the conversion
-           character */
-        if (conversion == '\0') {
-            conversion_str = Py_None;
-            Py_INCREF(conversion_str);
-        }
-        else
-            conversion_str = STRINGLIB_NEW(&conversion, 1);
-        if (conversion_str == NULL)
-            goto done;
-
-        tuple = PyTuple_Pack(4, literal_str, field_name_str, format_spec_str,
-                             conversion_str);
-    done:
-        Py_XDECREF(literal_str);
-        Py_XDECREF(field_name_str);
-        Py_XDECREF(format_spec_str);
-        Py_XDECREF(conversion_str);
-        return tuple;
-    }
-}
-
-static PyMethodDef formatteriter_methods[] = {
-    {NULL,              NULL}           /* sentinel */
-};
-
-static PyTypeObject PyFormatterIter_Type = {
-    PyVarObject_HEAD_INIT(&PyType_Type, 0)
-    "formatteriterator",                /* tp_name */
-    sizeof(formatteriterobject),        /* tp_basicsize */
-    0,                                  /* tp_itemsize */
-    /* methods */
-    (destructor)formatteriter_dealloc,  /* tp_dealloc */
-    0,                                  /* tp_print */
-    0,                                  /* tp_getattr */
-    0,                                  /* tp_setattr */
-    0,                                  /* tp_reserved */
-    0,                                  /* tp_repr */
-    0,                                  /* tp_as_number */
-    0,                                  /* tp_as_sequence */
-    0,                                  /* tp_as_mapping */
-    0,                                  /* tp_hash */
-    0,                                  /* tp_call */
-    0,                                  /* tp_str */
-    PyObject_GenericGetAttr,            /* tp_getattro */
-    0,                                  /* tp_setattro */
-    0,                                  /* tp_as_buffer */
-    Py_TPFLAGS_DEFAULT,                 /* tp_flags */
-    0,                                  /* tp_doc */
-    0,                                  /* tp_traverse */
-    0,                                  /* tp_clear */
-    0,                                  /* tp_richcompare */
-    0,                                  /* tp_weaklistoffset */
-    PyObject_SelfIter,                  /* tp_iter */
-    (iternextfunc)formatteriter_next,   /* tp_iternext */
-    formatteriter_methods,              /* tp_methods */
-    0,
-};
-
-/* unicode_formatter_parser is used to implement
-   string.Formatter.vformat.  it parses a string and returns tuples
-   describing the parsed elements.  It's a wrapper around
-   stringlib/string_format.h's MarkupIterator */
-static PyObject *
-formatter_parser(PyObject *ignored, STRINGLIB_OBJECT *self)
-{
-    formatteriterobject *it;
-
-    if (!PyUnicode_Check(self)) {
-        PyErr_Format(PyExc_TypeError, "expected str, got %s", Py_TYPE(self)->tp_name);
-        return NULL;
-    }
-
-    it = PyObject_New(formatteriterobject, &PyFormatterIter_Type);
-    if (it == NULL)
-        return NULL;
-
-    /* take ownership, give the object to the iterator */
-    Py_INCREF(self);
-    it->str = self;
-
-    /* initialize the contained MarkupIterator */
-    MarkupIterator_init(&it->it_markup,
-                        STRINGLIB_STR(self),
-                        STRINGLIB_LEN(self));
-
-    return (PyObject *)it;
-}
-
-
-/************************************************************************/
-/*********** fieldnameiterator ******************************************/
-/************************************************************************/
-
-
-/* This is used to implement string.Formatter.vparse().  It parses the
-   field name into attribute and item values.  It's a Python-callable
-   wrapper around FieldNameIterator */
-
-typedef struct {
-    PyObject_HEAD
-
-    STRINGLIB_OBJECT *str;
-
-    FieldNameIterator it_field;
-} fieldnameiterobject;
-
-static void
-fieldnameiter_dealloc(fieldnameiterobject *it)
-{
-    Py_XDECREF(it->str);
-    PyObject_FREE(it);
-}
-
-/* returns a tuple:
-   (is_attr, value)
-   is_attr is true if we used attribute syntax (e.g., '.foo')
-              false if we used index syntax (e.g., '[foo]')
-   value is an integer or string
-*/
-static PyObject *
-fieldnameiter_next(fieldnameiterobject *it)
-{
-    int result;
-    int is_attr;
-    Py_ssize_t idx;
-    SubString name;
-
-    result = FieldNameIterator_next(&it->it_field, &is_attr,
-                                    &idx, &name);
-    if (result == 0 || result == 1)
-        /* if 0, error has already been set, if 1, iterator is empty */
-        return NULL;
-    else {
-        PyObject* result = NULL;
-        PyObject* is_attr_obj = NULL;
-        PyObject* obj = NULL;
-
-        is_attr_obj = PyBool_FromLong(is_attr);
-        if (is_attr_obj == NULL)
-            goto done;
-
-        /* either an integer or a string */
-        if (idx != -1)
-            obj = PyLong_FromSsize_t(idx);
-        else
-            obj = SubString_new_object(&name);
-        if (obj == NULL)
-            goto done;
-
-        /* return a tuple of values */
-        result = PyTuple_Pack(2, is_attr_obj, obj);
-
-    done:
-        Py_XDECREF(is_attr_obj);
-        Py_XDECREF(obj);
-        return result;
-    }
-}
-
-static PyMethodDef fieldnameiter_methods[] = {
-    {NULL,              NULL}           /* sentinel */
-};
-
-static PyTypeObject PyFieldNameIter_Type = {
-    PyVarObject_HEAD_INIT(&PyType_Type, 0)
-    "fieldnameiterator",                /* tp_name */
-    sizeof(fieldnameiterobject),        /* tp_basicsize */
-    0,                                  /* tp_itemsize */
-    /* methods */
-    (destructor)fieldnameiter_dealloc,  /* tp_dealloc */
-    0,                                  /* tp_print */
-    0,                                  /* tp_getattr */
-    0,                                  /* tp_setattr */
-    0,                                  /* tp_reserved */
-    0,                                  /* tp_repr */
-    0,                                  /* tp_as_number */
-    0,                                  /* tp_as_sequence */
-    0,                                  /* tp_as_mapping */
-    0,                                  /* tp_hash */
-    0,                                  /* tp_call */
-    0,                                  /* tp_str */
-    PyObject_GenericGetAttr,            /* tp_getattro */
-    0,                                  /* tp_setattro */
-    0,                                  /* tp_as_buffer */
-    Py_TPFLAGS_DEFAULT,                 /* tp_flags */
-    0,                                  /* tp_doc */
-    0,                                  /* tp_traverse */
-    0,                                  /* tp_clear */
-    0,                                  /* tp_richcompare */
-    0,                                  /* tp_weaklistoffset */
-    PyObject_SelfIter,                  /* tp_iter */
-    (iternextfunc)fieldnameiter_next,   /* tp_iternext */
-    fieldnameiter_methods,              /* tp_methods */
-    0};
-
-/* unicode_formatter_field_name_split is used to implement
-   string.Formatter.vformat.  it takes an PEP 3101 "field name", and
-   returns a tuple of (first, rest): "first", the part before the
-   first '.' or '['; and "rest", an iterator for the rest of the field
-   name.  it's a wrapper around stringlib/string_format.h's
-   field_name_split.  The iterator it returns is a
-   FieldNameIterator */
-static PyObject *
-formatter_field_name_split(PyObject *ignored, STRINGLIB_OBJECT *self)
-{
-    SubString first;
-    Py_ssize_t first_idx;
-    fieldnameiterobject *it;
-
-    PyObject *first_obj = NULL;
-    PyObject *result = NULL;
-
-    if (!PyUnicode_Check(self)) {
-        PyErr_Format(PyExc_TypeError, "expected str, got %s", Py_TYPE(self)->tp_name);
-        return NULL;
-    }
-
-    it = PyObject_New(fieldnameiterobject, &PyFieldNameIter_Type);
-    if (it == NULL)
-        return NULL;
-
-    /* take ownership, give the object to the iterator.  this is
-       just to keep the field_name alive */
-    Py_INCREF(self);
-    it->str = self;
-
-    /* Pass in auto_number = NULL. We'll return an empty string for
-       first_obj in that case. */
-    if (!field_name_split(STRINGLIB_STR(self),
-                          STRINGLIB_LEN(self),
-                          &first, &first_idx, &it->it_field, NULL))
-        goto done;
-
-    /* first becomes an integer, if possible; else a string */
-    if (first_idx != -1)
-        first_obj = PyLong_FromSsize_t(first_idx);
-    else
-        /* convert "first" into a string object */
-        first_obj = SubString_new_object(&first);
-    if (first_obj == NULL)
-        goto done;
-
-    /* return a tuple of values */
-    result = PyTuple_Pack(2, first_obj, it);
-
-done:
-    Py_XDECREF(it);
-    Py_XDECREF(first_obj);
-    return result;
-}
diff --git a/Objects/stringlib/stringdefs.h b/Objects/stringlib/stringdefs.h
--- a/Objects/stringlib/stringdefs.h
+++ b/Objects/stringlib/stringdefs.h
@@ -6,6 +6,8 @@
    compiled as unicode. */
 #define STRINGLIB_IS_UNICODE     0
 
+#define FASTSEARCH fastsearch
+#define STRINGLIB(F) stringlib_##F
 #define STRINGLIB_OBJECT         PyBytesObject
 #define STRINGLIB_CHAR           char
 #define STRINGLIB_TYPE_NAME      "string"
diff --git a/Objects/stringlib/ucs1lib.h b/Objects/stringlib/ucs1lib.h
new file mode 100644
--- /dev/null
+++ b/Objects/stringlib/ucs1lib.h
@@ -0,0 +1,35 @@
+/* this is sort of a hack.  there's at least one place (formatting
+   floats) where some stringlib code takes a different path if it's
+   compiled as unicode. */
+#define STRINGLIB_IS_UNICODE     1
+
+#define FASTSEARCH               ucs1lib_fastsearch
+#define STRINGLIB(F)             ucs1lib_##F
+#define STRINGLIB_OBJECT         PyUnicodeObject
+#define STRINGLIB_CHAR           Py_UCS1
+#define STRINGLIB_TYPE_NAME      "unicode"
+#define STRINGLIB_PARSE_CODE     "U"
+#define STRINGLIB_EMPTY          unicode_empty
+#define STRINGLIB_ISSPACE        Py_UNICODE_ISSPACE
+#define STRINGLIB_ISLINEBREAK    BLOOM_LINEBREAK
+#define STRINGLIB_ISDECIMAL      Py_UNICODE_ISDECIMAL
+#define STRINGLIB_TODECIMAL      Py_UNICODE_TODECIMAL
+#define STRINGLIB_TOUPPER        Py_UNICODE_TOUPPER
+#define STRINGLIB_TOLOWER        Py_UNICODE_TOLOWER
+#define STRINGLIB_FILL           Py_UNICODE_FILL
+#define STRINGLIB_STR            PyUnicode_1BYTE_DATA
+#define STRINGLIB_LEN            PyUnicode_GET_LENGTH
+#define STRINGLIB_NEW            PyUnicode_FromUCS1
+#define STRINGLIB_RESIZE         not_supported
+#define STRINGLIB_CHECK          PyUnicode_Check
+#define STRINGLIB_CHECK_EXACT    PyUnicode_CheckExact
+#define STRINGLIB_GROUPING       _PyUnicode_InsertThousandsGrouping
+#define STRINGLIB_GROUPING_LOCALE _PyUnicode_InsertThousandsGroupingLocale
+
+#define STRINGLIB_TOSTR          PyObject_Str
+#define STRINGLIB_TOASCII        PyObject_ASCII
+
+#define _Py_InsertThousandsGrouping _PyUnicode_ucs1_InsertThousandsGrouping
+#define _Py_InsertThousandsGroupingLocale _PyUnicode_ucs1_InsertThousandsGroupingLocale
+
+
diff --git a/Objects/stringlib/ucs2lib.h b/Objects/stringlib/ucs2lib.h
new file mode 100644
--- /dev/null
+++ b/Objects/stringlib/ucs2lib.h
@@ -0,0 +1,34 @@
+/* this is sort of a hack.  there's at least one place (formatting
+   floats) where some stringlib code takes a different path if it's
+   compiled as unicode. */
+#define STRINGLIB_IS_UNICODE     1
+
+#define FASTSEARCH               ucs2lib_fastsearch
+#define STRINGLIB(F)             ucs2lib_##F
+#define STRINGLIB_OBJECT         PyUnicodeObject
+#define STRINGLIB_CHAR           Py_UCS2
+#define STRINGLIB_TYPE_NAME      "unicode"
+#define STRINGLIB_PARSE_CODE     "U"
+#define STRINGLIB_EMPTY          unicode_empty
+#define STRINGLIB_ISSPACE        Py_UNICODE_ISSPACE
+#define STRINGLIB_ISLINEBREAK    BLOOM_LINEBREAK
+#define STRINGLIB_ISDECIMAL      Py_UNICODE_ISDECIMAL
+#define STRINGLIB_TODECIMAL      Py_UNICODE_TODECIMAL
+#define STRINGLIB_TOUPPER        Py_UNICODE_TOUPPER
+#define STRINGLIB_TOLOWER        Py_UNICODE_TOLOWER
+#define STRINGLIB_FILL           Py_UNICODE_FILL
+#define STRINGLIB_STR            PyUnicode_1BYTE_DATA
+#define STRINGLIB_LEN            PyUnicode_GET_LENGTH
+#define STRINGLIB_NEW            PyUnicode_FromUCS2
+#define STRINGLIB_RESIZE         not_supported
+#define STRINGLIB_CHECK          PyUnicode_Check
+#define STRINGLIB_CHECK_EXACT    PyUnicode_CheckExact
+#define STRINGLIB_GROUPING       _PyUnicode_InsertThousandsGrouping
+#define STRINGLIB_GROUPING_LOCALE _PyUnicode_InsertThousandsGroupingLocale
+
+#define STRINGLIB_TOSTR          PyObject_Str
+#define STRINGLIB_TOASCII        PyObject_ASCII
+
+#define _Py_InsertThousandsGrouping _PyUnicode_ucs2_InsertThousandsGrouping
+#define _Py_InsertThousandsGroupingLocale _PyUnicode_ucs2_InsertThousandsGroupingLocale
+
diff --git a/Objects/stringlib/ucs4lib.h b/Objects/stringlib/ucs4lib.h
new file mode 100644
--- /dev/null
+++ b/Objects/stringlib/ucs4lib.h
@@ -0,0 +1,34 @@
+/* this is sort of a hack.  there's at least one place (formatting
+   floats) where some stringlib code takes a different path if it's
+   compiled as unicode. */
+#define STRINGLIB_IS_UNICODE     1
+
+#define FASTSEARCH               ucs4lib_fastsearch
+#define STRINGLIB(F)             ucs4lib_##F
+#define STRINGLIB_OBJECT         PyUnicodeObject
+#define STRINGLIB_CHAR           Py_UCS4
+#define STRINGLIB_TYPE_NAME      "unicode"
+#define STRINGLIB_PARSE_CODE     "U"
+#define STRINGLIB_EMPTY          unicode_empty
+#define STRINGLIB_ISSPACE        Py_UNICODE_ISSPACE
+#define STRINGLIB_ISLINEBREAK    BLOOM_LINEBREAK
+#define STRINGLIB_ISDECIMAL      Py_UNICODE_ISDECIMAL
+#define STRINGLIB_TODECIMAL      Py_UNICODE_TODECIMAL
+#define STRINGLIB_TOUPPER        Py_UNICODE_TOUPPER
+#define STRINGLIB_TOLOWER        Py_UNICODE_TOLOWER
+#define STRINGLIB_FILL           Py_UNICODE_FILL
+#define STRINGLIB_STR            PyUnicode_1BYTE_DATA
+#define STRINGLIB_LEN            PyUnicode_GET_LENGTH
+#define STRINGLIB_NEW            PyUnicode_FromUCS4
+#define STRINGLIB_RESIZE         not_supported
+#define STRINGLIB_CHECK          PyUnicode_Check
+#define STRINGLIB_CHECK_EXACT    PyUnicode_CheckExact
+#define STRINGLIB_GROUPING       _PyUnicode_InsertThousandsGrouping
+#define STRINGLIB_GROUPING_LOCALE _PyUnicode_InsertThousandsGroupingLocale
+
+#define STRINGLIB_TOSTR          PyObject_Str
+#define STRINGLIB_TOASCII        PyObject_ASCII
+
+#define _Py_InsertThousandsGrouping _PyUnicode_ucs4_InsertThousandsGrouping
+#define _Py_InsertThousandsGroupingLocale _PyUnicode_ucs4_InsertThousandsGroupingLocale
+
diff --git a/Objects/stringlib/undef.h b/Objects/stringlib/undef.h
new file mode 100644
--- /dev/null
+++ b/Objects/stringlib/undef.h
@@ -0,0 +1,10 @@
+#undef  FASTSEARCH
+#undef  STRINGLIB
+#undef  STRINGLIB_CHAR
+#undef  STRINGLIB_STR
+#undef  STRINGLIB_LEN
+#undef  STRINGLIB_NEW
+#undef  STRINGLIB_RESIZE
+#undef  _Py_InsertThousandsGrouping
+#undef  _Py_InsertThousandsGroupingLocale
+
diff --git a/Objects/stringlib/unicode_format.h b/Objects/stringlib/unicode_format.h
new file mode 100644
--- /dev/null
+++ b/Objects/stringlib/unicode_format.h
@@ -0,0 +1,1416 @@
+/*
+    unicode_format.h -- implementation of str.format().
+*/
+
+
+/* Defines for more efficiently reallocating the string buffer */
+#define INITIAL_SIZE_INCREMENT 100
+#define SIZE_MULTIPLIER 2
+#define MAX_SIZE_INCREMENT  3200
+
+
+/************************************************************************/
+/***********   Global data structures and forward declarations  *********/
+/************************************************************************/
+
+/*
+   A SubString consists of the characters between two string or
+   unicode pointers.
+*/
+typedef struct {
+    PyObject *str; /* borrowed reference */
+    Py_ssize_t start, end;
+} SubString;
+
+
+typedef enum {
+    ANS_INIT,
+    ANS_AUTO,
+    ANS_MANUAL
+} AutoNumberState;   /* Keep track if we're auto-numbering fields */
+
+/* Keeps track of our auto-numbering state, and which number field we're on */
+typedef struct {
+    AutoNumberState an_state;
+    int an_field_number;
+} AutoNumber;
+
+
+/* forward declaration for recursion */
+static PyObject *
+build_string(SubString *input, PyObject *args, PyObject *kwargs,
+             int recursion_depth, AutoNumber *auto_number);
+
+
+
+/************************************************************************/
+/**************************  Utility  functions  ************************/
+/************************************************************************/
+
+static void
+AutoNumber_Init(AutoNumber *auto_number)
+{
+    auto_number->an_state = ANS_INIT;
+    auto_number->an_field_number = 0;
+}
+
+/* fill in a SubString from a pointer and length */
+Py_LOCAL_INLINE(void)
+SubString_init(SubString *str, PyObject *s, int start, int end)
+{
+    str->str = s;
+    str->start = start;
+    str->end = end;
+}
+
+/* return a new string.  if str->str is NULL, return None */
+Py_LOCAL_INLINE(PyObject *)
+SubString_new_object(SubString *str)
+{
+    if (str->str == NULL) {
+        Py_INCREF(Py_None);
+        return Py_None;
+    }
+    return PyUnicode_Substring(str->str, str->start, str->end);
+}
+
+/* return a new string.  if str->str is NULL, return None */
+Py_LOCAL_INLINE(PyObject *)
+SubString_new_object_or_empty(SubString *str)
+{
+    if (str->str == NULL) {
+        return PyUnicode_FromUnicode(NULL, 0);
+    }
+    return SubString_new_object(str);
+}
+
+/* Return 1 if an error has been detected switching between automatic
+   field numbering and manual field specification, else return 0. Set
+   ValueError on error. */
+static int
+autonumber_state_error(AutoNumberState state, int field_name_is_empty)
+{
+    if (state == ANS_MANUAL) {
+        if (field_name_is_empty) {
+            PyErr_SetString(PyExc_ValueError, "cannot switch from "
+                            "manual field specification to "
+                            "automatic field numbering");
+            return 1;
+        }
+    }
+    else {
+        if (!field_name_is_empty) {
+            PyErr_SetString(PyExc_ValueError, "cannot switch from "
+                            "automatic field numbering to "
+                            "manual field specification");
+            return 1;
+        }
+    }
+    return 0;
+}
+
+
+/************************************************************************/
+/***********    Output string management functions       ****************/
+/************************************************************************/
+
+typedef struct {
+    char *data;
+    Py_UCS4 maxchar;
+    unsigned int kind;
+    Py_ssize_t pos, size;
+    Py_ssize_t size_increment;
+} OutputString;
+
+/* initialize an OutputString object, reserving size characters */
+static int
+output_initialize(OutputString *output, Py_ssize_t size)
+{
+    output->data = PyMem_Malloc(size);
+    if (output->data == NULL) {
+        PyErr_NoMemory();
+        return 0;
+    }
+
+    output->maxchar = 127;
+    output->kind = PyUnicode_1BYTE_KIND;
+    output->pos = 0;
+    output->size = size;
+    output->size_increment = INITIAL_SIZE_INCREMENT;
+
+    return 1;
+}
+
+/*
+    output_extend reallocates the output string buffer.
+    It returns a status:  0 for a failed reallocation,
+    1 for success.
+*/
+
+static int
+output_extend(OutputString *output, Py_ssize_t count)
+{
+    Py_ssize_t maxlen = output->size + count + output->size_increment;
+
+    output->data = PyMem_Realloc(output->data, maxlen << (output->kind-1));
+    output->size = maxlen;
+    if (output->data == 0) {
+        PyErr_NoMemory();
+        return 0;
+    }
+    if (output->size_increment < MAX_SIZE_INCREMENT)
+        output->size_increment *= SIZE_MULTIPLIER;
+    return 1;
+}
+
+static int
+output_widen(OutputString *output, Py_UCS4 maxchar)
+{
+    int kind;
+    void *data; 
+    Py_ssize_t i;
+    if (maxchar <= output->maxchar)
+        return 1;
+    if (maxchar < 256) {
+        output->maxchar = 255;
+        return 1;
+    }
+    if (maxchar < 65536) {
+        output->maxchar = 65535;
+        kind = 2;
+    }
+    else {
+        output->maxchar = 1<<21;
+        kind = 3;
+    }
+    data = PyMem_Malloc(output->size << (kind-1));
+    if (data == 0)
+        return 0;
+    for (i = 0; i < output->size; i++)
+        PyUnicode_WRITE(kind, data, i,
+                        PyUnicode_READ(output->kind, output->data, i));
+    PyMem_Free(output->data);
+    output->data = data;
+    output->kind = kind;
+    return 1;
+}
+
+/*
+    output_data dumps characters into our output string
+    buffer.
+
+    In some cases, it has to reallocate the string.
+
+    It returns a status:  0 for a failed reallocation,
+    1 for success.
+*/
+static int
+output_data(OutputString *output, PyObject *s, Py_ssize_t start, Py_ssize_t end)
+{
+    Py_ssize_t i;
+    int kind;
+    if ((output->pos + end - start > output->size) && 
+        !output_extend(output, end - start))
+        return 0;
+    kind = PyUnicode_KIND(s);
+    if (PyUnicode_MAX_CHAR_VALUE(s) > output->maxchar) {
+        Py_UCS4 maxchar = output->maxchar;
+        for (i = start; i < end; i++)
+            if (PyUnicode_READ(kind, PyUnicode_DATA(s), i) > maxchar)
+                maxchar = PyUnicode_READ(kind, PyUnicode_DATA(s), i);
+        if (!output_widen(output, maxchar))
+            return 0;
+    }
+    for (i = start; i < end; i++)
+        PyUnicode_WRITE(output->kind, output->data, output->pos++,
+                        PyUnicode_READ(kind, PyUnicode_DATA(s), i));
+    return 1;
+}
+
+/************************************************************************/
+/***********  Format string parsing -- integers and identifiers *********/
+/************************************************************************/
+
+static Py_ssize_t
+get_integer(const SubString *str)
+{
+    Py_ssize_t accumulator = 0;
+    Py_ssize_t digitval;
+    Py_ssize_t i;
+
+    /* empty string is an error */
+    if (str->start >= str->end)
+        return -1;
+
+    for (i = str->start; i < str->end; i++) {
+        digitval = Py_UNICODE_TODECIMAL(PyUnicode_READ_CHAR(str->str, i));
+        if (digitval < 0)
+            return -1;
+        /*
+           Detect possible overflow before it happens:
+
+              accumulator * 10 + digitval > PY_SSIZE_T_MAX if and only if
+              accumulator > (PY_SSIZE_T_MAX - digitval) / 10.
+        */
+        if (accumulator > (PY_SSIZE_T_MAX - digitval) / 10) {
+            PyErr_Format(PyExc_ValueError,
+                         "Too many decimal digits in format string");
+            return -1;
+        }
+        accumulator = accumulator * 10 + digitval;
+    }
+    return accumulator;
+}
+
+/************************************************************************/
+/******** Functions to get field objects and specification strings ******/
+/************************************************************************/
+
+/* do the equivalent of obj.name */
+static PyObject *
+getattr(PyObject *obj, SubString *name)
+{
+    PyObject *newobj;
+    PyObject *str = SubString_new_object(name);
+    if (str == NULL)
+        return NULL;
+    newobj = PyObject_GetAttr(obj, str);
+    Py_DECREF(str);
+    return newobj;
+}
+
+/* do the equivalent of obj[idx], where obj is a sequence */
+static PyObject *
+getitem_sequence(PyObject *obj, Py_ssize_t idx)
+{
+    return PySequence_GetItem(obj, idx);
+}
+
+/* do the equivalent of obj[idx], where obj is not a sequence */
+static PyObject *
+getitem_idx(PyObject *obj, Py_ssize_t idx)
+{
+    PyObject *newobj;
+    PyObject *idx_obj = PyLong_FromSsize_t(idx);
+    if (idx_obj == NULL)
+        return NULL;
+    newobj = PyObject_GetItem(obj, idx_obj);
+    Py_DECREF(idx_obj);
+    return newobj;
+}
+
+/* do the equivalent of obj[name] */
+static PyObject *
+getitem_str(PyObject *obj, SubString *name)
+{
+    PyObject *newobj;
+    PyObject *str = SubString_new_object(name);
+    if (str == NULL)
+        return NULL;
+    newobj = PyObject_GetItem(obj, str);
+    Py_DECREF(str);
+    return newobj;
+}
+
+typedef struct {
+    /* the entire string we're parsing.  we assume that someone else
+       is managing its lifetime, and that it will exist for the
+       lifetime of the iterator.  can be empty */
+    SubString str;
+
+    /* index to where we are inside field_name */
+    Py_ssize_t index;
+} FieldNameIterator;
+
+
+static int
+FieldNameIterator_init(FieldNameIterator *self, PyObject *s,
+                       Py_ssize_t start, Py_ssize_t end)
+{
+    SubString_init(&self->str, s, start, end);
+    self->index = start;
+    return 1;
+}
+
+static int
+_FieldNameIterator_attr(FieldNameIterator *self, SubString *name)
+{
+    Py_UCS4 c;
+
+    name->str = self->str.str;
+    name->start = self->index;
+
+    /* return everything until '.' or '[' */
+    while (self->index < self->str.end) {
+        c = PyUnicode_READ_CHAR(self->str.str, self->index++);
+        switch (c) {
+        case '[':
+        case '.':
+            /* backup so that we this character will be seen next time */
+            self->index--;
+            break;
+        default:
+            continue;
+        }
+        break;
+    }
+    /* end of string is okay */
+    name->end = self->index;
+    return 1;
+}
+
+static int
+_FieldNameIterator_item(FieldNameIterator *self, SubString *name)
+{
+    int bracket_seen = 0;
+    Py_UCS4 c;
+
+    name->str = self->str.str;
+    name->start = self->index;
+
+    /* return everything until ']' */
+    while (self->index < self->str.end) {
+        c = PyUnicode_READ_CHAR(self->str.str, self->index++);
+        switch (c) {
+        case ']':
+            bracket_seen = 1;
+            break;
+        default:
+            continue;
+        }
+        break;
+    }
+    /* make sure we ended with a ']' */
+    if (!bracket_seen) {
+        PyErr_SetString(PyExc_ValueError, "Missing ']' in format string");
+        return 0;
+    }
+
+    /* end of string is okay */
+    /* don't include the ']' */
+    name->end = self->index-1;
+    return 1;
+}
+
+/* returns 0 on error, 1 on non-error termination, and 2 if it returns a value */
+static int
+FieldNameIterator_next(FieldNameIterator *self, int *is_attribute,
+                       Py_ssize_t *name_idx, SubString *name)
+{
+    /* check at end of input */
+    if (self->index >= self->str.end)
+        return 1;
+
+    switch (PyUnicode_READ_CHAR(self->str.str, self->index++)) {
+    case '.':
+        *is_attribute = 1;
+        if (_FieldNameIterator_attr(self, name) == 0)
+            return 0;
+        *name_idx = -1;
+        break;
+    case '[':
+        *is_attribute = 0;
+        if (_FieldNameIterator_item(self, name) == 0)
+            return 0;
+        *name_idx = get_integer(name);
+        if (*name_idx == -1 && PyErr_Occurred())
+            return 0;
+        break;
+    default:
+        /* Invalid character follows ']' */
+        PyErr_SetString(PyExc_ValueError, "Only '.' or '[' may "
+                        "follow ']' in format field specifier");
+        return 0;
+    }
+
+    /* empty string is an error */
+    if (name->start == name->end) {
+        PyErr_SetString(PyExc_ValueError, "Empty attribute in format string");
+        return 0;
+    }
+
+    return 2;
+}
+
+
+/* input: field_name
+   output: 'first' points to the part before the first '[' or '.'
+           'first_idx' is -1 if 'first' is not an integer, otherwise
+                       it's the value of first converted to an integer
+           'rest' is an iterator to return the rest
+*/
+static int
+field_name_split(PyObject *str, Py_ssize_t start, Py_ssize_t end, SubString *first,
+                 Py_ssize_t *first_idx, FieldNameIterator *rest,
+                 AutoNumber *auto_number)
+{
+    Py_UCS4 c;
+    Py_ssize_t i = start;
+    int field_name_is_empty;
+    int using_numeric_index;
+
+    /* find the part up until the first '.' or '[' */
+    while (i < end) {
+        switch (c = PyUnicode_READ_CHAR(str, i++)) {
+        case '[':
+        case '.':
+            /* backup so that we this character is available to the
+               "rest" iterator */
+            i--;
+            break;
+        default:
+            continue;
+        }
+        break;
+    }
+
+    /* set up the return values */
+    SubString_init(first, str, start, i);
+    FieldNameIterator_init(rest, str, i, end);
+
+    /* see if "first" is an integer, in which case it's used as an index */
+    *first_idx = get_integer(first);
+    if (*first_idx == -1 && PyErr_Occurred())
+        return 0;
+
+    field_name_is_empty = first->start >= first->end;
+
+    /* If the field name is omitted or if we have a numeric index
+       specified, then we're doing numeric indexing into args. */
+    using_numeric_index = field_name_is_empty || *first_idx != -1;
+
+    /* We always get here exactly one time for each field we're
+       processing. And we get here in field order (counting by left
+       braces). So this is the perfect place to handle automatic field
+       numbering if the field name is omitted. */
+
+    /* Check if we need to do the auto-numbering. It's not needed if
+       we're called from string.Format routines, because it's handled
+       in that class by itself. */
+    if (auto_number) {
+        /* Initialize our auto numbering state if this is the first
+           time we're either auto-numbering or manually numbering. */
+        if (auto_number->an_state == ANS_INIT && using_numeric_index)
+            auto_number->an_state = field_name_is_empty ?
+                ANS_AUTO : ANS_MANUAL;
+
+        /* Make sure our state is consistent with what we're doing
+           this time through. Only check if we're using a numeric
+           index. */
+        if (using_numeric_index)
+            if (autonumber_state_error(auto_number->an_state,
+                                       field_name_is_empty))
+                return 0;
+        /* Zero length field means we want to do auto-numbering of the
+           fields. */
+        if (field_name_is_empty)
+            *first_idx = (auto_number->an_field_number)++;
+    }
+
+    return 1;
+}
+
+
+/*
+    get_field_object returns the object inside {}, before the
+    format_spec.  It handles getindex and getattr lookups and consumes
+    the entire input string.
+*/
+static PyObject *
+get_field_object(SubString *input, PyObject *args, PyObject *kwargs,
+                 AutoNumber *auto_number)
+{
+    PyObject *obj = NULL;
+    int ok;
+    int is_attribute;
+    SubString name;
+    SubString first;
+    Py_ssize_t index;
+    FieldNameIterator rest;
+
+    if (!field_name_split(input->str, input->start, input->end, &first,
+                          &index, &rest, auto_number)) {
+        goto error;
+    }
+
+    if (index == -1) {
+        /* look up in kwargs */
+        PyObject *key = SubString_new_object(&first);
+        if (key == NULL)
+            goto error;
+
+        /* Use PyObject_GetItem instead of PyDict_GetItem because this
+           code is no longer just used with kwargs. It might be passed
+           a non-dict when called through format_map. */
+        if ((kwargs == NULL) || (obj = PyObject_GetItem(kwargs, key)) == NULL) {
+            PyErr_SetObject(PyExc_KeyError, key);
+            Py_DECREF(key);
+            goto error;
+        }
+        Py_DECREF(key);
+    }
+    else {
+        /* If args is NULL, we have a format string with a positional field
+           with only kwargs to retrieve it from. This can only happen when
+           used with format_map(), where positional arguments are not
+           allowed. */
+        if (args == NULL) {
+            PyErr_SetString(PyExc_ValueError, "Format string contains "
+                            "positional fields");
+            goto error;
+        }
+
+        /* look up in args */
+        obj = PySequence_GetItem(args, index);
+        if (obj == NULL)
+            goto error;
+    }
+
+    /* iterate over the rest of the field_name */
+    while ((ok = FieldNameIterator_next(&rest, &is_attribute, &index,
+                                        &name)) == 2) {
+        PyObject *tmp;
+
+        if (is_attribute)
+            /* getattr lookup "." */
+            tmp = getattr(obj, &name);
+        else
+            /* getitem lookup "[]" */
+            if (index == -1)
+                tmp = getitem_str(obj, &name);
+            else
+                if (PySequence_Check(obj))
+                    tmp = getitem_sequence(obj, index);
+                else
+                    /* not a sequence */
+                    tmp = getitem_idx(obj, index);
+        if (tmp == NULL)
+            goto error;
+
+        /* assign to obj */
+        Py_DECREF(obj);
+        obj = tmp;
+    }
+    /* end of iterator, this is the non-error case */
+    if (ok == 1)
+        return obj;
+error:
+    Py_XDECREF(obj);
+    return NULL;
+}
+
+/************************************************************************/
+/*****************  Field rendering functions  **************************/
+/************************************************************************/
+
+/*
+    render_field() is the main function in this section.  It takes the
+    field object and field specification string generated by
+    get_field_and_spec, and renders the field into the output string.
+
+    render_field calls fieldobj.__format__(format_spec) method, and
+    appends to the output.
+*/
+static int
+render_field(PyObject *fieldobj, SubString *format_spec, OutputString *output)
+{
+    int ok = 0;
+    PyObject *result = NULL;
+    PyObject *format_spec_object = NULL;
+    PyObject *(*formatter)(PyObject *, PyObject *, Py_ssize_t, Py_ssize_t) = NULL;
+    
+    /* If we know the type exactly, skip the lookup of __format__ and just
+       call the formatter directly. */
+    if (PyUnicode_CheckExact(fieldobj))
+        formatter = _PyUnicode_FormatAdvanced;
+    else if (PyLong_CheckExact(fieldobj))
+        formatter =_PyLong_FormatAdvanced;
+    else if (PyFloat_CheckExact(fieldobj))
+        formatter = _PyFloat_FormatAdvanced;
+
+    /* XXX: for 2.6, convert format_spec to the appropriate type
+       (unicode, str) */
+
+    if (formatter) {
+        /* we know exactly which formatter will be called when __format__ is
+           looked up, so call it directly, instead. */
+        result = formatter(fieldobj, format_spec->str,
+                           format_spec->start, format_spec->end);
+    }
+    else {
+        /* We need to create an object out of the pointers we have, because
+           __format__ takes a string/unicode object for format_spec. */
+        if (format_spec->str)
+            format_spec_object = PyUnicode_Substring(format_spec->str,
+                                                     format_spec->start,
+                                                     format_spec->end);
+        else
+            format_spec_object = PyUnicode_New(0, 0);
+        if (format_spec_object == NULL)
+            goto done;
+
+        result = PyObject_Format(fieldobj, format_spec_object);
+    }
+    if (result == NULL || PyUnicode_READY(result) == -1)
+        goto done;
+
+    assert(PyUnicode_Check(result));
+    ok = output_data(output, result, 0, PyUnicode_GET_LENGTH(result));
+done:
+    Py_XDECREF(format_spec_object);
+    Py_XDECREF(result);
+    return ok;
+}
+
+static int
+parse_field(SubString *str, SubString *field_name, SubString *format_spec,
+            Py_UCS4 *conversion)
+{
+    /* Note this function works if the field name is zero length,
+       which is good.  Zero length field names are handled later, in
+       field_name_split. */
+
+    Py_UCS4 c = 0;
+
+    /* initialize these, as they may be empty */
+    *conversion = '\0';
+    SubString_init(format_spec, NULL, 0, 0);
+
+    /* Search for the field name.  it's terminated by the end of
+       the string, or a ':' or '!' */
+    field_name->str = str->str;
+    field_name->start = str->start;
+    while (str->start < str->end) {
+        switch ((c = PyUnicode_READ_CHAR(str->str, str->start++))) {
+        case ':':
+        case '!':
+            break;
+        default:
+            continue;
+        }
+        break;
+    }
+
+    if (c == '!' || c == ':') {
+        /* we have a format specifier and/or a conversion */
+        /* don't include the last character */
+        field_name->end = str->start-1;
+
+        /* the format specifier is the rest of the string */
+        format_spec->str = str->str;
+        format_spec->start = str->start;
+        format_spec->end = str->end;
+
+        /* see if there's a conversion specifier */
+        if (c == '!') {
+            /* there must be another character present */
+            if (format_spec->start >= format_spec->end) {
+                PyErr_SetString(PyExc_ValueError,
+                                "end of format while looking for conversion "
+                                "specifier");
+                return 0;
+            }
+            *conversion = PyUnicode_READ_CHAR(format_spec->str, format_spec->start++);
+
+            /* if there is another character, it must be a colon */
+            if (format_spec->start < format_spec->end) {
+                c = PyUnicode_READ_CHAR(format_spec->str, format_spec->start++);
+                if (c != ':') {
+                    PyErr_SetString(PyExc_ValueError,
+                                    "expected ':' after format specifier");
+                    return 0;
+                }
+            }
+        }
+    }
+    else
+        /* end of string, there's no format_spec or conversion */
+        field_name->end = str->start;
+
+    return 1;
+}
+
+/************************************************************************/
+/******* Output string allocation and escape-to-markup processing  ******/
+/************************************************************************/
+
+/* MarkupIterator breaks the string into pieces of either literal
+   text, or things inside {} that need to be marked up.  it is
+   designed to make it easy to wrap a Python iterator around it, for
+   use with the Formatter class */
+
+typedef struct {
+    SubString str;
+} MarkupIterator;
+
+static int
+MarkupIterator_init(MarkupIterator *self, PyObject *str, 
+                    Py_ssize_t start, Py_ssize_t end)
+{
+    SubString_init(&self->str, str, start, end);
+    return 1;
+}
+
+/* returns 0 on error, 1 on non-error termination, and 2 if it got a
+   string (or something to be expanded) */
+static int
+MarkupIterator_next(MarkupIterator *self, SubString *literal,
+                    int *field_present, SubString *field_name,
+                    SubString *format_spec, Py_UCS4 *conversion,
+                    int *format_spec_needs_expanding)
+{
+    int at_end;
+    Py_UCS4 c = 0;
+    Py_ssize_t start;
+    int count;
+    Py_ssize_t len;
+    int markup_follows = 0;
+
+    /* initialize all of the output variables */
+    SubString_init(literal, NULL, 0, 0);
+    SubString_init(field_name, NULL, 0, 0);
+    SubString_init(format_spec, NULL, 0, 0);
+    *conversion = '\0';
+    *format_spec_needs_expanding = 0;
+    *field_present = 0;
+
+    /* No more input, end of iterator.  This is the normal exit
+       path. */
+    if (self->str.start >= self->str.end)
+        return 1;
+
+    start = self->str.start;
+
+    /* First read any literal text. Read until the end of string, an
+       escaped '{' or '}', or an unescaped '{'.  In order to never
+       allocate memory and so I can just pass pointers around, if
+       there's an escaped '{' or '}' then we'll return the literal
+       including the brace, but no format object.  The next time
+       through, we'll return the rest of the literal, skipping past
+       the second consecutive brace. */
+    while (self->str.start < self->str.end) {
+        switch (c = PyUnicode_READ_CHAR(self->str.str, self->str.start++)) {
+        case '{':
+        case '}':
+            markup_follows = 1;
+            break;
+        default:
+            continue;
+        }
+        break;
+    }
+
+    at_end = self->str.start >= self->str.end;
+    len = self->str.start - start;
+
+    if ((c == '}') && (at_end || 
+                       (c != PyUnicode_READ_CHAR(self->str.str, 
+                                                 self->str.start)))) {
+        PyErr_SetString(PyExc_ValueError, "Single '}' encountered "
+                        "in format string");
+        return 0;
+    }
+    if (at_end && c == '{') {
+        PyErr_SetString(PyExc_ValueError, "Single '{' encountered "
+                        "in format string");
+        return 0;
+    }
+    if (!at_end) {
+        if (c == PyUnicode_READ_CHAR(self->str.str, self->str.start)) {
+            /* escaped } or {, skip it in the input.  there is no
+               markup object following us, just this literal text */
+            self->str.start++;
+            markup_follows = 0;
+        }
+        else
+            len--;
+    }
+
+    /* record the literal text */
+    literal->str = self->str.str;
+    literal->start = start;
+    literal->end = start + len;
+
+    if (!markup_follows)
+        return 2;
+
+    /* this is markup, find the end of the string by counting nested
+       braces.  note that this prohibits escaped braces, so that
+       format_specs cannot have braces in them. */
+    *field_present = 1;
+    count = 1;
+
+    start = self->str.start;
+
+    /* we know we can't have a zero length string, so don't worry
+       about that case */
+    while (self->str.start < self->str.end) {
+        switch (c = PyUnicode_READ_CHAR(self->str.str, self->str.start++)) {
+        case '{':
+            /* the format spec needs to be recursively expanded.
+               this is an optimization, and not strictly needed */
+            *format_spec_needs_expanding = 1;
+            count++;
+            break;
+        case '}':
+            count--;
+            if (count <= 0) {
+                /* we're done.  parse and get out */
+                SubString s;
+
+                SubString_init(&s, self->str.str, start, self->str.start - 1);
+                if (parse_field(&s, field_name, format_spec, conversion) == 0)
+                    return 0;
+
+                /* success */
+                return 2;
+            }
+            break;
+        }
+    }
+
+    /* end of string while searching for matching '}' */
+    PyErr_SetString(PyExc_ValueError, "unmatched '{' in format");
+    return 0;
+}
+
+
+/* do the !r or !s conversion on obj */
+static PyObject *
+do_conversion(PyObject *obj, Py_UCS4 conversion)
+{
+    /* XXX in pre-3.0, do we need to convert this to unicode, since it
+       might have returned a string? */
+    switch (conversion) {
+    case 'r':
+        return PyObject_Repr(obj);
+    case 's':
+        return PyObject_Str(obj);
+    case 'a':
+        return PyObject_ASCII(obj);
+    default:
+        if (conversion > 32 && conversion < 127) {
+                /* It's the ASCII subrange; casting to char is safe
+                   (assuming the execution character set is an ASCII
+                   superset). */
+                PyErr_Format(PyExc_ValueError,
+                     "Unknown conversion specifier %c",
+                     (char)conversion);
+        } else
+                PyErr_Format(PyExc_ValueError,
+                     "Unknown conversion specifier \\x%x",
+                     (unsigned int)conversion);
+        return NULL;
+    }
+}
+
+/* given:
+
+   {field_name!conversion:format_spec}
+
+   compute the result and write it to output.
+   format_spec_needs_expanding is an optimization.  if it's false,
+   just output the string directly, otherwise recursively expand the
+   format_spec string.
+
+   field_name is allowed to be zero length, in which case we
+   are doing auto field numbering.
+*/
+
+static int
+output_markup(SubString *field_name, SubString *format_spec,
+              int format_spec_needs_expanding, Py_UCS4 conversion,
+              OutputString *output, PyObject *args, PyObject *kwargs,
+              int recursion_depth, AutoNumber *auto_number)
+{
+    PyObject *tmp = NULL;
+    PyObject *fieldobj = NULL;
+    SubString expanded_format_spec;
+    SubString *actual_format_spec;
+    int result = 0;
+
+    /* convert field_name to an object */
+    fieldobj = get_field_object(field_name, args, kwargs, auto_number);
+    if (fieldobj == NULL)
+        goto done;
+
+    if (conversion != '\0') {
+        tmp = do_conversion(fieldobj, conversion);
+        if (tmp == NULL || PyUnicode_READY(tmp) == -1)
+            goto done;
+
+        /* do the assignment, transferring ownership: fieldobj = tmp */
+        Py_DECREF(fieldobj);
+        fieldobj = tmp;
+        tmp = NULL;
+    }
+
+    /* if needed, recurively compute the format_spec */
+    if (format_spec_needs_expanding) {
+        tmp = build_string(format_spec, args, kwargs, recursion_depth-1,
+                           auto_number);
+        if (tmp == NULL || PyUnicode_READY(tmp) == -1)
+            goto done;
+
+        /* note that in the case we're expanding the format string,
+           tmp must be kept around until after the call to
+           render_field. */
+        SubString_init(&expanded_format_spec, tmp, 0, PyUnicode_GET_LENGTH(tmp));
+        actual_format_spec = &expanded_format_spec;
+    }
+    else
+        actual_format_spec = format_spec;
+
+    if (render_field(fieldobj, actual_format_spec, output) == 0)
+        goto done;
+
+    result = 1;
+
+done:
+    Py_XDECREF(fieldobj);
+    Py_XDECREF(tmp);
+
+    return result;
+}
+
+/*
+    do_markup is the top-level loop for the format() method.  It
+    searches through the format string for escapes to markup codes, and
+    calls other functions to move non-markup text to the output,
+    and to perform the markup to the output.
+*/
+static int
+do_markup(SubString *input, PyObject *args, PyObject *kwargs,
+          OutputString *output, int recursion_depth, AutoNumber *auto_number)
+{
+    MarkupIterator iter;
+    int format_spec_needs_expanding;
+    int result;
+    int field_present;
+    SubString literal;
+    SubString field_name;
+    SubString format_spec;
+    Py_UCS4 conversion;
+
+    MarkupIterator_init(&iter, input->str, input->start, input->end);
+    while ((result = MarkupIterator_next(&iter, &literal, &field_present,
+                                         &field_name, &format_spec,
+                                         &conversion,
+                                         &format_spec_needs_expanding)) == 2) {
+        if (!output_data(output, literal.str, literal.start, literal.end))
+            return 0;
+        if (field_present)
+            if (!output_markup(&field_name, &format_spec,
+                               format_spec_needs_expanding, conversion, output,
+                               args, kwargs, recursion_depth, auto_number))
+                return 0;
+    }
+    return result;
+}
+
+
+/*
+    build_string allocates the output string and then
+    calls do_markup to do the heavy lifting.
+*/
+static PyObject *
+build_string(SubString *input, PyObject *args, PyObject *kwargs,
+             int recursion_depth, AutoNumber *auto_number)
+{
+    OutputString output;
+    PyObject *result = NULL;
+
+    output.data = NULL; /* needed so cleanup code always works */
+
+    /* check the recursion level */
+    if (recursion_depth <= 0) {
+        PyErr_SetString(PyExc_ValueError,
+                        "Max string recursion exceeded");
+        goto done;
+    }
+
+    /* initial size is the length of the format string, plus the size
+       increment.  seems like a reasonable default */
+    if (!output_initialize(&output,
+                           input->end - input->start +
+                           INITIAL_SIZE_INCREMENT))
+        goto done;
+
+    if (!do_markup(input, args, kwargs, &output, recursion_depth,
+                   auto_number)) {
+        goto done;
+    }
+
+    result = PyUnicode_New(output.pos, output.maxchar);
+    if (!result)
+        goto done;
+    memcpy(PyUnicode_DATA(result), output.data, output.pos << (output.kind-1));
+
+done:
+    if (output.data)
+        PyMem_Free(output.data);
+    return result;
+}
+
+/************************************************************************/
+/*********** main routine ***********************************************/
+/************************************************************************/
+
+/* this is the main entry point */
+static PyObject *
+do_string_format(PyObject *self, PyObject *args, PyObject *kwargs)
+{
+    SubString input;
+
+    /* PEP 3101 says only 2 levels, so that
+       "{0:{1}}".format('abc', 's')            # works
+       "{0:{1:{2}}}".format('abc', 's', '')    # fails
+    */
+    int recursion_depth = 2;
+
+    AutoNumber auto_number;
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+
+    AutoNumber_Init(&auto_number);
+    SubString_init(&input, self, 0, PyUnicode_GET_LENGTH(self));
+    return build_string(&input, args, kwargs, recursion_depth, &auto_number);
+}
+
+static PyObject *
+do_string_format_map(PyObject *self, PyObject *obj)
+{
+    return do_string_format(self, NULL, obj);
+}
+
+
+/************************************************************************/
+/*********** formatteriterator ******************************************/
+/************************************************************************/
+
+/* This is used to implement string.Formatter.vparse().  It exists so
+   Formatter can share code with the built in unicode.format() method.
+   It's really just a wrapper around MarkupIterator that is callable
+   from Python. */
+
+typedef struct {
+    PyObject_HEAD
+
+    PyUnicodeObject *str;
+
+    MarkupIterator it_markup;
+} formatteriterobject;
+
+static void
+formatteriter_dealloc(formatteriterobject *it)
+{
+    Py_XDECREF(it->str);
+    PyObject_FREE(it);
+}
+
+/* returns a tuple:
+   (literal, field_name, format_spec, conversion)
+
+   literal is any literal text to output.  might be zero length
+   field_name is the string before the ':'.  might be None
+   format_spec is the string after the ':'.  mibht be None
+   conversion is either None, or the string after the '!'
+*/
+static PyObject *
+formatteriter_next(formatteriterobject *it)
+{
+    SubString literal;
+    SubString field_name;
+    SubString format_spec;
+    Py_UCS4 conversion;
+    int format_spec_needs_expanding;
+    int field_present;
+    int result = MarkupIterator_next(&it->it_markup, &literal, &field_present,
+                                     &field_name, &format_spec, &conversion,
+                                     &format_spec_needs_expanding);
+
+    /* all of the SubString objects point into it->str, so no
+       memory management needs to be done on them */
+    assert(0 <= result && result <= 2);
+    if (result == 0 || result == 1)
+        /* if 0, error has already been set, if 1, iterator is empty */
+        return NULL;
+    else {
+        PyObject *literal_str = NULL;
+        PyObject *field_name_str = NULL;
+        PyObject *format_spec_str = NULL;
+        PyObject *conversion_str = NULL;
+        PyObject *tuple = NULL;
+
+        literal_str = SubString_new_object(&literal);
+        if (literal_str == NULL)
+            goto done;
+
+        field_name_str = SubString_new_object(&field_name);
+        if (field_name_str == NULL)
+            goto done;
+
+        /* if field_name is non-zero length, return a string for
+           format_spec (even if zero length), else return None */
+        format_spec_str = (field_present ?
+                           SubString_new_object_or_empty :
+                           SubString_new_object)(&format_spec);
+        if (format_spec_str == NULL)
+            goto done;
+
+        /* if the conversion is not specified, return a None,
+           otherwise create a one length string with the conversion
+           character */
+        if (conversion == '\0') {
+            conversion_str = Py_None;
+            Py_INCREF(conversion_str);
+        }
+        else
+            conversion_str = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                                       &conversion, 1);
+        if (conversion_str == NULL)
+            goto done;
+
+        tuple = PyTuple_Pack(4, literal_str, field_name_str, format_spec_str,
+                             conversion_str);
+    done:
+        Py_XDECREF(literal_str);
+        Py_XDECREF(field_name_str);
+        Py_XDECREF(format_spec_str);
+        Py_XDECREF(conversion_str);
+        return tuple;
+    }
+}
+
+static PyMethodDef formatteriter_methods[] = {
+    {NULL,              NULL}           /* sentinel */
+};
+
+static PyTypeObject PyFormatterIter_Type = {
+    PyVarObject_HEAD_INIT(&PyType_Type, 0)
+    "formatteriterator",                /* tp_name */
+    sizeof(formatteriterobject),        /* tp_basicsize */
+    0,                                  /* tp_itemsize */
+    /* methods */
+    (destructor)formatteriter_dealloc,  /* tp_dealloc */
+    0,                                  /* tp_print */
+    0,                                  /* tp_getattr */
+    0,                                  /* tp_setattr */
+    0,                                  /* tp_reserved */
+    0,                                  /* tp_repr */
+    0,                                  /* tp_as_number */
+    0,                                  /* tp_as_sequence */
+    0,                                  /* tp_as_mapping */
+    0,                                  /* tp_hash */
+    0,                                  /* tp_call */
+    0,                                  /* tp_str */
+    PyObject_GenericGetAttr,            /* tp_getattro */
+    0,                                  /* tp_setattro */
+    0,                                  /* tp_as_buffer */
+    Py_TPFLAGS_DEFAULT,                 /* tp_flags */
+    0,                                  /* tp_doc */
+    0,                                  /* tp_traverse */
+    0,                                  /* tp_clear */
+    0,                                  /* tp_richcompare */
+    0,                                  /* tp_weaklistoffset */
+    PyObject_SelfIter,                  /* tp_iter */
+    (iternextfunc)formatteriter_next,   /* tp_iternext */
+    formatteriter_methods,              /* tp_methods */
+    0,
+};
+
+/* unicode_formatter_parser is used to implement
+   string.Formatter.vformat.  it parses a string and returns tuples
+   describing the parsed elements.  It's a wrapper around
+   stringlib/string_format.h's MarkupIterator */
+static PyObject *
+formatter_parser(PyObject *ignored, PyUnicodeObject *self)
+{
+    formatteriterobject *it;
+
+    if (!PyUnicode_Check(self)) {
+        PyErr_Format(PyExc_TypeError, "expected str, got %s", Py_TYPE(self)->tp_name);
+        return NULL;
+    }
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+
+    it = PyObject_New(formatteriterobject, &PyFormatterIter_Type);
+    if (it == NULL)
+        return NULL;
+
+    /* take ownership, give the object to the iterator */
+    Py_INCREF(self);
+    it->str = self;
+
+    /* initialize the contained MarkupIterator */
+    MarkupIterator_init(&it->it_markup, (PyObject*)self, 0, PyUnicode_GET_LENGTH(self));
+    return (PyObject *)it;
+}
+
+
+/************************************************************************/
+/*********** fieldnameiterator ******************************************/
+/************************************************************************/
+
+
+/* This is used to implement string.Formatter.vparse().  It parses the
+   field name into attribute and item values.  It's a Python-callable
+   wrapper around FieldNameIterator */
+
+typedef struct {
+    PyObject_HEAD
+
+    PyUnicodeObject *str;
+
+    FieldNameIterator it_field;
+} fieldnameiterobject;
+
+static void
+fieldnameiter_dealloc(fieldnameiterobject *it)
+{
+    Py_XDECREF(it->str);
+    PyObject_FREE(it);
+}
+
+/* returns a tuple:
+   (is_attr, value)
+   is_attr is true if we used attribute syntax (e.g., '.foo')
+              false if we used index syntax (e.g., '[foo]')
+   value is an integer or string
+*/
+static PyObject *
+fieldnameiter_next(fieldnameiterobject *it)
+{
+    int result;
+    int is_attr;
+    Py_ssize_t idx;
+    SubString name;
+
+    result = FieldNameIterator_next(&it->it_field, &is_attr,
+                                    &idx, &name);
+    if (result == 0 || result == 1)
+        /* if 0, error has already been set, if 1, iterator is empty */
+        return NULL;
+    else {
+        PyObject* result = NULL;
+        PyObject* is_attr_obj = NULL;
+        PyObject* obj = NULL;
+
+        is_attr_obj = PyBool_FromLong(is_attr);
+        if (is_attr_obj == NULL)
+            goto done;
+
+        /* either an integer or a string */
+        if (idx != -1)
+            obj = PyLong_FromSsize_t(idx);
+        else
+            obj = SubString_new_object(&name);
+        if (obj == NULL)
+            goto done;
+
+        /* return a tuple of values */
+        result = PyTuple_Pack(2, is_attr_obj, obj);
+
+    done:
+        Py_XDECREF(is_attr_obj);
+        Py_XDECREF(obj);
+        return result;
+    }
+}
+
+static PyMethodDef fieldnameiter_methods[] = {
+    {NULL,              NULL}           /* sentinel */
+};
+
+static PyTypeObject PyFieldNameIter_Type = {
+    PyVarObject_HEAD_INIT(&PyType_Type, 0)
+    "fieldnameiterator",                /* tp_name */
+    sizeof(fieldnameiterobject),        /* tp_basicsize */
+    0,                                  /* tp_itemsize */
+    /* methods */
+    (destructor)fieldnameiter_dealloc,  /* tp_dealloc */
+    0,                                  /* tp_print */
+    0,                                  /* tp_getattr */
+    0,                                  /* tp_setattr */
+    0,                                  /* tp_reserved */
+    0,                                  /* tp_repr */
+    0,                                  /* tp_as_number */
+    0,                                  /* tp_as_sequence */
+    0,                                  /* tp_as_mapping */
+    0,                                  /* tp_hash */
+    0,                                  /* tp_call */
+    0,                                  /* tp_str */
+    PyObject_GenericGetAttr,            /* tp_getattro */
+    0,                                  /* tp_setattro */
+    0,                                  /* tp_as_buffer */
+    Py_TPFLAGS_DEFAULT,                 /* tp_flags */
+    0,                                  /* tp_doc */
+    0,                                  /* tp_traverse */
+    0,                                  /* tp_clear */
+    0,                                  /* tp_richcompare */
+    0,                                  /* tp_weaklistoffset */
+    PyObject_SelfIter,                  /* tp_iter */
+    (iternextfunc)fieldnameiter_next,   /* tp_iternext */
+    fieldnameiter_methods,              /* tp_methods */
+    0};
+
+/* unicode_formatter_field_name_split is used to implement
+   string.Formatter.vformat.  it takes an PEP 3101 "field name", and
+   returns a tuple of (first, rest): "first", the part before the
+   first '.' or '['; and "rest", an iterator for the rest of the field
+   name.  it's a wrapper around stringlib/string_format.h's
+   field_name_split.  The iterator it returns is a
+   FieldNameIterator */
+static PyObject *
+formatter_field_name_split(PyObject *ignored, PyUnicodeObject *self)
+{
+    SubString first;
+    Py_ssize_t first_idx;
+    fieldnameiterobject *it;
+
+    PyObject *first_obj = NULL;
+    PyObject *result = NULL;
+
+    if (!PyUnicode_Check(self)) {
+        PyErr_Format(PyExc_TypeError, "expected str, got %s", Py_TYPE(self)->tp_name);
+        return NULL;
+    }
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+
+    it = PyObject_New(fieldnameiterobject, &PyFieldNameIter_Type);
+    if (it == NULL)
+        return NULL;
+
+    /* take ownership, give the object to the iterator.  this is
+       just to keep the field_name alive */
+    Py_INCREF(self);
+    it->str = self;
+
+    /* Pass in auto_number = NULL. We'll return an empty string for
+       first_obj in that case. */
+    if (!field_name_split((PyObject*)self, 0, PyUnicode_GET_LENGTH(self),
+                          &first, &first_idx, &it->it_field, NULL))
+        goto done;
+
+    /* first becomes an integer, if possible; else a string */
+    if (first_idx != -1)
+        first_obj = PyLong_FromSsize_t(first_idx);
+    else
+        /* convert "first" into a string object */
+        first_obj = SubString_new_object(&first);
+    if (first_obj == NULL)
+        goto done;
+
+    /* return a tuple of values */
+    result = PyTuple_Pack(2, first_obj, it);
+
+done:
+    Py_XDECREF(it);
+    Py_XDECREF(first_obj);
+    return result;
+}
diff --git a/Objects/stringlib/unicodedefs.h b/Objects/stringlib/unicodedefs.h
--- a/Objects/stringlib/unicodedefs.h
+++ b/Objects/stringlib/unicodedefs.h
@@ -6,6 +6,8 @@
    compiled as unicode. */
 #define STRINGLIB_IS_UNICODE     1
 
+#define FASTSEARCH               fastsearch
+#define STRINGLIB(F)             stringlib_##F
 #define STRINGLIB_OBJECT         PyUnicodeObject
 #define STRINGLIB_CHAR           Py_UNICODE
 #define STRINGLIB_TYPE_NAME      "unicode"
diff --git a/Objects/typeobject.c b/Objects/typeobject.c
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -20,10 +20,11 @@
          >> (8*sizeof(unsigned int) - MCACHE_SIZE_EXP))
 #define MCACHE_HASH_METHOD(type, name)                                  \
         MCACHE_HASH((type)->tp_version_tag,                     \
-                    ((PyUnicodeObject *)(name))->hash)
+                    ((PyASCIIObject *)(name))->hash)
 #define MCACHE_CACHEABLE_NAME(name)                                     \
         PyUnicode_CheckExact(name) &&                            \
-        PyUnicode_GET_SIZE(name) <= MCACHE_MAX_ATTR_SIZE
+        PyUnicode_READY(name) != -1 &&                      \
+        PyUnicode_GET_LENGTH(name) <= MCACHE_MAX_ATTR_SIZE
 
 struct method_cache_entry {
     unsigned int version;
@@ -3489,7 +3490,7 @@
     if (self_as_str != NULL) {
         /* Issue 7994: If we're converting to a string, we
            should reject format specifications */
-        if (PyUnicode_GET_SIZE(format_spec) > 0) {
+        if (PyUnicode_GET_LENGTH(format_spec) > 0) {
             if (PyErr_WarnEx(PyExc_DeprecationWarning,
                              "object.__format__ with a non-empty format "
                              "string is deprecated", 1) < 0) {
@@ -5122,14 +5123,21 @@
         return res;
     }
     else {
-        PyObject *ress;
+        /* PyObject *ress; */
         PyErr_Clear();
         res = slot_tp_repr(self);
         if (!res)
             return NULL;
+        /* XXX this is non-sensical. Why should we return
+           a bytes object from __str__. Is this code even
+           used? - mvl */
+        assert(0);
+        return res;
+        /*
         ress = _PyUnicode_AsDefaultEncodedString(res);
         Py_DECREF(res);
         return ress;
+        */
     }
 }
 
@@ -6206,7 +6214,7 @@
         /* We want __class__ to return the class of the super object
            (i.e. super, or a subclass), not the class of su->obj. */
         skip = (PyUnicode_Check(name) &&
-            PyUnicode_GET_SIZE(name) == 9 &&
+            PyUnicode_GET_LENGTH(name) == 9 &&
             PyUnicode_CompareWithASCIIString(name, "__class__") == 0);
     }
 
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -90,6 +90,19 @@
 extern "C" {
 #endif
 
+#define _PyUnicode_WSTR(op) (((PyASCIIObject*)(op))->wstr)
+#define _PyUnicode_WSTR_LENGTH(op) (((PyCompactUnicodeObject*)(op))->wstr_length)
+#define _PyUnicode_LENGTH(op) (((PyASCIIObject *)(op))->length)
+#define _PyUnicode_STATE(op) (((PyASCIIObject *)(op))->state)
+#define _PyUnicode_HASH(op) (((PyASCIIObject *)(op))->hash)
+#define _PyUnicode_KIND(op) \
+    (assert(PyUnicode_Check(op)), \
+     ((PyASCIIObject *)(op))->state.kind)
+#define _PyUnicode_GET_LENGTH(op)                \
+    (assert(PyUnicode_Check(op)),               \
+     ((PyASCIIObject *)(op))->length)
+
+
 /* This dictionary holds all interned unicode strings.  Note that references
    to strings in this dictionary are *not* counted in the string's ob_refcnt.
    When the interned string reaches a refcnt of 0 the string deallocation
@@ -100,10 +113,6 @@
 */
 static PyObject *interned;
 
-/* Free list for Unicode objects */
-static PyUnicodeObject *free_list;
-static int numfree;
-
 /* The empty Unicode object is shared to improve performance. */
 static PyUnicodeObject *unicode_empty;
 
@@ -226,7 +235,7 @@
      (BLOOM(bloom_linebreak, (ch)) && Py_UNICODE_ISLINEBREAK(ch)))
 
 Py_LOCAL_INLINE(BLOOM_MASK)
-make_bloom_mask(Py_UNICODE* ptr, Py_ssize_t len)
+make_bloom_mask(int kind, void* ptr, Py_ssize_t len)
 {
     /* calculate simple bloom-style bitmask for a given unicode string */
 
@@ -235,36 +244,58 @@
 
     mask = 0;
     for (i = 0; i < len; i++)
-        BLOOM_ADD(mask, ptr[i]);
+        BLOOM_ADD(mask, PyUnicode_READ(kind, ptr, i));
 
     return mask;
 }
 
-Py_LOCAL_INLINE(int)
-unicode_member(Py_UNICODE chr, Py_UNICODE* set, Py_ssize_t setlen)
-{
+#define BLOOM_MEMBER(mask, chr, str) \
+    (BLOOM(mask, chr) \
+     && (PyUnicode_FindChar(str, chr, 0, PyUnicode_GET_LENGTH(str), 1) >= 0))
+
+/* --- Unicode Object ----------------------------------------------------- */
+
+static PyObject *
+substring(PyUnicodeObject *self, Py_ssize_t start, Py_ssize_t len);
+
+static PyObject *
+fixup(PyUnicodeObject *self, Py_UCS4 (*fixfct)(PyUnicodeObject *s));
+
+Py_LOCAL_INLINE(char *) findchar(void *s, int kind,
+                                 Py_ssize_t size, Py_UCS4 ch,
+                                 int direction)
+{
+    /* like wcschr, but doesn't stop at NULL characters */
     Py_ssize_t i;
-
-    for (i = 0; i < setlen; i++)
-        if (set[i] == chr)
-            return 1;
-
-    return 0;
-}
-
-#define BLOOM_MEMBER(mask, chr, set, setlen)                    \
-    BLOOM(mask, chr) && unicode_member(chr, set, setlen)
-
-/* --- Unicode Object ----------------------------------------------------- */
+    if (direction == 1) {
+        for(i = 0; i < size; i++)
+            if (PyUnicode_READ(kind, s, i) == ch)
+                return (char*)s + PyUnicode_KIND_SIZE(kind, i);
+    }
+    else {
+        for(i = size-1; i >= 0; i--)
+            if (PyUnicode_READ(kind, s, i) == ch)
+                return (char*)s + PyUnicode_KIND_SIZE(kind, i);
+    }
+    return NULL;
+}
 
 static int
 unicode_resize(register PyUnicodeObject *unicode,
-	       Py_ssize_t length)
+               Py_ssize_t length)
 {
     void *oldstr;
 
+    /* Resizing is only supported for old unicode objects. */
+    assert(!PyUnicode_IS_COMPACT(unicode));
+    assert(_PyUnicode_WSTR(unicode) != NULL);
+
+    /* ... and only if they have not been readied yet, because
+       callees usually rely on the wstr representation when resizing. */
+    assert(unicode->data.any == NULL);
+
     /* Shortcut if there's nothing much to do. */
-    if (unicode->length == length)
+    if (_PyUnicode_WSTR_LENGTH(unicode) == length)
         goto reset;
 
     /* Resizing shared object (unicode_empty or single character
@@ -272,9 +303,9 @@
        instead ! */
 
     if (unicode == unicode_empty ||
-        (unicode->length == 1 &&
-         unicode->str[0] < 256U &&
-         unicode_latin1[unicode->str[0]] == unicode)) {
+        (_PyUnicode_WSTR_LENGTH(unicode) == 1 &&
+         _PyUnicode_WSTR(unicode)[0] < 256U &&
+         unicode_latin1[_PyUnicode_WSTR(unicode)[0]] == unicode)) {
         PyErr_SetString(PyExc_SystemError,
                         "can't resize shared str objects");
         return -1;
@@ -285,23 +316,31 @@
        safe to look at str[length] (without making any assumptions about what
        it contains). */
 
-    oldstr = unicode->str;
-    unicode->str = PyObject_REALLOC(unicode->str,
-                                    sizeof(Py_UNICODE) * (length + 1));
-    if (!unicode->str) {
-        unicode->str = (Py_UNICODE *)oldstr;
+    oldstr = _PyUnicode_WSTR(unicode);
+    _PyUnicode_WSTR(unicode) = PyObject_REALLOC(_PyUnicode_WSTR(unicode),
+                                     sizeof(Py_UNICODE) * (length + 1));
+    if (!_PyUnicode_WSTR(unicode)) {
+        _PyUnicode_WSTR(unicode) = (Py_UNICODE *)oldstr;
         PyErr_NoMemory();
         return -1;
     }
-    unicode->str[length] = 0;
-    unicode->length = length;
+    _PyUnicode_WSTR(unicode)[length] = 0;
+    _PyUnicode_WSTR_LENGTH(unicode) = length;
 
   reset:
-    /* Reset the object caches */
-    if (unicode->defenc) {
-        Py_CLEAR(unicode->defenc);
-    }
-    unicode->hash = -1;
+    if (unicode->data.any != NULL) {
+        PyObject_FREE(unicode->data.any);
+        if (unicode->_base.utf8 && unicode->_base.utf8 != unicode->data.any) {
+            PyObject_FREE(unicode->_base.utf8);
+        }
+        unicode->_base.utf8 = NULL;
+        unicode->_base.utf8_length = 0;
+        unicode->data.any = NULL;
+        _PyUnicode_LENGTH(unicode) = 0;
+        _PyUnicode_STATE(unicode).interned = _PyUnicode_STATE(unicode).interned;
+        _PyUnicode_STATE(unicode).kind = PyUnicode_WCHAR_KIND;
+    }
+    _PyUnicode_HASH(unicode) = -1;
 
     return 0;
 }
@@ -315,10 +354,15 @@
 
 */
 
+#ifdef Py_DEBUG
+int unicode_old_new_calls = 0;
+#endif
+
 static PyUnicodeObject *
 _PyUnicode_New(Py_ssize_t length)
 {
     register PyUnicodeObject *unicode;
+    size_t new_size;
 
     /* Optimization for empty strings */
     if (length == 0 && unicode_empty != NULL) {
@@ -330,40 +374,26 @@
     if (length > ((PY_SSIZE_T_MAX / sizeof(Py_UNICODE)) - 1)) {
         return (PyUnicodeObject *)PyErr_NoMemory();
     }
-
-    /* Unicode freelist & memory allocation */
-    if (free_list) {
-        unicode = free_list;
-        free_list = *(PyUnicodeObject **)unicode;
-        numfree--;
-        if (unicode->str) {
-            /* Keep-Alive optimization: we only upsize the buffer,
-               never downsize it. */
-            if ((unicode->length < length) &&
-                unicode_resize(unicode, length) < 0) {
-                PyObject_DEL(unicode->str);
-                unicode->str = NULL;
-            }
-        }
-        else {
-            size_t new_size = sizeof(Py_UNICODE) * ((size_t)length + 1);
-            unicode->str = (Py_UNICODE*) PyObject_MALLOC(new_size);
-        }
-        PyObject_INIT(unicode, &PyUnicode_Type);
-    }
-    else {
-        size_t new_size;
-        unicode = PyObject_New(PyUnicodeObject, &PyUnicode_Type);
-        if (unicode == NULL)
-            return NULL;
-        new_size = sizeof(Py_UNICODE) * ((size_t)length + 1);
-        unicode->str = (Py_UNICODE*) PyObject_MALLOC(new_size);
-    }
-
-    if (!unicode->str) {
+    if (length < 0) {
+        PyErr_SetString(PyExc_SystemError,
+                        "Negative size passed to _PyUnicode_New");
+        return NULL;
+    }
+
+#ifdef Py_DEBUG
+    ++unicode_old_new_calls;
+#endif
+
+    unicode = PyObject_New(PyUnicodeObject, &PyUnicode_Type);
+    if (unicode == NULL)
+        return NULL;
+    new_size = sizeof(Py_UNICODE) * ((size_t)length + 1);
+    _PyUnicode_WSTR(unicode) = (Py_UNICODE*) PyObject_MALLOC(new_size);
+    if (!_PyUnicode_WSTR(unicode)) {
         PyErr_NoMemory();
         goto onError;
     }
+
     /* Initialize the first element to guard against cases where
      * the caller fails before initializing str -- unicode_resize()
      * reads str[0], and the Keep-Alive optimization can keep memory
@@ -371,12 +401,19 @@
      * We don't want unicode_resize to read uninitialized memory in
      * that case.
      */
-    unicode->str[0] = 0;
-    unicode->str[length] = 0;
-    unicode->length = length;
-    unicode->hash = -1;
-    unicode->state = 0;
-    unicode->defenc = NULL;
+    _PyUnicode_WSTR(unicode)[0] = 0;
+    _PyUnicode_WSTR(unicode)[length] = 0;
+    _PyUnicode_WSTR_LENGTH(unicode) = length;
+    _PyUnicode_HASH(unicode) = -1;
+    _PyUnicode_STATE(unicode).interned = 0;
+    _PyUnicode_STATE(unicode).kind = 0;
+    _PyUnicode_STATE(unicode).compact = 0;
+    _PyUnicode_STATE(unicode).ready = 0;
+    _PyUnicode_STATE(unicode).ascii = 0;
+    unicode->data.any = NULL;
+    _PyUnicode_LENGTH(unicode) = 0;
+    unicode->_base.utf8 = NULL;
+    unicode->_base.utf8_length = 0;
     return unicode;
 
   onError:
@@ -387,6 +424,467 @@
     return NULL;
 }
 
+#ifdef Py_DEBUG
+int unicode_new_new_calls = 0;
+
+/* Functions wrapping macros for use in debugger */
+char *_PyUnicode_utf8(void *unicode){
+    return _PyUnicode_UTF8(unicode);
+}
+
+void *_PyUnicode_compact_data(void *unicode) {
+    return _PyUnicode_COMPACT_DATA(unicode);
+}
+void *_PyUnicode_data(void *unicode){
+    printf("obj %p\n", unicode);
+    printf("compact %d\n", PyUnicode_IS_COMPACT(unicode));
+    printf("compact ascii %d\n", PyUnicode_IS_COMPACT_ASCII(unicode));
+    printf("ascii op %p\n", ((void*)((PyASCIIObject*)(unicode) + 1)));
+    printf("compact op %p\n", ((void*)((PyCompactUnicodeObject*)(unicode) + 1)));
+    printf("compact data %p\n", _PyUnicode_COMPACT_DATA(unicode));
+    return PyUnicode_DATA(unicode);
+}
+#endif
+
+PyObject *
+PyUnicode_New(Py_ssize_t size, Py_UCS4 maxchar)
+{
+    PyObject *obj;
+    PyCompactUnicodeObject *unicode;
+    void *data;
+    int kind_state;
+    int is_sharing = 0, is_ascii = 0;
+    Py_ssize_t char_size;
+    Py_ssize_t struct_size;
+
+    /* Optimization for empty strings */
+    if (size == 0 && unicode_empty != NULL) {
+        Py_INCREF(unicode_empty);
+        return (PyObject *)unicode_empty;
+    }
+
+#ifdef Py_DEBUG
+    ++unicode_new_new_calls;
+#endif
+
+    struct_size = sizeof(PyCompactUnicodeObject);
+    if (maxchar < 128) {
+        kind_state = PyUnicode_1BYTE_KIND;
+        char_size = 1;
+        is_ascii = 1;
+        struct_size = sizeof(PyASCIIObject);
+    }
+    else if (maxchar < 256) {
+        kind_state = PyUnicode_1BYTE_KIND;
+        char_size = 1;
+    }
+    else if (maxchar < 65536) {
+        kind_state = PyUnicode_2BYTE_KIND;
+        char_size = 2;
+        if (sizeof(wchar_t) == 2)
+            is_sharing = 1;
+    }
+    else {
+        kind_state = PyUnicode_4BYTE_KIND;
+        char_size = 4;
+        if (sizeof(wchar_t) == 4)
+            is_sharing = 1;
+    }
+
+    /* Ensure we won't overflow the size. */
+    if (size < 0) {
+        PyErr_SetString(PyExc_SystemError,
+                        "Negative size passed to PyUnicode_New");
+        return NULL;
+    }
+    if (size > ((PY_SSIZE_T_MAX - struct_size) / char_size - 1))
+        return PyErr_NoMemory();
+
+    /* Duplicated allocation code from _PyObject_New() instead of a call to
+     * PyObject_New() so we are able to allocate space for the object and
+     * it's data buffer.
+     */
+    obj = (PyObject *) PyObject_MALLOC(struct_size + (size + 1) * char_size);
+    if (obj == NULL)
+        return PyErr_NoMemory();
+    obj = PyObject_INIT(obj, &PyUnicode_Type);
+    if (obj == NULL)
+        return NULL;
+
+    unicode = (PyCompactUnicodeObject *)obj;
+    if (is_ascii)
+        data = ((PyASCIIObject*)obj) + 1;
+    else
+        data = unicode + 1;
+    _PyUnicode_LENGTH(unicode) = size;
+    _PyUnicode_HASH(unicode) = -1;
+    _PyUnicode_STATE(unicode).interned = 0;
+    _PyUnicode_STATE(unicode).kind = kind_state;
+    _PyUnicode_STATE(unicode).compact = 1;
+    _PyUnicode_STATE(unicode).ready = 1;
+    _PyUnicode_STATE(unicode).ascii = is_ascii;
+    if (is_ascii) {
+        ((char*)data)[size] = 0;
+        _PyUnicode_WSTR(unicode) = NULL;
+    }
+    else if (kind_state == PyUnicode_1BYTE_KIND) {
+        ((char*)data)[size] = 0;
+        _PyUnicode_WSTR(unicode) = NULL;
+        _PyUnicode_WSTR_LENGTH(unicode) = 0;
+        unicode->utf8_length = 0;
+        unicode->utf8 = NULL;
+        }
+    else {
+        unicode->utf8 = NULL;
+        if (kind_state == PyUnicode_2BYTE_KIND)
+            ((Py_UCS2*)data)[size] = 0;
+        else /* kind_state == PyUnicode_4BYTE_KIND */
+            ((Py_UCS4*)data)[size] = 0;
+        if (is_sharing) {
+            _PyUnicode_WSTR_LENGTH(unicode) = size;
+            _PyUnicode_WSTR(unicode) = (wchar_t *)data;
+        }
+        else {
+            _PyUnicode_WSTR_LENGTH(unicode) = 0;
+            _PyUnicode_WSTR(unicode) = NULL;
+        }
+    }
+    return obj;
+}
+
+#if SIZEOF_WCHAR_T == 2
+/* Helper function to convert a 16-bits wchar_t representation to UCS4, this
+   will decode surrogate pairs, the other conversions are implemented as macros
+   for efficency.
+
+   This function assumes that unicode can hold one more code point than wstr
+   characters for a terminating null character. */
+static int
+unicode_convert_wchar_to_ucs4(const wchar_t *begin, const wchar_t *end,
+                              PyUnicodeObject *unicode)
+{
+    const wchar_t *iter;
+    Py_UCS4 *ucs4_out;
+
+    assert(unicode && PyUnicode_Check(unicode));
+    assert(_PyUnicode_KIND(unicode) == PyUnicode_4BYTE_KIND);
+    ucs4_out = PyUnicode_4BYTE_DATA(unicode);
+
+    for (iter = begin; iter < end; ) {
+        assert(ucs4_out < (PyUnicode_4BYTE_DATA(unicode) +
+                           _PyUnicode_GET_LENGTH(unicode)));
+        if (*iter >= 0xD800 && *iter <= 0xDBFF
+            && (iter+1) < end && iter[1] >= 0xDC00 && iter[1] <= 0xDFFF)
+        {
+            *ucs4_out++ = (((iter[0] & 0x3FF)<<10) | (iter[1] & 0x3FF)) + 0x10000;
+            iter += 2;
+        }
+        else {
+            *ucs4_out++ = *iter;
+            iter++;
+        }
+    }
+    assert(ucs4_out == (PyUnicode_4BYTE_DATA(unicode) +
+                        _PyUnicode_GET_LENGTH(unicode)));
+
+    return 0;
+}
+#endif
+
+int
+PyUnicode_CopyCharacters(PyObject *to, Py_ssize_t to_start,
+                         PyObject *from, Py_ssize_t from_start,
+                         Py_ssize_t how_many)
+{
+    int from_kind;
+    int to_kind;
+
+    assert(PyUnicode_Check(from));
+    assert(PyUnicode_Check(to));
+
+    if (PyUnicode_READY(from))
+        return -1;
+    if (PyUnicode_READY(to))
+        return -1;
+
+    from_kind = PyUnicode_KIND(from);
+    to_kind = PyUnicode_KIND(to);
+
+    if (from_kind == to_kind) {
+        const Py_ssize_t char_size = PyUnicode_CHARACTER_SIZE(to);
+        Py_MEMCPY(PyUnicode_1BYTE_DATA(to) + (to_start * char_size),
+                  PyUnicode_1BYTE_DATA(from) + (from_start * char_size),
+                  how_many * char_size);
+        return 0;
+    }
+
+    switch (from_kind) {
+        case PyUnicode_1BYTE_KIND:
+            switch (to_kind) {
+                case PyUnicode_2BYTE_KIND:
+                    PyUnicode_CONVERT_BYTES(
+                        unsigned char, Py_UCS2,
+                        PyUnicode_1BYTE_DATA(from) + from_start,
+                        PyUnicode_1BYTE_DATA(from) + from_start + how_many,
+                        PyUnicode_2BYTE_DATA(to) + to_start
+                        );
+                    break;
+                case PyUnicode_4BYTE_KIND:
+                    PyUnicode_CONVERT_BYTES(
+                        unsigned char, Py_UCS4,
+                        PyUnicode_1BYTE_DATA(from) + from_start,
+                        PyUnicode_1BYTE_DATA(from) + from_start + how_many,
+                        PyUnicode_4BYTE_DATA(to) + to_start
+                        );
+                    break;
+                default:
+                    goto invalid_state;
+            }
+            break;
+        case PyUnicode_2BYTE_KIND:
+            switch (to_kind) {
+                case PyUnicode_1BYTE_KIND:
+                    PyUnicode_CONVERT_BYTES(
+                        Py_UCS2, unsigned char,
+                        PyUnicode_2BYTE_DATA(from) + from_start,
+                        PyUnicode_2BYTE_DATA(from) + from_start + how_many,
+                        PyUnicode_1BYTE_DATA(to) + to_start
+                        );
+                    break;
+                case PyUnicode_4BYTE_KIND:
+                    PyUnicode_CONVERT_BYTES(
+                        Py_UCS2, Py_UCS4,
+                        PyUnicode_2BYTE_DATA(from) + from_start,
+                        PyUnicode_2BYTE_DATA(from) + from_start + how_many,
+                        PyUnicode_4BYTE_DATA(to) + to_start
+                        );
+                    break;
+                default:
+                    goto invalid_state;
+            }
+            break;
+        case PyUnicode_4BYTE_KIND:
+            switch (to_kind) {
+                case PyUnicode_1BYTE_KIND:
+                    PyUnicode_CONVERT_BYTES(
+                        Py_UCS4, unsigned char,
+                        PyUnicode_4BYTE_DATA(from) + from_start,
+                        PyUnicode_4BYTE_DATA(from) + from_start + how_many,
+                        PyUnicode_1BYTE_DATA(to) + to_start
+                        );
+                    break;
+                case PyUnicode_2BYTE_KIND:
+                    PyUnicode_CONVERT_BYTES(
+                        Py_UCS4, Py_UCS2,
+                        PyUnicode_4BYTE_DATA(from) + from_start,
+                        PyUnicode_4BYTE_DATA(from) + from_start + how_many,
+                        PyUnicode_2BYTE_DATA(to) + to_start
+                        );
+                    break;
+                default:
+                    goto invalid_state;
+            }
+            break;
+        default:
+            goto invalid_state;
+    }
+    return 0;
+
+invalid_state:
+    PyErr_Format(PyExc_ValueError,
+                 "Impossible kind state (from=%i, to=%i) "
+                 "in PyUnicode_CopyCharacters",
+                 from_kind, to_kind);
+    return -1;
+}
+
+int
+_PyUnicode_FindMaxCharAndNumSurrogatePairs(const wchar_t *begin,
+                                           const wchar_t *end,
+                                           Py_UCS4 *maxchar,
+                                           Py_ssize_t *num_surrogates)
+{
+    const wchar_t *iter;
+
+    if (num_surrogates == NULL || maxchar == NULL) {
+        PyErr_SetString(PyExc_SystemError,
+                        "unexpected NULL arguments to "
+                        "PyUnicode_FindMaxCharAndNumSurrogatePairs");
+        return -1;
+    }
+
+    *num_surrogates = 0;
+    *maxchar = 0;
+
+    for (iter = begin; iter < end; ) {
+        if (*iter > *maxchar)
+            *maxchar = *iter;
+#if SIZEOF_WCHAR_T == 2
+        if (*iter >= 0xD800 && *iter <= 0xDBFF
+            && (iter+1) < end && iter[1] >= 0xDC00 && iter[1] <= 0xDFFF)
+        {
+            Py_UCS4 surrogate_val;
+            surrogate_val = (((iter[0] & 0x3FF)<<10)
+                             | (iter[1] & 0x3FF)) + 0x10000;
+            ++(*num_surrogates);
+            if (surrogate_val > *maxchar)
+                *maxchar = surrogate_val;
+            iter += 2;
+        }
+        else
+            iter++;
+#else
+        iter++;
+#endif
+    }
+    return 0;
+}
+
+#ifdef Py_DEBUG
+int unicode_ready_calls = 0;
+#endif
+
+int
+_PyUnicode_Ready(PyUnicodeObject *unicode)
+{
+    wchar_t *end;
+    Py_UCS4 maxchar = 0;
+    Py_ssize_t num_surrogates;
+#if SIZEOF_WCHAR_T == 2
+    Py_ssize_t length_wo_surrogates;
+#endif
+
+    assert(PyUnicode_Check(unicode));
+
+    if (unicode->data.any != NULL) {
+        assert(PyUnicode_KIND(unicode) != PyUnicode_WCHAR_KIND);
+        return 0;
+    }
+
+    /* _PyUnicode_Ready() is only intented for old-style API usage where
+     * strings were created using _PyObject_New() and where no canonical
+     * representation (the str field) has been set yet aka strings
+     * which are not yet ready.
+     */
+    assert(_PyUnicode_WSTR(unicode) != NULL);
+    assert(_PyUnicode_KIND(unicode) == PyUnicode_WCHAR_KIND);
+    assert(!PyUnicode_IS_COMPACT(unicode));
+    assert(!PyUnicode_IS_READY(unicode));
+    /* Actually, it should neither be interned nor be anything else: */
+    assert(_PyUnicode_STATE(unicode).interned == 0);
+    assert(unicode->_base.utf8 == NULL);
+
+#ifdef Py_DEBUG
+    ++unicode_ready_calls;
+#endif
+
+    end = _PyUnicode_WSTR(unicode) + _PyUnicode_WSTR_LENGTH(unicode);
+    if (_PyUnicode_FindMaxCharAndNumSurrogatePairs(_PyUnicode_WSTR(unicode), end,
+                                                  &maxchar,
+                                                  &num_surrogates) == -1) {
+        assert(0 && "PyUnicode_FindMaxCharAndNumSurrogatePairs failed");
+        return -1;
+    }
+
+    if (maxchar < 256) {
+        unicode->data.any = PyObject_MALLOC(_PyUnicode_WSTR_LENGTH(unicode) + 1);
+        if (!unicode->data.any) {
+            PyErr_NoMemory();
+            return -1;
+        }
+        PyUnicode_CONVERT_BYTES(wchar_t, unsigned char,
+                                _PyUnicode_WSTR(unicode), end,
+                                PyUnicode_1BYTE_DATA(unicode));
+        PyUnicode_1BYTE_DATA(unicode)[_PyUnicode_WSTR_LENGTH(unicode)] = '\0';
+        _PyUnicode_LENGTH(unicode) = _PyUnicode_WSTR_LENGTH(unicode);
+        _PyUnicode_STATE(unicode).kind = PyUnicode_1BYTE_KIND;
+        if (maxchar < 128) {
+            unicode->_base.utf8 = unicode->data.any;
+            unicode->_base.utf8_length = _PyUnicode_WSTR_LENGTH(unicode);
+        }
+        else {
+            unicode->_base.utf8 = NULL;
+            unicode->_base.utf8_length = 0;
+        }
+        PyObject_FREE(_PyUnicode_WSTR(unicode));
+        _PyUnicode_WSTR(unicode) = NULL;
+        _PyUnicode_WSTR_LENGTH(unicode) = 0;
+    }
+    /* In this case we might have to convert down from 4-byte native
+       wchar_t to 2-byte unicode. */
+    else if (maxchar < 65536) {
+        assert(num_surrogates == 0 &&
+               "FindMaxCharAndNumSurrogatePairs() messed up");
+
+        if (sizeof(wchar_t) == 2) {
+            /* We can share representations and are done. */
+            unicode->data.any = _PyUnicode_WSTR(unicode);
+            PyUnicode_2BYTE_DATA(unicode)[_PyUnicode_WSTR_LENGTH(unicode)] = '\0';
+            _PyUnicode_LENGTH(unicode) = _PyUnicode_WSTR_LENGTH(unicode);
+            _PyUnicode_STATE(unicode).kind = PyUnicode_2BYTE_KIND;
+            unicode->_base.utf8 = NULL;
+            unicode->_base.utf8_length = 0;
+        }
+        else {
+            assert(sizeof(wchar_t) == 4);
+
+            unicode->data.any = PyObject_MALLOC(
+                2 * (_PyUnicode_WSTR_LENGTH(unicode) + 1));
+            if (!unicode->data.any) {
+                PyErr_NoMemory();
+                return -1;
+            }
+            PyUnicode_CONVERT_BYTES(wchar_t, Py_UCS2,
+                                    _PyUnicode_WSTR(unicode), end,
+                                    PyUnicode_2BYTE_DATA(unicode));
+            PyUnicode_2BYTE_DATA(unicode)[_PyUnicode_WSTR_LENGTH(unicode)] = '\0';
+            _PyUnicode_LENGTH(unicode) = _PyUnicode_WSTR_LENGTH(unicode);
+            _PyUnicode_STATE(unicode).kind = PyUnicode_2BYTE_KIND;
+            unicode->_base.utf8 = NULL;
+            unicode->_base.utf8_length = 0;
+            PyObject_FREE(_PyUnicode_WSTR(unicode));
+            _PyUnicode_WSTR(unicode) = NULL;
+            _PyUnicode_WSTR_LENGTH(unicode) = 0;
+        }
+    }
+    /* maxchar exeeds 16 bit, wee need 4 bytes for unicode characters */
+    else {
+#if SIZEOF_WCHAR_T == 2
+        /* in case the native representation is 2-bytes, we need to allocate a
+           new normalized 4-byte version. */
+        length_wo_surrogates = _PyUnicode_WSTR_LENGTH(unicode) - num_surrogates;
+        unicode->data.any = PyObject_MALLOC(4 * (length_wo_surrogates + 1));
+        if (!unicode->data.any) {
+            PyErr_NoMemory();
+            return -1;
+        }
+        _PyUnicode_LENGTH(unicode) = length_wo_surrogates;
+        _PyUnicode_STATE(unicode).kind = PyUnicode_4BYTE_KIND;
+        unicode->_base.utf8 = NULL;
+        unicode->_base.utf8_length = 0;
+        if (unicode_convert_wchar_to_ucs4(_PyUnicode_WSTR(unicode), end,
+                                          unicode) < 0) {
+            assert(0 && "ConvertWideCharToUCS4 failed");
+            return -1;
+        }
+        PyObject_FREE(_PyUnicode_WSTR(unicode));
+        _PyUnicode_WSTR(unicode) = NULL;
+        _PyUnicode_WSTR_LENGTH(unicode) = 0;
+#else
+        assert(num_surrogates == 0);
+
+        unicode->data.any = _PyUnicode_WSTR(unicode);
+        _PyUnicode_LENGTH(unicode) = _PyUnicode_WSTR_LENGTH(unicode);
+        unicode->_base.utf8 = NULL;
+        unicode->_base.utf8_length = 0;
+        _PyUnicode_STATE(unicode).kind = PyUnicode_4BYTE_KIND;
+#endif
+        PyUnicode_4BYTE_DATA(unicode)[_PyUnicode_LENGTH(unicode)] = '\0';
+    }
+    _PyUnicode_STATE(unicode).ready = 1;
+    return 0;
+}
+
 static void
 unicode_dealloc(register PyUnicodeObject *unicode)
 {
@@ -409,25 +907,19 @@
         Py_FatalError("Inconsistent interned string state.");
     }
 
-    if (PyUnicode_CheckExact(unicode) &&
-        numfree < PyUnicode_MAXFREELIST) {
-        /* Keep-Alive optimization */
-        if (unicode->length >= KEEPALIVE_SIZE_LIMIT) {
-            PyObject_DEL(unicode->str);
-            unicode->str = NULL;
-            unicode->length = 0;
-        }
-        if (unicode->defenc) {
-            Py_CLEAR(unicode->defenc);
-        }
-        /* Add to free list */
-        *(PyUnicodeObject **)unicode = free_list;
-        free_list = unicode;
-        numfree++;
+    if (_PyUnicode_WSTR(unicode) &&
+        (!PyUnicode_IS_READY(unicode) ||
+         _PyUnicode_WSTR(unicode) != PyUnicode_DATA(unicode)))
+        PyObject_DEL(_PyUnicode_WSTR(unicode));
+    if (_PyUnicode_UTF8(unicode) && _PyUnicode_UTF8(unicode) != PyUnicode_DATA(unicode))
+        PyObject_DEL(unicode->_base.utf8);
+
+    if (PyUnicode_IS_COMPACT(unicode)) {
+        Py_TYPE(unicode)->tp_free((PyObject *)unicode);
     }
     else {
-        PyObject_DEL(unicode->str);
-        Py_XDECREF(unicode->defenc);
+        if (unicode->data.any)
+            PyObject_DEL(unicode->data.any);
         Py_TYPE(unicode)->tp_free((PyObject *)unicode);
     }
 }
@@ -443,21 +935,26 @@
         return -1;
     }
     v = *unicode;
-    if (v == NULL || !PyUnicode_Check(v) || Py_REFCNT(v) != 1 || length < 0) {
+    if (v == NULL || !PyUnicode_Check(v) || Py_REFCNT(v) != 1 || length < 0 ||
+        PyUnicode_IS_COMPACT(v) || _PyUnicode_WSTR(v) == NULL) {
         PyErr_BadInternalCall();
         return -1;
     }
 
     /* Resizing unicode_empty and single character objects is not
-       possible since these are being shared. We simply return a fresh
-       copy with the same Unicode content. */
-    if (v->length != length &&
-        (v == unicode_empty || v->length == 1)) {
+       possible since these are being shared.
+       The same goes for new-representation unicode objects or objects which
+       have already been readied.
+       For these, we simply return a fresh copy with the same Unicode content.
+       */
+    if ((_PyUnicode_WSTR_LENGTH(v) != length &&
+         (v == unicode_empty || _PyUnicode_WSTR_LENGTH(v) == 1)) ||
+        PyUnicode_IS_COMPACT(v) || v->data.any) {
         PyUnicodeObject *w = _PyUnicode_New(length);
         if (w == NULL)
             return -1;
-        Py_UNICODE_COPY(w->str, v->str,
-                        length < v->length ? length : v->length);
+        Py_UNICODE_COPY(_PyUnicode_WSTR(w), _PyUnicode_WSTR(v),
+                        length < _PyUnicode_WSTR_LENGTH(v) ? length : _PyUnicode_WSTR_LENGTH(v));
         Py_DECREF(*unicode);
         *unicode = w;
         return 0;
@@ -474,44 +971,85 @@
     return _PyUnicode_Resize((PyUnicodeObject **)unicode, length);
 }
 
+static PyObject*
+get_latin1_char(unsigned char ch)
+{
+    PyUnicodeObject *unicode = unicode_latin1[ch];
+    if (!unicode) {
+        unicode = (PyUnicodeObject *)PyUnicode_New(1, ch);
+        if (!unicode)
+            return NULL;
+        PyUnicode_1BYTE_DATA(unicode)[0] = ch;
+        unicode_latin1[ch] = unicode;
+    }
+    Py_INCREF(unicode);
+    return (PyObject *)unicode;
+}
+
 PyObject *
 PyUnicode_FromUnicode(const Py_UNICODE *u, Py_ssize_t size)
 {
     PyUnicodeObject *unicode;
+    Py_UCS4 maxchar = 0;
+    Py_ssize_t num_surrogates;
+
+    if (u == NULL)
+        return (PyObject*)_PyUnicode_New(size);
 
     /* If the Unicode data is known at construction time, we can apply
        some optimizations which share commonly used objects. */
-    if (u != NULL) {
-
-        /* Optimization for empty strings */
-        if (size == 0 && unicode_empty != NULL) {
-            Py_INCREF(unicode_empty);
-            return (PyObject *)unicode_empty;
-        }
-
-        /* Single character Unicode objects in the Latin-1 range are
-           shared when using this constructor */
-        if (size == 1 && *u < 256) {
-            unicode = unicode_latin1[*u];
-            if (!unicode) {
-                unicode = _PyUnicode_New(1);
-                if (!unicode)
-                    return NULL;
-                unicode->str[0] = *u;
-                unicode_latin1[*u] = unicode;
-            }
-            Py_INCREF(unicode);
-            return (PyObject *)unicode;
-        }
-    }
-
-    unicode = _PyUnicode_New(size);
+
+    /* Optimization for empty strings */
+    if (size == 0 && unicode_empty != NULL) {
+        Py_INCREF(unicode_empty);
+        return (PyObject *)unicode_empty;
+    }
+
+    /* Single character Unicode objects in the Latin-1 range are
+       shared when using this constructor */
+    if (size == 1 && *u < 256)
+        return get_latin1_char((unsigned char)*u);
+
+    /* If not empty and not single character, copy the Unicode data
+       into the new object */
+    if (_PyUnicode_FindMaxCharAndNumSurrogatePairs(u, u + size, &maxchar,
+                                                  &num_surrogates) == -1)
+        return NULL;
+
+    unicode = (PyUnicodeObject *) PyUnicode_New(size - num_surrogates,
+                                                maxchar);
     if (!unicode)
         return NULL;
 
-    /* Copy the Unicode data into the new object */
-    if (u != NULL)
-        Py_UNICODE_COPY(unicode->str, u, size);
+    switch (PyUnicode_KIND(unicode)) {
+    case PyUnicode_1BYTE_KIND:
+        PyUnicode_CONVERT_BYTES(Py_UNICODE, unsigned char,
+                                u, u + size, PyUnicode_1BYTE_DATA(unicode));
+        break;
+    case PyUnicode_2BYTE_KIND:
+#if Py_UNICODE_SIZE == 2
+        Py_MEMCPY(PyUnicode_2BYTE_DATA(unicode), u, size * 2);
+#else
+        PyUnicode_CONVERT_BYTES(Py_UNICODE, Py_UCS2,
+                                u, u + size, PyUnicode_2BYTE_DATA(unicode));
+#endif
+        break;
+    case PyUnicode_4BYTE_KIND:
+#if SIZEOF_WCHAR_T == 2
+        /* This is the only case which has to process surrogates, thus
+           a simple copy loop is not enough and we need a function. */
+        if (unicode_convert_wchar_to_ucs4(u, u + size, unicode) < 0) {
+            Py_DECREF(unicode);
+            return NULL;
+        }
+#else
+        assert(num_surrogates == 0);
+        Py_MEMCPY(PyUnicode_4BYTE_DATA(unicode), u, size * 4);
+#endif
+        break;
+    default:
+        assert(0 && "Impossible state");
+    }
 
     return (PyObject *)unicode;
 }
@@ -541,18 +1079,8 @@
 
         /* Single characters are shared when using this constructor.
            Restrict to ASCII, since the input must be UTF-8. */
-        if (size == 1 && Py_CHARMASK(*u) < 128) {
-            unicode = unicode_latin1[Py_CHARMASK(*u)];
-            if (!unicode) {
-                unicode = _PyUnicode_New(1);
-                if (!unicode)
-                    return NULL;
-                unicode->str[0] = Py_CHARMASK(*u);
-                unicode_latin1[Py_CHARMASK(*u)] = unicode;
-            }
-            Py_INCREF(unicode);
-            return (PyObject *)unicode;
-        }
+        if (size == 1 && Py_CHARMASK(*u) < 128)
+            return get_latin1_char(Py_CHARMASK(*u));
 
         return PyUnicode_DecodeUTF8(u, size, NULL);
     }
@@ -576,28 +1104,196 @@
     return PyUnicode_FromStringAndSize(u, size);
 }
 
+PyObject*
+PyUnicode_FromUCS1(const unsigned char* u, Py_ssize_t size)
+{
+    PyObject *res;
+    unsigned char max = 127;
+    Py_ssize_t i;
+    for (i = 0; i < size; i++) {
+        if (u[i] & 0x80) {
+            max = 255;
+            break;
+        }
+    }
+    res = PyUnicode_New(size, max);
+    if (!res)
+        return NULL;
+    memcpy(PyUnicode_1BYTE_DATA(res), u, size);
+    return res;
+}
+
+PyObject*
+PyUnicode_FromUCS2(const Py_UCS2 *u, Py_ssize_t size)
+{
+    PyObject *res;
+    Py_UCS2 max = 0;
+    Py_ssize_t i;
+    for (i = 0; i < size; i++)
+        if (u[i] > max)
+            max = u[i];
+    res = PyUnicode_New(size, max);
+    if (!res)
+        return NULL;
+    if (max >= 256)
+        memcpy(PyUnicode_2BYTE_DATA(res), u, sizeof(Py_UCS2)*size);
+    else
+        for (i = 0; i < size; i++)
+            PyUnicode_1BYTE_DATA(res)[i] = (Py_UCS1)u[i];
+    return res;
+}
+
+PyObject*
+PyUnicode_FromUCS4(const Py_UCS4 *u, Py_ssize_t size)
+{
+    PyObject *res;
+    Py_UCS4 max = 0;
+    Py_ssize_t i;
+    for (i = 0; i < size; i++)
+        if (u[i] > max)
+            max = u[i];
+    res = PyUnicode_New(size, max);
+    if (!res)
+        return NULL;
+    if (max >= 0x10000)
+        memcpy(PyUnicode_4BYTE_DATA(res), u, sizeof(Py_UCS4)*size);
+    else {
+        int kind = PyUnicode_KIND(res);
+        void *data = PyUnicode_DATA(res);
+        for (i = 0; i < size; i++)
+            PyUnicode_WRITE(kind, data, i, u[i]);
+    }
+    return res;
+}
+
+PyObject*
+PyUnicode_FromKindAndData(int kind, const void *buffer, Py_ssize_t size)
+{
+    switch(kind) {
+    case PyUnicode_1BYTE_KIND:
+        return PyUnicode_FromUCS1(buffer, size);
+    case PyUnicode_2BYTE_KIND:
+        return PyUnicode_FromUCS2(buffer, size);
+    case PyUnicode_4BYTE_KIND:
+        return PyUnicode_FromUCS4(buffer, size);
+    }
+    assert(0);
+    return NULL;
+}
+
+
+/* Widen Unicode objects to larger buffers.
+   Return NULL if the string is too wide already. */
+
+void*
+_PyUnicode_AsKind(PyObject *s, unsigned int kind)
+{
+    Py_ssize_t i;
+    Py_ssize_t len = PyUnicode_GET_LENGTH(s);
+    void *d = PyUnicode_DATA(s);
+    unsigned int skind = PyUnicode_KIND(s);
+    if (PyUnicode_KIND(s) >= kind) {
+        PyErr_SetString(PyExc_RuntimeError, "invalid widening attempt");
+        return NULL;
+    }
+    switch(kind) {
+    case PyUnicode_2BYTE_KIND: {
+        Py_UCS2 *result = PyMem_Malloc(PyUnicode_GET_LENGTH(s) * sizeof(Py_UCS2));
+        if (!result) {
+            PyErr_NoMemory();
+            return 0;
+        }
+        for (i = 0; i < len; i++)
+            result[i] = ((Py_UCS1*)d)[i];
+        return result;
+    }
+    case PyUnicode_4BYTE_KIND: {
+        Py_UCS4 *result = PyMem_Malloc(PyUnicode_GET_LENGTH(s) * sizeof(Py_UCS4));
+        if (!result) {
+            PyErr_NoMemory();
+            return 0;
+        }
+        for (i = 0; i < len; i++)
+            result[i] = PyUnicode_READ(skind, d, i);
+        return result;
+    }
+    }
+    Py_FatalError("invalid kind");
+    return NULL;
+}
+
+static Py_UCS4*
+as_ucs4(PyObject *string, Py_UCS4 *target, Py_ssize_t targetsize,
+        int copy_null)
+{
+    int kind;
+    void *data;
+    Py_ssize_t len, targetlen;
+    if (PyUnicode_READY(string) == -1)
+        return NULL;
+    kind = PyUnicode_KIND(string);
+    data = PyUnicode_DATA(string);
+    len = PyUnicode_GET_LENGTH(string);
+    targetlen = len;
+    if (copy_null)
+        targetlen++;
+    if (!target) {
+        if (PY_SSIZE_T_MAX / sizeof(Py_UCS4) < targetlen) {
+            PyErr_NoMemory();
+            return NULL;
+        }
+        target = PyMem_Malloc(targetlen * sizeof(Py_UCS4));
+        if (!target) {
+            PyErr_NoMemory();
+            return NULL;
+        }
+    }
+    else {
+        if (targetsize < targetlen) {
+            PyErr_Format(PyExc_SystemError,
+                         "string is longer than the buffer");
+            if (copy_null && 0 < targetsize)
+                target[0] = 0;
+            return NULL;
+        }
+    }
+    if (kind != PyUnicode_4BYTE_KIND) {
+        Py_ssize_t i;
+        for (i = 0; i < len; i++)
+            target[i] = PyUnicode_READ(kind, data, i);
+    }
+    else
+        Py_MEMCPY(target, data, len * sizeof(Py_UCS4));
+    if (copy_null)
+        target[len] = 0;
+    return target;
+}
+
+Py_UCS4*
+PyUnicode_AsUCS4(PyObject *string, Py_UCS4 *target, Py_ssize_t targetsize,
+                 int copy_null)
+{
+    if (target == NULL || targetsize < 1) {
+        PyErr_BadInternalCall();
+        return NULL;
+    }
+    return as_ucs4(string, target, targetsize, copy_null);
+}
+
+Py_UCS4*
+PyUnicode_AsUCS4Copy(PyObject *string)
+{
+    return as_ucs4(string, NULL, 0, 1);
+}
+
 #ifdef HAVE_WCHAR_H
 
-#if (Py_UNICODE_SIZE == 2) && defined(SIZEOF_WCHAR_T) && (SIZEOF_WCHAR_T == 4)
-# define CONVERT_WCHAR_TO_SURROGATES
-#endif
-
-#ifdef CONVERT_WCHAR_TO_SURROGATES
-
-/* Here sizeof(wchar_t) is 4 but Py_UNICODE_SIZE == 2, so we need
-   to convert from UTF32 to UTF16. */
-
 PyObject *
 PyUnicode_FromWideChar(register const wchar_t *w, Py_ssize_t size)
 {
-    PyUnicodeObject *unicode;
-    register Py_ssize_t i;
-    Py_ssize_t alloc;
-    const wchar_t *orig_w;
-
     if (w == NULL) {
         if (size == 0)
-            return PyUnicode_FromStringAndSize(NULL, 0);
+            return PyUnicode_New(0, 0);
         PyErr_BadInternalCall();
         return NULL;
     }
@@ -606,77 +1302,10 @@
         size = wcslen(w);
     }
 
-    alloc = size;
-    orig_w = w;
-    for (i = size; i > 0; i--) {
-        if (*w > 0xFFFF)
-            alloc++;
-        w++;
-    }
-    w = orig_w;
-    unicode = _PyUnicode_New(alloc);
-    if (!unicode)
-        return NULL;
-
-    /* Copy the wchar_t data into the new object */
-    {
-        register Py_UNICODE *u;
-        u = PyUnicode_AS_UNICODE(unicode);
-        for (i = size; i > 0; i--) {
-            if (*w > 0xFFFF) {
-                wchar_t ordinal = *w++;
-                ordinal -= 0x10000;
-                *u++ = 0xD800 | (ordinal >> 10);
-                *u++ = 0xDC00 | (ordinal & 0x3FF);
-            }
-            else
-                *u++ = *w++;
-        }
-    }
-    return (PyObject *)unicode;
-}
-
-#else
-
-PyObject *
-PyUnicode_FromWideChar(register const wchar_t *w, Py_ssize_t size)
-{
-    PyUnicodeObject *unicode;
-
-    if (w == NULL) {
-        if (size == 0)
-            return PyUnicode_FromStringAndSize(NULL, 0);
-        PyErr_BadInternalCall();
-        return NULL;
-    }
-
-    if (size == -1) {
-        size = wcslen(w);
-    }
-
-    unicode = _PyUnicode_New(size);
-    if (!unicode)
-        return NULL;
-
-    /* Copy the wchar_t data into the new object */
-#if Py_UNICODE_SIZE == SIZEOF_WCHAR_T
-    memcpy(unicode->str, w, size * sizeof(wchar_t));
-#else
-    {
-        register Py_UNICODE *u;
-        register Py_ssize_t i;
-        u = PyUnicode_AS_UNICODE(unicode);
-        for (i = size; i > 0; i--)
-            *u++ = *w++;
-    }
-#endif
-
-    return (PyObject *)unicode;
-}
-
-#endif /* CONVERT_WCHAR_TO_SURROGATES */
-
-#undef CONVERT_WCHAR_TO_SURROGATES
+    return PyUnicode_FromUnicode(w, size);
+}
+
+#endif /* HAVE_WCHAR_H */
 
 static void
 makefmt(char *fmt, int longflag, int longlongflag, int size_tflag,
@@ -779,10 +1408,6 @@
     return f;
 }
 
-#define appendstring(string) {for (copy = string;*copy;) *s++ = *copy++;}
-
-/* size of fixed-size buffer for formatting single arguments */
-#define ITEM_BUFFER_LEN 21
 /* maximum number of characters required for output of %ld.  21 characters
    allows for 64-bit integers (in decimal) and an optional sign. */
 #define MAX_LONG_CHARS 21
@@ -803,80 +1428,35 @@
     int precision = 0;
     int zeropad;
     const char* f;
-    Py_UNICODE *s;
-    PyObject *string;
+    PyUnicodeObject *string;
     /* used by sprintf */
-    char buffer[ITEM_BUFFER_LEN+1];
-    /* use abuffer instead of buffer, if we need more space
-     * (which can happen if there's a format specifier with width). */
-    char *abuffer = NULL;
-    char *realbuffer;
-    Py_ssize_t abuffersize = 0;
     char fmt[61]; /* should be enough for %0width.precisionlld */
-    const char *copy;
+    Py_UCS4 maxchar = 127; /* result is ASCII by default */
+    Py_UCS4 argmaxchar;
+    Py_ssize_t numbersize = 0;
+    char *numberresults = NULL;
+    char *numberresult = NULL;
+    Py_ssize_t i;
+    int kind;
+    void *data;
 
     Py_VA_COPY(count, vargs);
     /* step 1: count the number of %S/%R/%A/%s format specifications
      * (we call PyObject_Str()/PyObject_Repr()/PyObject_ASCII()/
      * PyUnicode_DecodeUTF8() for these objects once during step 3 and put the
-     * result in an array) */
-    for (f = format; *f; f++) {
-         if (*f == '%') {
-             /* skip width or width.precision (eg. "1.2" of "%1.2f") */
-             f = parse_format_flags(f, NULL, NULL, NULL, NULL, NULL);
-             if (*f == 's' || *f=='S' || *f=='R' || *f=='A' || *f=='V')
-                 ++callcount;
-         }
-         else if (128 <= (unsigned char)*f) {
-             PyErr_Format(PyExc_ValueError,
-                "PyUnicode_FromFormatV() expects an ASCII-encoded format "
-                "string, got a non-ASCII byte: 0x%02x",
-                (unsigned char)*f);
-             return NULL;
-         }
-    }
-    /* step 2: allocate memory for the results of
-     * PyObject_Str()/PyObject_Repr()/PyUnicode_DecodeUTF8() calls */
-    if (callcount) {
-        callresults = PyObject_Malloc(sizeof(PyObject *)*callcount);
-        if (!callresults) {
-            PyErr_NoMemory();
-            return NULL;
-        }
-        callresult = callresults;
-    }
-    /* step 3: figure out how large a buffer we need */
+     * result in an array)
+     * also esimate a upper bound for all the number formats in the string,
+     * numbers will be formated in step 3 and be keept in a '\0'-separated
+     * buffer before putting everything together. */
     for (f = format; *f; f++) {
         if (*f == '%') {
-#ifdef HAVE_LONG_LONG
             int longlongflag;
-#endif
-            const char* p;
-
-            p = f;
-            f = parse_format_flags(f, &width, NULL,
-                                   NULL, &longlongflag, NULL);
-
-            switch (*f) {
-            case 'c':
-            {
-#ifndef Py_UNICODE_WIDE
-                int ordinal = va_arg(count, int);
-                if (ordinal > 0xffff)
-                    n += 2;
-                else
-                    n++;
-#else
-                (void)va_arg(count, int);
-                n++;
-#endif
-                break;
-            }
-            case '%':
-                n++;
-                break;
-            case 'd': case 'u': case 'i': case 'x':
-                (void) va_arg(count, int);
+            /* skip width or width.precision (eg. "1.2" of "%1.2f") */
+            f = parse_format_flags(f, &width, NULL, NULL, &longlongflag, NULL);
+            if (*f == 's' || *f=='S' || *f=='R' || *f=='A' || *f=='V')
+                ++callcount;
+
+            else if (*f == 'd' || *f=='u' || *f=='i' || *f=='x' || *f=='p') {
 #ifdef HAVE_LONG_LONG
                 if (longlongflag) {
                     if (width < MAX_LONG_LONG_CHARS)
@@ -890,10 +1470,142 @@
                        need more (which we allocate later). */
                     if (width < MAX_LONG_CHARS)
                         width = MAX_LONG_CHARS;
-                n += width;
-                /* XXX should allow for large precision here too. */
-                if (abuffersize < width)
-                    abuffersize = width;
+
+                /* account for the size + '\0' to separate numbers
+                   inside of the numberresults buffer */
+                numbersize += (width + 1);
+            }
+        }
+        else if ((unsigned char)*f > 127) {
+            PyErr_Format(PyExc_ValueError,
+                "PyUnicode_FromFormatV() expects an ASCII-encoded format "
+                "string, got a non-ASCII byte: 0x%02x",
+                (unsigned char)*f);
+            return NULL;
+        }
+    }
+    /* step 2: allocate memory for the results of
+     * PyObject_Str()/PyObject_Repr()/PyUnicode_DecodeUTF8() calls */
+    if (callcount) {
+        callresults = PyObject_Malloc(sizeof(PyObject *) * callcount);
+        if (!callresults) {
+            PyErr_NoMemory();
+            return NULL;
+        }
+        callresult = callresults;
+    }
+    /* step 2.5: allocate memory for the results of formating numbers */
+    if (numbersize) {
+        numberresults = PyObject_Malloc(numbersize);
+        if (!numberresults) {
+            PyErr_NoMemory();
+            goto fail;
+        }
+        numberresult = numberresults;
+    }
+
+    /* step 3: format numbers and figure out how large a buffer we need */
+    for (f = format; *f; f++) {
+        if (*f == '%') {
+            const char* p;
+            int longflag;
+            int longlongflag;
+            int size_tflag;
+            int numprinted;
+
+            p = f;
+            zeropad = (f[1] == '0');
+            f = parse_format_flags(f, &width, &precision,
+                                   &longflag, &longlongflag, &size_tflag);
+            switch (*f) {
+            case 'c':
+            {
+                Py_UCS4 ordinal = va_arg(count, int);
+                maxchar = PY_MAX(maxchar, ordinal);
+                n++;
+                break;
+            }
+            case '%':
+                n++;
+                break;
+            case 'i':
+            case 'd':
+                makefmt(fmt, longflag, longlongflag, size_tflag, zeropad,
+                        width, precision, *f);
+                if (longflag)
+                    numprinted = sprintf(numberresult, fmt,
+                                         va_arg(count, long));
+#ifdef HAVE_LONG_LONG
+                else if (longlongflag)
+                    numprinted = sprintf(numberresult, fmt,
+                                         va_arg(count, PY_LONG_LONG));
+#endif
+                else if (size_tflag)
+                    numprinted = sprintf(numberresult, fmt,
+                                         va_arg(count, Py_ssize_t));
+                else
+                    numprinted = sprintf(numberresult, fmt,
+                                         va_arg(count, int));
+                n += numprinted;
+                /* advance by +1 to skip over the '\0' */
+                numberresult += (numprinted + 1);
+                assert(*(numberresult - 1) == '\0');
+                assert(*(numberresult - 2) != '\0');
+                assert(numprinted >= 0);
+                assert(numberresult <= numberresults + numbersize);
+                break;
+            case 'u':
+                makefmt(fmt, longflag, longlongflag, size_tflag, zeropad,
+                        width, precision, 'u');
+                if (longflag)
+                    numprinted = sprintf(numberresult, fmt,
+                                         va_arg(count, unsigned long));
+#ifdef HAVE_LONG_LONG
+                else if (longlongflag)
+                    numprinted = sprintf(numberresult, fmt,
+                                         va_arg(count, unsigned PY_LONG_LONG));
+#endif
+                else if (size_tflag)
+                    numprinted = sprintf(numberresult, fmt,
+                                         va_arg(count, size_t));
+                else
+                    numprinted = sprintf(numberresult, fmt,
+                                         va_arg(count, unsigned int));
+                n += numprinted;
+                numberresult += (numprinted + 1);
+                assert(*(numberresult - 1) == '\0');
+                assert(*(numberresult - 2) != '\0');
+                assert(numprinted >= 0);
+                assert(numberresult <= numberresults + numbersize);
+                break;
+            case 'x':
+                makefmt(fmt, 0, 0, 0, zeropad, width, precision, 'x');
+                numprinted = sprintf(numberresult, fmt, va_arg(count, int));
+                n += numprinted;
+                numberresult += (numprinted + 1);
+                assert(*(numberresult - 1) == '\0');
+                assert(*(numberresult - 2) != '\0');
+                assert(numprinted >= 0);
+                assert(numberresult <= numberresults + numbersize);
+                break;
+            case 'p':
+                numprinted = sprintf(numberresult, "%p", va_arg(count, void*));
+                /* %p is ill-defined:  ensure leading 0x. */
+                if (numberresult[1] == 'X')
+                    numberresult[1] = 'x';
+                else if (numberresult[1] != 'x') {
+                    memmove(numberresult + 2, numberresult,
+                            strlen(numberresult) + 1);
+                    numberresult[0] = '0';
+                    numberresult[1] = 'x';
+                    numprinted += 2;
+                }
+                n += numprinted;
+                numberresult += (numprinted + 1);
+                assert(*(numberresult - 1) == '\0');
+                assert(*(numberresult - 2) != '\0');
+                assert(numprinted >= 0);
+                assert(numberresult <= numberresults + numbersize);
                 break;
             case 's':
             {
@@ -902,7 +1614,11 @@
                 PyObject *str = PyUnicode_DecodeUTF8(s, strlen(s), "replace");
                 if (!str)
                     goto fail;
-                n += PyUnicode_GET_SIZE(str);
+                /* since PyUnicode_DecodeUTF8 returns already flexible
+                   unicode objects, there is no need to call ready on them */
+                argmaxchar = PyUnicode_MAX_CHAR_VALUE(str);
+                maxchar = PY_MAX(maxchar, argmaxchar);
+                n += PyUnicode_GET_LENGTH(str);
                 /* Remember the str and switch to the next slot */
                 *callresult++ = str;
                 break;
@@ -911,7 +1627,11 @@
             {
                 PyObject *obj = va_arg(count, PyObject *);
                 assert(obj && PyUnicode_Check(obj));
-                n += PyUnicode_GET_SIZE(obj);
+                if (PyUnicode_READY(obj) == -1)
+                    goto fail;
+                argmaxchar = PyUnicode_MAX_CHAR_VALUE(obj);
+                maxchar = PY_MAX(maxchar, argmaxchar);
+                n += PyUnicode_GET_LENGTH(obj);
                 break;
             }
             case 'V':
@@ -922,14 +1642,20 @@
                 assert(obj || str);
                 assert(!obj || PyUnicode_Check(obj));
                 if (obj) {
-                    n += PyUnicode_GET_SIZE(obj);
+                    if (PyUnicode_READY(obj) == -1)
+                        goto fail;
+                    argmaxchar = PyUnicode_MAX_CHAR_VALUE(obj);
+                    maxchar = PY_MAX(maxchar, argmaxchar);
+                    n += PyUnicode_GET_LENGTH(obj);
                     *callresult++ = NULL;
                 }
                 else {
                     str_obj = PyUnicode_DecodeUTF8(str, strlen(str), "replace");
                     if (!str_obj)
                         goto fail;
-                    n += PyUnicode_GET_SIZE(str_obj);
+                    argmaxchar = PyUnicode_MAX_CHAR_VALUE(str_obj);
+                    maxchar = PY_MAX(maxchar, argmaxchar);
+                    n += PyUnicode_GET_LENGTH(str_obj);
                     *callresult++ = str_obj;
                 }
                 break;
@@ -940,9 +1666,11 @@
                 PyObject *str;
                 assert(obj);
                 str = PyObject_Str(obj);
-                if (!str)
+                if (!str || PyUnicode_READY(str) == -1)
                     goto fail;
-                n += PyUnicode_GET_SIZE(str);
+                argmaxchar = PyUnicode_MAX_CHAR_VALUE(str);
+                maxchar = PY_MAX(maxchar, argmaxchar);
+                n += PyUnicode_GET_LENGTH(str);
                 /* Remember the str and switch to the next slot */
                 *callresult++ = str;
                 break;
@@ -953,9 +1681,11 @@
                 PyObject *repr;
                 assert(obj);
                 repr = PyObject_Repr(obj);
-                if (!repr)
+                if (!repr || PyUnicode_READY(repr) == -1)
                     goto fail;
-                n += PyUnicode_GET_SIZE(repr);
+                argmaxchar = PyUnicode_MAX_CHAR_VALUE(repr);
+                maxchar = PY_MAX(maxchar, argmaxchar);
+                n += PyUnicode_GET_LENGTH(repr);
                 /* Remember the repr and switch to the next slot */
                 *callresult++ = repr;
                 break;
@@ -966,22 +1696,15 @@
                 PyObject *ascii;
                 assert(obj);
                 ascii = PyObject_ASCII(obj);
-                if (!ascii)
+                if (!ascii || PyUnicode_READY(ascii) == -1)
                     goto fail;
-                n += PyUnicode_GET_SIZE(ascii);
+                argmaxchar = PyUnicode_MAX_CHAR_VALUE(ascii);
+                maxchar = PY_MAX(maxchar, argmaxchar);
+                n += PyUnicode_GET_LENGTH(ascii);
                 /* Remember the repr and switch to the next slot */
                 *callresult++ = ascii;
                 break;
             }
-            case 'p':
-                (void) va_arg(count, int);
-                /* maximum 64-bit pointer representation:
-                 * 0xffffffffffffffff
-                 * so 19 characters is enough.
-                 * XXX I count 18 -- what's the extra for?
-                 */
-                n += 19;
-                break;
             default:
                 /* if we stumble upon an unknown
                    formatting code, copy the rest of
@@ -996,98 +1719,65 @@
             n++;
     }
   expand:
-    if (abuffersize > ITEM_BUFFER_LEN) {
-        /* add 1 for sprintf's trailing null byte */
-        abuffer = PyObject_Malloc(abuffersize + 1);
-        if (!abuffer) {
-            PyErr_NoMemory();
-            goto fail;
-        }
-        realbuffer = abuffer;
-    }
-    else
-        realbuffer = buffer;
     /* step 4: fill the buffer */
-    /* Since we've analyzed how much space we need for the worst case,
+    /* Since we've analyzed how much space we need,
        we don't have to resize the string.
        There can be no errors beyond this point. */
-    string = PyUnicode_FromUnicode(NULL, n);
+    string = (PyUnicodeObject *)PyUnicode_New(n, maxchar);
     if (!string)
         goto fail;
-
-    s = PyUnicode_AS_UNICODE(string);
+    kind = PyUnicode_KIND(string);
+    data = PyUnicode_DATA(string);
     callresult = callresults;
-
-    for (f = format; *f; f++) {
+    numberresult = numberresults;
+
+    for (i = 0, f = format; *f; f++) {
         if (*f == '%') {
             const char* p;
-            int longflag;
-            int longlongflag;
-            int size_tflag;
 
             p = f;
-            zeropad = (f[1] == '0');
-            f = parse_format_flags(f, &width, &precision,
-                                   &longflag, &longlongflag, &size_tflag);
+            f = parse_format_flags(f, NULL, NULL, NULL, NULL, NULL);
+            /* checking for == because the last argument could be a empty
+               string, which causes i to point to end, the assert at the end of
+               the loop */
+            assert(i <= PyUnicode_GET_LENGTH(string));
 
             switch (*f) {
             case 'c':
             {
-                int ordinal = va_arg(vargs, int);
-#ifndef Py_UNICODE_WIDE
-                if (ordinal > 0xffff) {
-                    ordinal -= 0x10000;
-                    *s++ = 0xD800 | (ordinal >> 10);
-                    *s++ = 0xDC00 | (ordinal & 0x3FF);
-                } else
-#endif
-                *s++ = ordinal;
+                const int ordinal = va_arg(vargs, int);
+                PyUnicode_WRITE(kind, data, i++, ordinal);
                 break;
             }
             case 'i':
             case 'd':
-                makefmt(fmt, longflag, longlongflag, size_tflag, zeropad,
-                        width, precision, *f);
-                if (longflag)
-                    sprintf(realbuffer, fmt, va_arg(vargs, long));
-#ifdef HAVE_LONG_LONG
-                else if (longlongflag)
-                    sprintf(realbuffer, fmt, va_arg(vargs, PY_LONG_LONG));
-#endif
-                else if (size_tflag)
-                    sprintf(realbuffer, fmt, va_arg(vargs, Py_ssize_t));
+            case 'u':
+            case 'x':
+            case 'p':
+                /* unused, since we already have the result */
+                if (*f == 'p')
+                    (void) va_arg(vargs, void *);
                 else
-                    sprintf(realbuffer, fmt, va_arg(vargs, int));
-                appendstring(realbuffer);
-                break;
-            case 'u':
-                makefmt(fmt, longflag, longlongflag, size_tflag, zeropad,
-                        width, precision, 'u');
-                if (longflag)
-                    sprintf(realbuffer, fmt, va_arg(vargs, unsigned long));
-#ifdef HAVE_LONG_LONG
-                else if (longlongflag)
-                    sprintf(realbuffer, fmt, va_arg(vargs,
-                                                    unsigned PY_LONG_LONG));
-#endif
-                else if (size_tflag)
-                    sprintf(realbuffer, fmt, va_arg(vargs, size_t));
-                else
-                    sprintf(realbuffer, fmt, va_arg(vargs, unsigned int));
-                appendstring(realbuffer);
-                break;
-            case 'x':
-                makefmt(fmt, 0, 0, 0, zeropad, width, precision, 'x');
-                sprintf(realbuffer, fmt, va_arg(vargs, int));
-                appendstring(realbuffer);
+                    (void) va_arg(vargs, int);
+                /* extract the result from numberresults and append. */
+                for (; *numberresult; ++i, ++numberresult)
+                    PyUnicode_WRITE(kind, data, i, *numberresult);
+                /* skip over the separating '\0' */
+                assert(*numberresult == '\0');
+                numberresult++;
+                assert(numberresult <= numberresults + numbersize);
                 break;
             case 's':
             {
                 /* unused, since we already have the result */
+                Py_ssize_t size;
                 (void) va_arg(vargs, char *);
-                Py_UNICODE_COPY(s, PyUnicode_AS_UNICODE(*callresult),
-                                PyUnicode_GET_SIZE(*callresult));
-                s += PyUnicode_GET_SIZE(*callresult);
+                size = PyUnicode_GET_LENGTH(*callresult);
+                assert(PyUnicode_KIND(*callresult) <= PyUnicode_KIND(string));
+                PyUnicode_CopyCharacters((PyObject*)string, i,
+                                         *callresult, 0,
+                                         size);
+                i += size;
                 /* We're done with the unicode()/repr() => forget it */
                 Py_DECREF(*callresult);
                 /* switch to next unicode()/repr() result */
@@ -1097,23 +1787,35 @@
             case 'U':
             {
                 PyObject *obj = va_arg(vargs, PyObject *);
-                Py_ssize_t size = PyUnicode_GET_SIZE(obj);
-                Py_UNICODE_COPY(s, PyUnicode_AS_UNICODE(obj), size);
-                s += size;
+                Py_ssize_t size;
+                assert(PyUnicode_KIND(obj) <= PyUnicode_KIND(string));
+                size = PyUnicode_GET_LENGTH(obj);
+                PyUnicode_CopyCharacters((PyObject*)string, i,
+                                         obj, 0,
+                                         size);
+                i += size;
                 break;
             }
             case 'V':
             {
+                Py_ssize_t size;
                 PyObject *obj = va_arg(vargs, PyObject *);
                 va_arg(vargs, const char *);
                 if (obj) {
-                    Py_ssize_t size = PyUnicode_GET_SIZE(obj);
-                    Py_UNICODE_COPY(s, PyUnicode_AS_UNICODE(obj), size);
-                    s += size;
+                    size = PyUnicode_GET_LENGTH(obj);
+                    assert(PyUnicode_KIND(obj) <= PyUnicode_KIND(string));
+                    PyUnicode_CopyCharacters((PyObject*)string, i,
+                                             obj, 0,
+                                             size);
+                    i += size;
                 } else {
-                    Py_UNICODE_COPY(s, PyUnicode_AS_UNICODE(*callresult),
-                                    PyUnicode_GET_SIZE(*callresult));
-                    s += PyUnicode_GET_SIZE(*callresult);
+                    size = PyUnicode_GET_LENGTH(*callresult);
+                    assert(PyUnicode_KIND(*callresult) <=
+                           PyUnicode_KIND(string));
+                    PyUnicode_CopyCharacters((PyObject*)string, i,
+                                             *callresult,
+                                             0, size);
+                    i += size;
                     Py_DECREF(*callresult);
                 }
                 ++callresult;
@@ -1123,52 +1825,42 @@
             case 'R':
             case 'A':
             {
-                Py_UNICODE *ucopy;
-                Py_ssize_t usize;
-                Py_ssize_t upos;
                 /* unused, since we already have the result */
                 (void) va_arg(vargs, PyObject *);
-                ucopy = PyUnicode_AS_UNICODE(*callresult);
-                usize = PyUnicode_GET_SIZE(*callresult);
-                for (upos = 0; upos<usize;)
-                    *s++ = ucopy[upos++];
+                assert(PyUnicode_KIND(*callresult) <= PyUnicode_KIND(string));
+                PyUnicode_CopyCharacters((PyObject*)string, i,
+                                         *callresult, 0,
+                                         PyUnicode_GET_LENGTH(*callresult));
+                i += PyUnicode_GET_LENGTH(*callresult);
                 /* We're done with the unicode()/repr() => forget it */
                 Py_DECREF(*callresult);
                 /* switch to next unicode()/repr() result */
                 ++callresult;
                 break;
             }
-            case 'p':
-                sprintf(buffer, "%p", va_arg(vargs, void*));
-                /* %p is ill-defined:  ensure leading 0x. */
-                if (buffer[1] == 'X')
-                    buffer[1] = 'x';
-                else if (buffer[1] != 'x') {
-                    memmove(buffer+2, buffer, strlen(buffer)+1);
-                    buffer[0] = '0';
-                    buffer[1] = 'x';
-                }
-                appendstring(buffer);
-                break;
             case '%':
-                *s++ = '%';
+                PyUnicode_WRITE(kind, data, i++, '%');
                 break;
             default:
-                appendstring(p);
+                for (; *p; ++p, ++i)
+                    PyUnicode_WRITE(kind, data, i, *p);
+                assert(i == PyUnicode_GET_LENGTH(string));
                 goto end;
             }
         }
-        else
-            *s++ = *f;
-    }
+        else {
+            assert(i < PyUnicode_GET_LENGTH(string));
+            PyUnicode_WRITE(kind, data, i++, *f);
+        }
+    }
+    assert(i == PyUnicode_GET_LENGTH(string));
 
   end:
     if (callresults)
         PyObject_Free(callresults);
-    if (abuffer)
-        PyObject_Free(abuffer);
-    PyUnicode_Resize(&string, s - PyUnicode_AS_UNICODE(string));
-    return string;
+    if (numberresults)
+        PyObject_Free(numberresults);
+    return (PyObject *)string;
   fail:
     if (callresults) {
         PyObject **callresult2 = callresults;
@@ -1178,13 +1870,11 @@
         }
         PyObject_Free(callresults);
     }
-    if (abuffer)
-        PyObject_Free(abuffer);
+    if (numberresults)
+        PyObject_Free(numberresults);
     return NULL;
 }
 
-#undef appendstring
-
 PyObject *
 PyUnicode_FromFormat(const char *format, ...)
 {
@@ -1201,6 +1891,8 @@
     return ret;
 }
 
+#ifdef HAVE_WCHAR_H
+
 /* Helper function for PyUnicode_AsWideChar() and PyUnicode_AsWideCharString():
    convert a Unicode object to a wide character string.
 
@@ -1215,99 +1907,23 @@
                    wchar_t *w,
                    Py_ssize_t size)
 {
-#if Py_UNICODE_SIZE == SIZEOF_WCHAR_T
     Py_ssize_t res;
+    const wchar_t *wstr;
+
+    wstr = PyUnicode_AsUnicodeAndSize((PyObject *)unicode, &res);
+    if (wstr == NULL)
+        return -1;
+
     if (w != NULL) {
-        res = PyUnicode_GET_SIZE(unicode);
         if (size > res)
             size = res + 1;
         else
             res = size;
-        memcpy(w, unicode->str, size * sizeof(wchar_t));
+        Py_MEMCPY(w, wstr, size * sizeof(wchar_t));
         return res;
     }
     else
-        return PyUnicode_GET_SIZE(unicode) + 1;
-#elif Py_UNICODE_SIZE == 2 && SIZEOF_WCHAR_T == 4
-    register const Py_UNICODE *u;
-    const Py_UNICODE *uend;
-    const wchar_t *worig, *wend;
-    Py_ssize_t nchar;
-
-    u = PyUnicode_AS_UNICODE(unicode);
-    uend = u + PyUnicode_GET_SIZE(unicode);
-    if (w != NULL) {
-        worig = w;
-        wend = w + size;
-        while (u != uend && w != wend) {
-            if (0xD800 <= u[0] && u[0] <= 0xDBFF
-                && 0xDC00 <= u[1] && u[1] <= 0xDFFF)
-            {
-                *w = (((u[0] & 0x3FF) << 10) | (u[1] & 0x3FF)) + 0x10000;
-                u += 2;
-            }
-            else {
-                *w = *u;
-                u++;
-            }
-            w++;
-        }
-        if (w != wend)
-            *w = L'\0';
-        return w - worig;
-    }
-    else {
-        nchar = 1; /* null character at the end */
-        while (u != uend) {
-            if (0xD800 <= u[0] && u[0] <= 0xDBFF
-                && 0xDC00 <= u[1] && u[1] <= 0xDFFF)
-                u += 2;
-            else
-                u++;
-            nchar++;
-        }
-    }
-    return nchar;
-#elif Py_UNICODE_SIZE == 4 && SIZEOF_WCHAR_T == 2
-    register Py_UNICODE *u, *uend, ordinal;
-    register Py_ssize_t i;
-    wchar_t *worig, *wend;
-    Py_ssize_t nchar;
-
-    u = PyUnicode_AS_UNICODE(unicode);
-    uend = u + PyUnicode_GET_SIZE(u);
-    if (w != NULL) {
-        worig = w;
-        wend = w + size;
-        while (u != uend && w != wend) {
-            ordinal = *u;
-            if (ordinal > 0xffff) {
-                ordinal -= 0x10000;
-                *w++ = 0xD800 | (ordinal >> 10);
-                *w++ = 0xDC00 | (ordinal & 0x3FF);
-            }
-            else
-                *w++ = ordinal;
-            u++;
-        }
-        if (w != wend)
-            *w = 0;
-        return w - worig;
-    }
-    else {
-        nchar = 1; /* null character */
-        while (u != uend) {
-            if (*u > 0xffff)
-                nchar += 2;
-            else
-                nchar++;
-            u++;
-        }
-        return nchar;
-    }
-#else
-#  error "unsupported wchar_t and Py_UNICODE sizes, see issue #8670"
-#endif
+        return res + 1;
 }
 
 Py_ssize_t
@@ -1335,6 +1951,8 @@
     }
 
     buflen = unicode_aswidechar((PyUnicodeObject *)unicode, NULL, 0);
+    if (buflen == -1)
+        return NULL;
     if (PY_SSIZE_T_MAX / sizeof(wchar_t) < buflen) {
         PyErr_NoMemory();
         return NULL;
@@ -1346,35 +1964,33 @@
         return NULL;
     }
     buflen = unicode_aswidechar((PyUnicodeObject *)unicode, buffer, buflen);
+    if (buflen == -1)
+        return NULL;
     if (size != NULL)
         *size = buflen;
     return buffer;
 }
 
-#endif
+#endif /* HAVE_WCHAR_H */
 
 PyObject *
 PyUnicode_FromOrdinal(int ordinal)
 {
-    Py_UNICODE s[2];
-
+    PyObject *v;
     if (ordinal < 0 || ordinal > 0x10ffff) {
         PyErr_SetString(PyExc_ValueError,
                         "chr() arg not in range(0x110000)");
         return NULL;
     }
 
-#ifndef Py_UNICODE_WIDE
-    if (ordinal > 0xffff) {
-        ordinal -= 0x10000;
-        s[0] = 0xD800 | (ordinal >> 10);
-        s[1] = 0xDC00 | (ordinal & 0x3FF);
-        return PyUnicode_FromUnicode(s, 2);
-    }
-#endif
-
-    s[0] = (Py_UNICODE)ordinal;
-    return PyUnicode_FromUnicode(s, 1);
+    if (ordinal < 256)
+        return get_latin1_char(ordinal);
+
+    v = PyUnicode_New(1, ordinal);
+    if (v == NULL)
+        return NULL;
+    PyUnicode_WRITE(PyUnicode_KIND(v), PyUnicode_DATA(v), 0, ordinal);
+    return v;
 }
 
 PyObject *
@@ -1389,8 +2005,9 @@
     if (PyUnicode_Check(obj)) {
         /* For a Unicode subtype that's not a Unicode object,
            return a true Unicode object with the same data. */
-        return PyUnicode_FromUnicode(PyUnicode_AS_UNICODE(obj),
-                                     PyUnicode_GET_SIZE(obj));
+        if (PyUnicode_READY(obj) == -1)
+            return NULL;
+        return substring((PyUnicodeObject *)obj, 0, PyUnicode_GET_LENGTH(obj));
     }
     PyErr_Format(PyExc_TypeError,
                  "Can't convert '%.100s' object to str implicitly",
@@ -1536,6 +2153,10 @@
         goto onError;
     }
     Py_DECREF(buffer);
+    if (PyUnicode_READY(unicode)) {
+        Py_DECREF(unicode);
+        return NULL;
+    }
     return unicode;
 
   onError:
@@ -1649,9 +2270,7 @@
                                 PyUnicode_GET_SIZE(unicode),
                                 NULL);
 #elif defined(__APPLE__)
-    return PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(unicode),
-                                PyUnicode_GET_SIZE(unicode),
-                                "surrogateescape");
+    return _PyUnicode_AsUTF8String(unicode, "surrogateescape");
 #else
     PyInterpreterState *interp = PyThreadState_GET()->interp;
     /* Bootstrap check: if the filesystem codec is implemented in Python, we
@@ -1721,11 +2340,9 @@
 
     if (encoding == NULL) {
         if (errors == NULL || strcmp(errors, "strict") == 0)
-            return PyUnicode_AsUTF8String(unicode);
+            return _PyUnicode_AsUTF8String(unicode, NULL);
         else
-            return PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(unicode),
-                                        PyUnicode_GET_SIZE(unicode),
-                                        errors);
+            return _PyUnicode_AsUTF8String(unicode, errors);
     }
 
     /* Shortcuts for common default encodings */
@@ -1734,18 +2351,14 @@
             (strcmp(lower, "utf8") == 0))
         {
             if (errors == NULL || strcmp(errors, "strict") == 0)
-                return PyUnicode_AsUTF8String(unicode);
+                return _PyUnicode_AsUTF8String(unicode, NULL);
             else
-                return PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(unicode),
-                                            PyUnicode_GET_SIZE(unicode),
-                                            errors);
+                return _PyUnicode_AsUTF8String(unicode, errors);
         }
         else if ((strcmp(lower, "latin-1") == 0) ||
                  (strcmp(lower, "latin1") == 0) ||
                  (strcmp(lower, "iso-8859-1") == 0))
-            return PyUnicode_EncodeLatin1(PyUnicode_AS_UNICODE(unicode),
-                                          PyUnicode_GET_SIZE(unicode),
-                                          errors);
+            return _PyUnicode_AsLatin1String(unicode, errors);
 #ifdef HAVE_MBCS
         else if (strcmp(lower, "mbcs") == 0)
             return PyUnicode_EncodeMBCS(PyUnicode_AS_UNICODE(unicode),
@@ -1753,9 +2366,7 @@
                                         errors);
 #endif
         else if (strcmp(lower, "ascii") == 0)
-            return PyUnicode_EncodeASCII(PyUnicode_AS_UNICODE(unicode),
-                                         PyUnicode_GET_SIZE(unicode),
-                                         errors);
+            return _PyUnicode_AsASCIIString(unicode, errors);
     }
 
     /* Encode via the codec registry */
@@ -1824,21 +2435,6 @@
     return NULL;
 }
 
-PyObject *
-_PyUnicode_AsDefaultEncodedString(PyObject *unicode)
-{
-    PyObject *v = ((PyUnicodeObject *)unicode)->defenc;
-    if (v)
-        return v;
-    v = PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(unicode),
-                             PyUnicode_GET_SIZE(unicode),
-                             NULL);
-    if (!v)
-        return NULL;
-    ((PyUnicodeObject *)unicode)->defenc = v;
-    return v;
-}
-
 PyObject*
 PyUnicode_DecodeFSDefault(const char *s) {
     Py_ssize_t size = (Py_ssize_t)strlen(s);
@@ -1935,13 +2531,13 @@
 PyUnicode_FSDecoder(PyObject* arg, void* addr)
 {
     PyObject *output = NULL;
-    Py_ssize_t size;
-    void *data;
     if (arg == NULL) {
         Py_DECREF(*(PyObject**)addr);
         return 1;
     }
     if (PyUnicode_Check(arg)) {
+        if (PyUnicode_READY(arg))
+            return 0;
         output = arg;
         Py_INCREF(output);
     }
@@ -1960,9 +2556,8 @@
             return 0;
         }
     }
-    size = PyUnicode_GET_SIZE(output);
-    data = PyUnicode_AS_UNICODE(output);
-    if (size != Py_UNICODE_strlen(data)) {
+    if (findchar(PyUnicode_DATA(output), PyUnicode_KIND(output),
+                 PyUnicode_GET_LENGTH(output), 0, 1)) {
         PyErr_SetString(PyExc_TypeError, "embedded NUL character");
         Py_DECREF(output);
         return 0;
@@ -1973,53 +2568,219 @@
 
 
 char*
-_PyUnicode_AsStringAndSize(PyObject *unicode, Py_ssize_t *psize)
+PyUnicode_AsUTF8AndSize(PyObject *unicode, Py_ssize_t *psize)
 {
     PyObject *bytes;
+    PyUnicodeObject *u = (PyUnicodeObject *)unicode;
+
     if (!PyUnicode_Check(unicode)) {
         PyErr_BadArgument();
         return NULL;
     }
-    bytes = _PyUnicode_AsDefaultEncodedString(unicode);
-    if (bytes == NULL)
-        return NULL;
-    if (psize != NULL)
-        *psize = PyBytes_GET_SIZE(bytes);
-    return PyBytes_AS_STRING(bytes);
+    if (PyUnicode_READY(u) == -1)
+        return NULL;
+
+    if (_PyUnicode_UTF8(unicode) == NULL) {
+        bytes = _PyUnicode_AsUTF8String(unicode, "strict");
+        if (bytes == NULL)
+            return NULL;
+        u->_base.utf8 = PyObject_MALLOC(PyBytes_GET_SIZE(bytes) + 1);
+        if (u->_base.utf8 == NULL) {
+            Py_DECREF(bytes);
+            return NULL;
+        }
+        u->_base.utf8_length = PyBytes_GET_SIZE(bytes);
+        Py_MEMCPY(u->_base.utf8, PyBytes_AS_STRING(bytes), u->_base.utf8_length + 1);
+        Py_DECREF(bytes);
+    }
+
+    if (psize)
+        *psize = _PyUnicode_UTF8_LENGTH(unicode);
+    return _PyUnicode_UTF8(unicode);
 }
 
 char*
-_PyUnicode_AsString(PyObject *unicode)
-{
-    return _PyUnicode_AsStringAndSize(unicode, NULL);
+PyUnicode_AsUTF8(PyObject *unicode)
+{
+    return PyUnicode_AsUTF8AndSize(unicode, NULL);
+}
+
+#ifdef Py_DEBUG
+int unicode_as_unicode_calls = 0;
+#endif
+
+
+Py_UNICODE *
+PyUnicode_AsUnicodeAndSize(PyObject *unicode, Py_ssize_t *size)
+{
+    PyUnicodeObject *u;
+    const unsigned char *one_byte;
+#if SIZEOF_WCHAR_T == 4
+    const Py_UCS2 *two_bytes;
+#else
+    const Py_UCS4 *four_bytes;
+    const Py_UCS4 *ucs4_end;
+    Py_ssize_t num_surrogates;
+#endif
+    wchar_t *w;
+    wchar_t *wchar_end;
+
+    if (!PyUnicode_Check(unicode)) {
+        PyErr_BadArgument();
+        return NULL;
+    }
+    u = (PyUnicodeObject*)unicode;
+    if (_PyUnicode_WSTR(u) == NULL) {
+        /* Non-ASCII compact unicode object */
+        assert(_PyUnicode_KIND(u) != 0);
+        assert(PyUnicode_IS_READY(u));
+
+#ifdef Py_DEBUG
+        ++unicode_as_unicode_calls;
+#endif
+
+        if (PyUnicode_KIND(u) == PyUnicode_4BYTE_KIND) {
+#if SIZEOF_WCHAR_T == 2
+            four_bytes = PyUnicode_4BYTE_DATA(u);
+            ucs4_end = four_bytes + _PyUnicode_LENGTH(u);
+            num_surrogates = 0;
+
+            for (; four_bytes < ucs4_end; ++four_bytes) {
+                if (*four_bytes > 0xFFFF)
+                    ++num_surrogates;
+            }
+
+            _PyUnicode_WSTR(u) = (wchar_t *) PyObject_MALLOC(
+                    sizeof(wchar_t) * (_PyUnicode_LENGTH(u) + 1 + num_surrogates));
+            if (!_PyUnicode_WSTR(u)) {
+                PyErr_NoMemory();
+                return NULL;
+            }
+            _PyUnicode_WSTR_LENGTH(u) = _PyUnicode_LENGTH(u) + num_surrogates;
+
+            w = _PyUnicode_WSTR(u);
+            wchar_end = w + _PyUnicode_WSTR_LENGTH(u);
+            four_bytes = PyUnicode_4BYTE_DATA(u);
+            for (; four_bytes < ucs4_end; ++four_bytes, ++w) {
+                if (*four_bytes > 0xFFFF) {
+                    /* encode surrogate pair in this case */
+                    *w++ = 0xD800 | ((*four_bytes - 0x10000) >> 10);
+                    *w   = 0xDC00 | ((*four_bytes - 0x10000) & 0x3FF);
+                }
+                else
+                    *w = *four_bytes;
+
+                if (w > wchar_end) {
+                    assert(0 && "Miscalculated string end");
+                }
+            }
+            *w = 0;
+#else
+            /* sizeof(wchar_t) == 4 */
+            Py_FatalError("Impossible unicode object state, wstr and str "
+                          "should share memory already.");
+            return NULL;
+#endif
+        }
+        else {
+            _PyUnicode_WSTR(u) = (wchar_t *) PyObject_MALLOC(sizeof(wchar_t) *
+                                                  (_PyUnicode_LENGTH(u) + 1));
+            if (!_PyUnicode_WSTR(u)) {
+                PyErr_NoMemory();
+                return NULL;
+            }
+            if (!PyUnicode_IS_COMPACT_ASCII(u))
+                _PyUnicode_WSTR_LENGTH(u) = _PyUnicode_LENGTH(u);
+            w = _PyUnicode_WSTR(u);
+            wchar_end = w + _PyUnicode_LENGTH(u);
+
+            if (PyUnicode_KIND(u) == PyUnicode_1BYTE_KIND) {
+                one_byte = PyUnicode_1BYTE_DATA(u);
+                for (; w < wchar_end; ++one_byte, ++w)
+                    *w = *one_byte;
+                /* null-terminate the wstr */
+                *w = 0;
+            }
+            else if (PyUnicode_KIND(u) == PyUnicode_2BYTE_KIND) {
+#if SIZEOF_WCHAR_T == 4
+                two_bytes = PyUnicode_2BYTE_DATA(u);
+                for (; w < wchar_end; ++two_bytes, ++w)
+                    *w = *two_bytes;
+                /* null-terminate the wstr */
+                *w = 0;
+#else
+                /* sizeof(wchar_t) == 2 */
+                PyObject_FREE(_PyUnicode_WSTR(u));
+                _PyUnicode_WSTR(u) = NULL;
+                Py_FatalError("Impossible unicode object state, wstr "
+                              "and str should share memory already.");
+                return NULL;
+#endif
+            }
+            else {
+                assert(0 && "This should never happen.");
+            }
+        }
+    }
+    if (size != NULL)
+        *size = PyUnicode_WSTR_LENGTH(u);
+    return _PyUnicode_WSTR(u);
 }
 
 Py_UNICODE *
 PyUnicode_AsUnicode(PyObject *unicode)
 {
+    return PyUnicode_AsUnicodeAndSize(unicode, NULL);
+}
+
+
+Py_ssize_t
+PyUnicode_GetSize(PyObject *unicode)
+{
     if (!PyUnicode_Check(unicode)) {
         PyErr_BadArgument();
         goto onError;
     }
-    return PyUnicode_AS_UNICODE(unicode);
-
-  onError:
-    return NULL;
-}
-
-Py_ssize_t
-PyUnicode_GetSize(PyObject *unicode)
-{
-    if (!PyUnicode_Check(unicode)) {
-        PyErr_BadArgument();
-        goto onError;
-    }
     return PyUnicode_GET_SIZE(unicode);
 
   onError:
     return -1;
 }
 
+Py_ssize_t
+PyUnicode_GetLength(PyObject *unicode)
+{
+    if (!PyUnicode_Check(unicode) || PyUnicode_READY(unicode) != -1) {
+        PyErr_BadArgument();
+        return -1;
+    }
+
+    return PyUnicode_GET_LENGTH(unicode);
+}
+
+Py_UCS4
+PyUnicode_ReadChar(PyObject *unicode, Py_ssize_t index)
+{
+    if (!PyUnicode_Check(unicode) || PyUnicode_READY(unicode) != -1) {
+        return PyErr_BadArgument();
+        return (Py_UCS4)-1;
+    }
+    return PyUnicode_READ_CHAR(unicode, index);
+}
+
+int
+PyUnicode_WriteChar(PyObject *unicode, Py_ssize_t index, Py_UCS4 ch)
+{
+    if (!PyUnicode_Check(unicode) || !PyUnicode_IS_COMPACT(unicode)) {
+        return PyErr_BadArgument();
+        return -1;
+    }
+
+    PyUnicode_WRITE(PyUnicode_KIND(unicode), PyUnicode_DATA(unicode),
+                    index, ch);
+    return 0;
+}
+
 const char *
 PyUnicode_GetDefaultEncoding(void)
 {
@@ -2075,7 +2836,7 @@
     Py_ssize_t insize;
     Py_ssize_t requiredsize;
     Py_ssize_t newpos;
-    Py_UNICODE *repptr;
+    const Py_UNICODE *repptr;
     PyObject *inputobj = NULL;
     Py_ssize_t repsize;
     int res = -1;
@@ -2281,7 +3042,7 @@
         return (PyObject *)unicode;
     }
 
-    p = unicode->str;
+    p = PyUnicode_AS_UNICODE(unicode);
     shiftOutStart = p;
     e = s + size;
 
@@ -2431,6 +3192,10 @@
 
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
+    if (PyUnicode_READY(unicode) == -1) {
+        Py_DECREF(unicode);
+        return NULL;
+    }
     return (PyObject *)unicode;
 
   onError:
@@ -2588,38 +3353,197 @@
 # error C 'long' size should be either 4 or 8!
 #endif
 
+/* Scans a UTF-8 string and returns the maximum character to be expected,
+   the size of the decoded unicode string and if any major errors were
+   encountered.
+
+   This function does check basic UTF-8 sanity, it does however NOT CHECK
+   if the string contains surrogates, and if all continuation bytes are
+   within the correct ranges, these checks are performed in
+   PyUnicode_DecodeUTF8Stateful.
+
+   If it sets has_errors to 1, it means the value of unicode_size and max_char
+   will be bogus and you should not rely on useful information in them.
+   */
+static Py_UCS4
+utf8_max_char_size_and_has_errors(const char *s, Py_ssize_t string_size,
+                                  Py_ssize_t *unicode_size, Py_ssize_t* consumed,
+                                  int *has_errors)
+{
+    Py_ssize_t n;
+    Py_ssize_t char_count = 0;
+    Py_UCS4 max_char = 127, new_max;
+    Py_UCS4 upper_bound;
+    const unsigned char *p = (const unsigned char *)s;
+    const unsigned char *end = p + string_size;
+    const unsigned char *aligned_end = (const unsigned char *) ((size_t) end & ~LONG_PTR_MASK);
+    int err = 0;
+
+    for (; p < end && !err; ++p, ++char_count) {
+        /* Only check value if it's not a ASCII char... */
+        if (*p < 0x80) {
+            /* Fast path, see below in PyUnicode_DecodeUTF8Stateful for
+               an explanation. */
+            if (!((size_t) p & LONG_PTR_MASK)) {
+                /* Help register allocation */
+                register const unsigned char *_p = p;
+                while (_p < aligned_end) {
+                    unsigned long value = *(unsigned long *) _p;
+                    if (value & ASCII_CHAR_MASK)
+                        break;
+                    _p += SIZEOF_LONG;
+                    char_count += SIZEOF_LONG;
+                }
+                p = _p;
+                if (p == end)
+                    break;
+            }
+        }
+        if (*p >= 0x80) {
+            n = utf8_code_length[*p];
+            new_max = max_char;
+            switch (n) {
+            /* invalid start byte */
+            case 0:
+                err = 1;
+                break;
+            case 2:
+                /* Code points between 0x00FF and 0x07FF inclusive.
+                   Approximate the upper bound of the code point,
+                   if this flips over 255 we can be sure it will be more
+                   than 255 and the string will need 2 bytes per code coint,
+                   if it stays under or equal to 255, we can be sure 1 byte
+                   is enough.
+                   ((*p & 0b00011111) << 6) | 0b00111111 */
+                upper_bound = ((*p & 0x1F) << 6) | 0x3F;
+                if (max_char < upper_bound)
+                    new_max = upper_bound;
+                /* Ensure we track at least that we left ASCII space. */
+                if (new_max < 128)
+                    new_max = 128;
+                break;
+            case 3:
+                /* Between 0x0FFF and 0xFFFF inclusive, so values are
+                   always > 255 and <= 65535 and will always need 2 bytes. */
+                if (max_char < 65535)
+                    new_max = 65535;
+                break;
+            case 4:
+                /* Code point will be above 0xFFFF for sure in this case. */
+                new_max = 65537;
+                break;
+            /* Internal error, this should be caught by the first if */
+            case 1:
+            default:
+                assert(0 && "Impossible case in utf8_max_char_and_size");
+                err = 1;
+            }
+            /* Instead of number of overall bytes for this code point,
+               n containts the number of following bytes: */
+            --n;
+            /* Check if the follow up chars are all valid continuation bytes */
+            if (n >= 1) {
+                const unsigned char *cont;
+                if ((p + n) >= end) {
+                    if (consumed == 0)
+                        /* incomplete data, non-incremental decoding */
+                        err = 1;
+                    break;
+                }
+                for (cont = p + 1; cont < (p + n); ++cont) {
+                    if ((*cont & 0xc0) != 0x80) {
+                        err = 1;
+                        break;
+                    }
+                }
+                p += n;
+            }
+            else
+                err = 1;
+            max_char = new_max;
+        }
+    }
+
+    if (unicode_size)
+        *unicode_size = char_count;
+    if (has_errors)
+        *has_errors = err;
+    return max_char;
+}
+
+/* Similar to PyUnicode_WRITE but can also write into wstr field
+   of the legacy unicode representation */
+#define WRITE_FLEXIBLE_OR_WSTR(kind, buf, index, value) \
+    do { \
+        const int k_ = (kind); \
+        if (k_ == PyUnicode_WCHAR_KIND) \
+            ((Py_UNICODE *)(buf))[(index)] = (Py_UNICODE)(value); \
+        else if (k_ == PyUnicode_1BYTE_KIND) \
+            ((unsigned char *)(buf))[(index)] = (unsigned char)(value); \
+        else if (k_ == PyUnicode_2BYTE_KIND) \
+            ((Py_UCS2 *)(buf))[(index)] = (Py_UCS2)(value); \
+        else \
+            ((Py_UCS4 *)(buf))[(index)] = (Py_UCS4)(value); \
+    } while (0)
+
 PyObject *
 PyUnicode_DecodeUTF8Stateful(const char *s,
-			     Py_ssize_t size,
-			     const char *errors,
-			     Py_ssize_t *consumed)
+                             Py_ssize_t size,
+                             const char *errors,
+                             Py_ssize_t *consumed)
 {
     const char *starts = s;
     int n;
     int k;
     Py_ssize_t startinpos;
     Py_ssize_t endinpos;
-    Py_ssize_t outpos;
     const char *e, *aligned_end;
     PyUnicodeObject *unicode;
-    Py_UNICODE *p;
     const char *errmsg = "";
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
-
-    /* Note: size will always be longer than the resulting Unicode
-       character count */
-    unicode = _PyUnicode_New(size);
-    if (!unicode)
-        return NULL;
+    Py_UCS4 maxchar = 0;
+    Py_ssize_t unicode_size;
+    Py_ssize_t i;
+    int kind;
+    void *data;
+    int has_errors;
+    Py_UNICODE *error_outptr;
+#if SIZEOF_WCHAR_T == 2
+    Py_ssize_t wchar_offset = 0;
+#endif
+
     if (size == 0) {
         if (consumed)
             *consumed = 0;
-        return (PyObject *)unicode;
-    }
-
+        return (PyObject *)PyUnicode_New(0, 0);
+    }
+    maxchar = utf8_max_char_size_and_has_errors(s, size, &unicode_size,
+                                                consumed, &has_errors);
+    if (has_errors) {
+        unicode = _PyUnicode_New(size);
+        if (!unicode)
+            return NULL;
+        kind = PyUnicode_WCHAR_KIND;
+        data = PyUnicode_AS_UNICODE(unicode);
+        assert(data != NULL);
+    }
+    else {
+        unicode = (PyUnicodeObject *)PyUnicode_New(unicode_size, maxchar);
+        if (!unicode)
+            return NULL;
+        /* When the string is ASCII only, just use memcpy and return.
+           unicode_size may be != size if there is an incomplete UTF-8
+           sequence at the end of the ASCII block.  */
+        if (maxchar < 128 && size == unicode_size) {
+            Py_MEMCPY(PyUnicode_1BYTE_DATA(unicode), s, unicode_size);
+            return (PyObject *)unicode;
+        }
+        kind = PyUnicode_KIND(unicode);
+        data = PyUnicode_DATA(unicode);
+    }
     /* Unpack UTF-8 encoded data */
-    p = unicode->str;
+    i = 0;
     e = s + size;
     aligned_end = (const char *) ((size_t) e & ~LONG_PTR_MASK);
 
@@ -2637,29 +3561,29 @@
             if (!((size_t) s & LONG_PTR_MASK)) {
                 /* Help register allocation */
                 register const char *_s = s;
-                register Py_UNICODE *_p = p;
+                register Py_ssize_t _i = i;
                 while (_s < aligned_end) {
                     /* Read a whole long at a time (either 4 or 8 bytes),
                        and do a fast unrolled copy if it only contains ASCII
                        characters. */
-                    unsigned long data = *(unsigned long *) _s;
-                    if (data & ASCII_CHAR_MASK)
+                    unsigned long value = *(unsigned long *) _s;
+                    if (value & ASCII_CHAR_MASK)
                         break;
-                    _p[0] = (unsigned char) _s[0];
-                    _p[1] = (unsigned char) _s[1];
-                    _p[2] = (unsigned char) _s[2];
-                    _p[3] = (unsigned char) _s[3];
+                    WRITE_FLEXIBLE_OR_WSTR(kind, data, _i+0, _s[0]);
+                    WRITE_FLEXIBLE_OR_WSTR(kind, data, _i+1, _s[1]);
+                    WRITE_FLEXIBLE_OR_WSTR(kind, data, _i+2, _s[2]);
+                    WRITE_FLEXIBLE_OR_WSTR(kind, data, _i+3, _s[3]);
 #if (SIZEOF_LONG == 8)
-                    _p[4] = (unsigned char) _s[4];
-                    _p[5] = (unsigned char) _s[5];
-                    _p[6] = (unsigned char) _s[6];
-                    _p[7] = (unsigned char) _s[7];
+                    WRITE_FLEXIBLE_OR_WSTR(kind, data, _i+4, _s[4]);
+                    WRITE_FLEXIBLE_OR_WSTR(kind, data, _i+5, _s[5]);
+                    WRITE_FLEXIBLE_OR_WSTR(kind, data, _i+6, _s[6]);
+                    WRITE_FLEXIBLE_OR_WSTR(kind, data, _i+7, _s[7]);
 #endif
                     _s += SIZEOF_LONG;
-                    _p += SIZEOF_LONG;
+                    _i += SIZEOF_LONG;
                 }
                 s = _s;
-                p = _p;
+                i = _i;
                 if (s == e)
                     break;
                 ch = (unsigned char)*s;
@@ -2667,7 +3591,7 @@
         }
 
         if (ch < 0x80) {
-            *p++ = (Py_UNICODE)ch;
+            WRITE_FLEXIBLE_OR_WSTR(kind, data, i++, ch);
             s++;
             continue;
         }
@@ -2710,7 +3634,7 @@
             }
             ch = ((s[0] & 0x1f) << 6) + (s[1] & 0x3f);
             assert ((ch > 0x007F) && (ch <= 0x07FF));
-            *p++ = (Py_UNICODE)ch;
+            WRITE_FLEXIBLE_OR_WSTR(kind, data, i++, ch);
             break;
 
         case 3:
@@ -2739,7 +3663,7 @@
             }
             ch = ((s[0] & 0x0f) << 12) + ((s[1] & 0x3f) << 6) + (s[2] & 0x3f);
             assert ((ch > 0x07FF) && (ch <= 0xFFFF));
-            *p++ = (Py_UNICODE)ch;
+            WRITE_FLEXIBLE_OR_WSTR(kind, data, i++, ch);
             break;
 
         case 4:
@@ -2764,19 +3688,27 @@
                  ((s[2] & 0x3f) << 6) + (s[3] & 0x3f);
             assert ((ch > 0xFFFF) && (ch <= 0x10ffff));
 
-#ifdef Py_UNICODE_WIDE
-            *p++ = (Py_UNICODE)ch;
-#else
-            /*  compute and append the two surrogates: */
-
-            /*  translate from 10000..10FFFF to 0..FFFF */
-            ch -= 0x10000;
-
-            /*  high surrogate = top 10 bits added to D800 */
-            *p++ = (Py_UNICODE)(0xD800 + (ch >> 10));
-
-            /*  low surrogate = bottom 10 bits added to DC00 */
-            *p++ = (Py_UNICODE)(0xDC00 + (ch & 0x03FF));
+            /* If the string is flexible or we have native UCS-4, write
+               directly.. */
+            if (sizeof(Py_UNICODE) > 2 || kind != PyUnicode_WCHAR_KIND)
+                WRITE_FLEXIBLE_OR_WSTR(kind, data, i++, ch);
+
+            else {
+                /* compute and append the two surrogates: */
+
+                /* translate from 10000..10FFFF to 0..FFFF */
+                ch -= 0x10000;
+
+                /* high surrogate = top 10 bits added to D800 */
+                WRITE_FLEXIBLE_OR_WSTR(kind, data, i++,
+                                       (Py_UNICODE)(0xD800 + (ch >> 10)));
+
+                /* low surrogate = bottom 10 bits added to DC00 */
+                WRITE_FLEXIBLE_OR_WSTR(kind, data, i++,
+                                       (Py_UNICODE)(0xDC00 + (ch & 0x03FF)));
+            }
+#if SIZEOF_WCHAR_T == 2
+            wchar_offset++;
 #endif
             break;
         }
@@ -2784,24 +3716,57 @@
         continue;
 
       utf8Error:
-        outpos = p-PyUnicode_AS_UNICODE(unicode);
+        /* If this is not yet a resizable string, make it one.. */
+        if (kind != PyUnicode_WCHAR_KIND) {
+            const Py_UNICODE *u;
+            PyUnicodeObject *new_unicode = _PyUnicode_New(size);
+            if (!new_unicode)
+                goto onError;
+            u = PyUnicode_AsUnicode((PyObject *)unicode);
+            if (!u)
+                goto onError;
+#if SIZEOF_WCHAR_T == 2
+            i += wchar_offset;
+#endif
+            Py_UNICODE_COPY(PyUnicode_AS_UNICODE(new_unicode), u, i);
+            Py_DECREF(unicode);
+            unicode = new_unicode;
+            kind = 0;
+            data = PyUnicode_AS_UNICODE(new_unicode);
+            assert(data != NULL);
+        }
+        error_outptr = PyUnicode_AS_UNICODE(unicode) + i;
         if (unicode_decode_call_errorhandler(
                 errors, &errorHandler,
                 "utf8", errmsg,
                 &starts, &e, &startinpos, &endinpos, &exc, &s,
-                &unicode, &outpos, &p))
+                &unicode, &i, &error_outptr))
             goto onError;
+        /* Update data because unicode_decode_call_errorhandler might have
+           re-created or resized the unicode object. */
+        data = PyUnicode_AS_UNICODE(unicode);
         aligned_end = (const char *) ((size_t) e & ~LONG_PTR_MASK);
     }
+    /* Ensure the unicode_size calculation above was correct: */
+    assert(kind == PyUnicode_WCHAR_KIND || i == unicode_size);
+
     if (consumed)
         *consumed = s-starts;
 
-    /* Adjust length */
-    if (_PyUnicode_Resize(&unicode, p - unicode->str) < 0)
-        goto onError;
+    /* Adjust length and ready string when it contained errors and
+       is of the old resizable kind. */
+    if (kind == PyUnicode_WCHAR_KIND) {
+        if (_PyUnicode_Resize(&unicode, i) < 0 ||
+            PyUnicode_READY(unicode) == -1)
+            goto onError;
+    }
 
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
+    if (PyUnicode_READY(unicode) == -1) {
+        Py_DECREF(unicode);
+        return NULL;
+    }
     return (PyObject *)unicode;
 
   onError:
@@ -2811,7 +3776,7 @@
     return NULL;
 }
 
-#undef ASCII_CHAR_MASK
+#undef WRITE_FLEXIBLE_OR_WSTR
 
 #ifdef __APPLE__
 
@@ -2882,7 +3847,7 @@
             }
             ch = ((s[0] & 0x0f) << 12) + ((s[1] & 0x3f) << 6) + (s[2] & 0x3f);
             assert ((ch > 0x07FF) && (ch <= 0xFFFF));
-            *p++ = (Py_UNICODE)ch;
+            *p++ = (wchar_t)ch;
             break;
 
         case 4:
@@ -2928,15 +3893,15 @@
 
 #endif /* __APPLE__ */
 
-/* Allocation strategy:  if the string is short, convert into a stack buffer
+/* Primary internal function which creates utf8 encoded bytes objects.
+
+   Allocation strategy:  if the string is short, convert into a stack buffer
    and allocate exactly as much space needed at the end.  Else allocate the
    maximum possible needed (4 result bytes per Unicode character), and return
    the excess memory at the end.
 */
 PyObject *
-PyUnicode_EncodeUTF8(const Py_UNICODE *s,
-                     Py_ssize_t size,
-                     const char *errors)
+_PyUnicode_AsUTF8String(PyObject *obj, const char *errors)
 {
 #define MAX_SHORT_UNICHARS 300  /* largest size we'll do on the stack */
 
@@ -2948,8 +3913,30 @@
     char stackbuf[MAX_SHORT_UNICHARS * 4];
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
-
-    assert(s != NULL);
+    int kind;
+    void *data;
+    Py_ssize_t size;
+    PyUnicodeObject *unicode = (PyUnicodeObject *)obj;
+#if SIZEOF_WCHAR_T == 2
+    Py_ssize_t wchar_offset = 0;
+#endif
+
+    if (!PyUnicode_Check(unicode)) {
+        PyErr_BadArgument();
+        return NULL;
+    }
+
+    if (PyUnicode_READY(unicode) == -1)
+        return NULL;
+
+    if (_PyUnicode_UTF8(unicode))
+        return PyBytes_FromStringAndSize(_PyUnicode_UTF8(unicode),
+                                         _PyUnicode_UTF8_LENGTH(unicode));
+
+    kind = PyUnicode_KIND(unicode);
+    data = PyUnicode_DATA(unicode);
+    size = PyUnicode_GET_LENGTH(unicode);
+
     assert(size >= 0);
 
     if (size <= MAX_SHORT_UNICHARS) {
@@ -2973,7 +3960,7 @@
     }
 
     for (i = 0; i < size;) {
-        Py_UCS4 ch = s[i++];
+        Py_UCS4 ch = PyUnicode_READ(kind, data, i++);
 
         if (ch < 0x80)
             /* Encode ASCII */
@@ -2984,83 +3971,72 @@
             *p++ = (char)(0xc0 | (ch >> 6));
             *p++ = (char)(0x80 | (ch & 0x3f));
         } else if (0xD800 <= ch && ch <= 0xDFFF) {
-#ifndef Py_UNICODE_WIDE
-            /* Special case: check for high and low surrogate */
-            if (ch <= 0xDBFF && i != size && 0xDC00 <= s[i] && s[i] <= 0xDFFF) {
-                Py_UCS4 ch2 = s[i];
-                /* Combine the two surrogates to form a UCS4 value */
-                ch = ((ch - 0xD800) << 10 | (ch2 - 0xDC00)) + 0x10000;
-                i++;
-
-                /* Encode UCS4 Unicode ordinals */
-                *p++ = (char)(0xf0 | (ch >> 18));
-                *p++ = (char)(0x80 | ((ch >> 12) & 0x3f));
-                *p++ = (char)(0x80 | ((ch >> 6) & 0x3f));
-                *p++ = (char)(0x80 | (ch & 0x3f));
-            } else {
-#endif
-                Py_ssize_t newpos;
-                PyObject *rep;
-                Py_ssize_t repsize, k;
-                rep = unicode_encode_call_errorhandler
-                    (errors, &errorHandler, "utf-8", "surrogates not allowed",
-                     s, size, &exc, i-1, i, &newpos);
-                if (!rep)
+            Py_ssize_t newpos;
+            PyObject *rep;
+            Py_ssize_t repsize, k, startpos;
+            startpos = i-1;
+#if SIZEOF_WCHAR_T == 2
+            startpos += wchar_offset;
+#endif
+            rep = unicode_encode_call_errorhandler(
+                  errors, &errorHandler, "utf-8", "surrogates not allowed",
+                  PyUnicode_AS_UNICODE(unicode), PyUnicode_GET_SIZE(unicode),
+                  &exc, startpos, startpos+1, &newpos);
+            if (!rep)
+                goto error;
+
+            if (PyBytes_Check(rep))
+                repsize = PyBytes_GET_SIZE(rep);
+            else
+                repsize = PyUnicode_GET_SIZE(rep);
+
+            if (repsize > 4) {
+                Py_ssize_t offset;
+
+                if (result == NULL)
+                    offset = p - stackbuf;
+                else
+                    offset = p - PyBytes_AS_STRING(result);
+
+                if (nallocated > PY_SSIZE_T_MAX - repsize + 4) {
+                    /* integer overflow */
+                    PyErr_NoMemory();
                     goto error;
-
-                if (PyBytes_Check(rep))
-                    repsize = PyBytes_GET_SIZE(rep);
-                else
-                    repsize = PyUnicode_GET_SIZE(rep);
-
-                if (repsize > 4) {
-                    Py_ssize_t offset;
-
+                }
+                nallocated += repsize - 4;
+                if (result != NULL) {
+                    if (_PyBytes_Resize(&result, nallocated) < 0)
+                        goto error;
+                } else {
+                    result = PyBytes_FromStringAndSize(NULL, nallocated);
                     if (result == NULL)
-                        offset = p - stackbuf;
-                    else
-                        offset = p - PyBytes_AS_STRING(result);
-
-                    if (nallocated > PY_SSIZE_T_MAX - repsize + 4) {
-                        /* integer overflow */
-                        PyErr_NoMemory();
+                        goto error;
+                    Py_MEMCPY(PyBytes_AS_STRING(result), stackbuf, offset);
+                }
+                p = PyBytes_AS_STRING(result) + offset;
+            }
+
+            if (PyBytes_Check(rep)) {
+                char *prep = PyBytes_AS_STRING(rep);
+                for(k = repsize; k > 0; k--)
+                    *p++ = *prep++;
+            } else /* rep is unicode */ {
+                const Py_UNICODE *prep = PyUnicode_AS_UNICODE(rep);
+                Py_UNICODE c;
+
+                for(k=0; k<repsize; k++) {
+                    c = prep[k];
+                    if (0x80 <= c) {
+                        raise_encode_exception(&exc, "utf-8",
+                                               PyUnicode_AS_UNICODE(unicode),
+                                               size, i-1, i,
+                                               "surrogates not allowed");
                         goto error;
                     }
-                    nallocated += repsize - 4;
-                    if (result != NULL) {
-                        if (_PyBytes_Resize(&result, nallocated) < 0)
-                            goto error;
-                    } else {
-                        result = PyBytes_FromStringAndSize(NULL, nallocated);
-                        if (result == NULL)
-                            goto error;
-                        Py_MEMCPY(PyBytes_AS_STRING(result), stackbuf, offset);
-                    }
-                    p = PyBytes_AS_STRING(result) + offset;
+                    *p++ = (char)prep[k];
                 }
-
-                if (PyBytes_Check(rep)) {
-                    char *prep = PyBytes_AS_STRING(rep);
-                    for(k = repsize; k > 0; k--)
-                        *p++ = *prep++;
-                } else /* rep is unicode */ {
-                    Py_UNICODE *prep = PyUnicode_AS_UNICODE(rep);
-                    Py_UNICODE c;
-
-                    for(k=0; k<repsize; k++) {
-                        c = prep[k];
-                        if (0x80 <= c) {
-                            raise_encode_exception(&exc, "utf-8", s, size,
-                                                   i-1, i, "surrogates not allowed");
-                            goto error;
-                        }
-                        *p++ = (char)prep[k];
-                    }
-                }
-                Py_DECREF(rep);
-#ifndef Py_UNICODE_WIDE
-            }
-#endif
+            }
+            Py_DECREF(rep);
         } else if (ch < 0x10000) {
             *p++ = (char)(0xe0 | (ch >> 12));
             *p++ = (char)(0x80 | ((ch >> 6) & 0x3f));
@@ -3071,6 +4047,9 @@
             *p++ = (char)(0x80 | ((ch >> 12) & 0x3f));
             *p++ = (char)(0x80 | ((ch >> 6) & 0x3f));
             *p++ = (char)(0x80 | (ch & 0x3f));
+#if SIZEOF_WCHAR_T == 2
+            wchar_offset++;
+#endif
         }
     }
 
@@ -3086,6 +4065,7 @@
         assert(nneeded <= nallocated);
         _PyBytes_Resize(&result, nneeded);
     }
+
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
     return result;
@@ -3099,18 +4079,24 @@
 }
 
 PyObject *
+PyUnicode_EncodeUTF8(const Py_UNICODE *s,
+                     Py_ssize_t size,
+                     const char *errors)
+{
+    PyObject *v, *unicode;
+
+    unicode = PyUnicode_FromUnicode(s, size);
+    if (unicode == NULL)
+        return NULL;
+    v = _PyUnicode_AsUTF8String(unicode, errors);
+    Py_DECREF(unicode);
+    return v;
+}
+
+PyObject *
 PyUnicode_AsUTF8String(PyObject *unicode)
 {
-    PyObject *utf8;
-    if (!PyUnicode_Check(unicode)) {
-        PyErr_BadArgument();
-        return NULL;
-    }
-    utf8 = _PyUnicode_AsDefaultEncodedString(unicode);
-    if (utf8 == NULL)
-        return NULL;
-    Py_INCREF(utf8);
-    return utf8;
+    return _PyUnicode_AsUTF8String(unicode, NULL);
 }
 
 /* --- UTF-32 Codec ------------------------------------------------------- */
@@ -3222,7 +4208,7 @@
         return (PyObject *)unicode;
 
     /* Unpack UTF-32 encoded data */
-    p = unicode->str;
+    p = PyUnicode_AS_UNICODE(unicode);
 
     while (q < e) {
         Py_UCS4 ch;
@@ -3275,11 +4261,15 @@
         *consumed = (const char *)q-starts;
 
     /* Adjust length */
-    if (_PyUnicode_Resize(&unicode, p - unicode->str) < 0)
+    if (_PyUnicode_Resize(&unicode, p - PyUnicode_AS_UNICODE(unicode)) < 0)
         goto onError;
 
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
+    if (PyUnicode_READY(unicode) == -1) {
+        Py_DECREF(unicode);
+        return NULL;
+    }
     return (PyObject *)unicode;
 
   onError:
@@ -3452,7 +4442,7 @@
         return (PyObject *)unicode;
 
     /* Unpack UTF-16 encoded data */
-    p = unicode->str;
+    p = PyUnicode_AS_UNICODE(unicode);
     q = (unsigned char *)s;
     e = q + size - 1;
 
@@ -3669,11 +4659,15 @@
         *consumed = (const char *)q-starts;
 
     /* Adjust length */
-    if (_PyUnicode_Resize(&unicode, p - unicode->str) < 0)
+    if (_PyUnicode_Resize(&unicode, p - PyUnicode_AS_UNICODE(unicode)) < 0)
         goto onError;
 
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
+    if (PyUnicode_READY(unicode) == -1) {
+        Py_DECREF(unicode);
+        return NULL;
+    }
     return (PyObject *)unicode;
 
   onError:
@@ -3782,6 +4776,76 @@
 
 /* --- Unicode Escape Codec ----------------------------------------------- */
 
+/* Helper function for PyUnicode_DecodeUnicodeEscape, determines
+   if all the escapes in the string make it still a valid ASCII string.
+   Returns -1 if any escapes were found which cause the string to
+   pop out of ASCII range.  Otherwise returns the length of the
+   required buffer to hold the string.
+   */
+Py_ssize_t
+length_of_escaped_ascii_string(const char *s, Py_ssize_t size)
+{
+    const unsigned char *p = (const unsigned char *)s;
+    const unsigned char *end = p + size;
+    Py_ssize_t length = 0;
+
+    if (size < 0)
+        return -1;
+
+    for (; p < end; ++p) {
+        if (*p > 127) {
+            /* Non-ASCII */
+            return -1;
+        }
+        else if (*p != '\\') {
+            /* Normal character */
+            ++length;
+        }
+        else {
+            /* Backslash-escape, check next char */
+            ++p;
+            /* Escape sequence reaches till end of string or
+               non-ASCII follow-up. */
+            if (p >= end || *p > 127)
+                return -1;
+            switch (*p) {
+            case '\n':
+                /* backslash + \n result in zero characters */
+                break;
+            case '\\': case '\'': case '\"':
+            case 'b': case 'f': case 't':
+            case 'n': case 'r': case 'v': case 'a':
+                ++length;
+                break;
+            case '0': case '1': case '2': case '3':
+            case '4': case '5': case '6': case '7':
+            case 'x': case 'u': case 'U': case 'N':
+                /* these do not guarantee ASCII characters */
+                return -1;
+            default:
+                /* count the backslash + the other character */
+                length += 2;
+            }
+        }
+    }
+    return length;
+}
+
+/* Similar to PyUnicode_WRITE but either write into wstr field
+   or treat string as ASCII. */
+#define WRITE_ASCII_OR_WSTR(kind, buf, index, value) \
+    do { \
+        if ((kind) != PyUnicode_WCHAR_KIND) \
+            ((unsigned char *)(buf))[(index)] = (unsigned char)(value); \
+        else \
+            ((Py_UNICODE *)(buf))[(index)] = (Py_UNICODE)(value); \
+    } while (0)
+
+#define WRITE_WSTR(buf, index, value) \
+    assert(kind == PyUnicode_WCHAR_KIND), \
+    ((Py_UNICODE *)(buf))[(index)] = (Py_UNICODE)(value)
+
+
 static _PyUnicode_Name_CAPI *ucnhash_CAPI = NULL;
 
 PyObject *
@@ -3792,8 +4856,7 @@
     const char *starts = s;
     Py_ssize_t startinpos;
     Py_ssize_t endinpos;
-    Py_ssize_t outpos;
-    int i;
+    int j;
     PyUnicodeObject *v;
     Py_UNICODE *p;
     const char *end;
@@ -3801,19 +4864,42 @@
     Py_UCS4 chr = 0xffffffff; /* in case 'getcode' messes up */
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
-
-    /* Escaped strings will always be longer than the resulting
-       Unicode string, so we start with size here and then reduce the
-       length after conversion to the true value.
-       (but if the error callback returns a long replacement string
-       we'll have to allocate more space) */
-    v = _PyUnicode_New(size);
-    if (v == NULL)
-        goto onError;
+    Py_ssize_t ascii_length;
+    Py_ssize_t i;
+    int kind;
+    void *data;
+
+    ascii_length = length_of_escaped_ascii_string(s, size);
+
+    /* After length_of_escaped_ascii_string() there are two alternatives,
+       either the string is pure ASCII with named escapes like \n, etc.
+       and we determined it's exact size (common case)
+       or it contains \x, \u, ... escape sequences.  then we create a
+       legacy wchar string and resize it at the end of this function. */
+    if (ascii_length >= 0) {
+        v = (PyUnicodeObject *)PyUnicode_New(ascii_length, 127);
+        if (!v)
+            goto onError;
+        assert(PyUnicode_KIND(v) == PyUnicode_1BYTE_KIND);
+        kind = PyUnicode_1BYTE_KIND;
+        data = PyUnicode_DATA(v);
+    }
+    else {
+        /* Escaped strings will always be longer than the resulting
+           Unicode string, so we start with size here and then reduce the
+           length after conversion to the true value.
+           (but if the error callback returns a long replacement string
+           we'll have to allocate more space) */
+        v = _PyUnicode_New(size);
+        if (!v)
+            goto onError;
+        kind = PyUnicode_WCHAR_KIND;
+        data = PyUnicode_AS_UNICODE(v);
+    }
+
     if (size == 0)
         return (PyObject *)v;
-
-    p = PyUnicode_AS_UNICODE(v);
+    i = 0;
     end = s + size;
 
     while (s < end) {
@@ -3821,9 +4907,18 @@
         Py_UNICODE x;
         int digits;
 
+        if (kind == PyUnicode_WCHAR_KIND) {
+            assert(i < _PyUnicode_WSTR_LENGTH(v));
+        }
+        else {
+            /* The only case in which i == ascii_length is a backslash
+               followed by a newline. */
+            assert(i <= ascii_length);
+        }
+
         /* Non-escape characters are interpreted as Unicode ordinals */
         if (*s != '\\') {
-            *p++ = (unsigned char) *s++;
+            WRITE_ASCII_OR_WSTR(kind, data, i++, (unsigned char) *s++);
             continue;
         }
 
@@ -3833,20 +4928,33 @@
         c = *s++;
         if (s > end)
             c = '\0'; /* Invalid after \ */
+
+        if (kind == PyUnicode_WCHAR_KIND) {
+            assert(i < _PyUnicode_WSTR_LENGTH(v));
+        }
+        else {
+            /* The only case in which i == ascii_length is a backslash
+               followed by a newline. */
+            assert(i < ascii_length || (i == ascii_length && c == '\n'));
+        }
+
         switch (c) {
 
             /* \x escapes */
         case '\n': break;
-        case '\\': *p++ = '\\'; break;
-        case '\'': *p++ = '\''; break;
-        case '\"': *p++ = '\"'; break;
-        case 'b': *p++ = '\b'; break;
-        case 'f': *p++ = '\014'; break; /* FF */
-        case 't': *p++ = '\t'; break;
-        case 'n': *p++ = '\n'; break;
-        case 'r': *p++ = '\r'; break;
-        case 'v': *p++ = '\013'; break; /* VT */
-        case 'a': *p++ = '\007'; break; /* BEL, not classic C */
+        case '\\': WRITE_ASCII_OR_WSTR(kind, data, i++, '\\'); break;
+        case '\'': WRITE_ASCII_OR_WSTR(kind, data, i++, '\''); break;
+        case '\"': WRITE_ASCII_OR_WSTR(kind, data, i++, '\"'); break;
+        case 'b': WRITE_ASCII_OR_WSTR(kind, data, i++, '\b'); break;
+        /* FF */
+        case 'f': WRITE_ASCII_OR_WSTR(kind, data, i++, '\014'); break;
+        case 't': WRITE_ASCII_OR_WSTR(kind, data, i++, '\t'); break;
+        case 'n': WRITE_ASCII_OR_WSTR(kind, data, i++, '\n'); break;
+        case 'r': WRITE_ASCII_OR_WSTR(kind, data, i++, '\r'); break;
+        /* VT */
+        case 'v': WRITE_ASCII_OR_WSTR(kind, data, i++, '\013'); break;
+        /* BEL, not classic C */
+        case 'a': WRITE_ASCII_OR_WSTR(kind, data, i++, '\007'); break;
 
             /* \OOO (octal) escapes */
         case '0': case '1': case '2': case '3':
@@ -3857,7 +4965,7 @@
                 if (s < end && '0' <= *s && *s <= '7')
                     x = (x<<3) + *s++ - '0';
             }
-            *p++ = x;
+            WRITE_WSTR(data, i++, x);
             break;
 
             /* hex escapes */
@@ -3879,27 +4987,30 @@
             message = "truncated \\UXXXXXXXX escape";
         hexescape:
             chr = 0;
-            outpos = p-PyUnicode_AS_UNICODE(v);
+            p = PyUnicode_AS_UNICODE(v) + i;
             if (s+digits>end) {
                 endinpos = size;
                 if (unicode_decode_call_errorhandler(
                         errors, &errorHandler,
                         "unicodeescape", "end of string in escape sequence",
                         &starts, &end, &startinpos, &endinpos, &exc, &s,
-                        &v, &outpos, &p))
+                        &v, &i, &p))
                     goto onError;
+                data = PyUnicode_AS_UNICODE(v);
                 goto nextByte;
             }
-            for (i = 0; i < digits; ++i) {
-                c = (unsigned char) s[i];
+            for (j = 0; j < digits; ++j) {
+                c = (unsigned char) s[j];
                 if (!Py_ISXDIGIT(c)) {
-                    endinpos = (s+i+1)-starts;
+                    endinpos = (s+j+1)-starts;
+                    p = PyUnicode_AS_UNICODE(v) + i;
                     if (unicode_decode_call_errorhandler(
                             errors, &errorHandler,
                             "unicodeescape", message,
                             &starts, &end, &startinpos, &endinpos, &exc, &s,
-                            &v, &outpos, &p))
+                            &v, &i, &p))
                         goto onError;
+                    data = PyUnicode_AS_UNICODE(v);
                     goto nextByte;
                 }
                 chr = (chr<<4) & ~0xF;
@@ -3910,7 +5021,7 @@
                 else
                     chr += 10 + c - 'A';
             }
-            s += i;
+            s += j;
             if (chr == 0xffffffff && PyErr_Occurred())
                 /* _decoding_error will have already written into the
                    target buffer. */
@@ -3919,26 +5030,27 @@
             /* when we get here, chr is a 32-bit unicode character */
             if (chr <= 0xffff)
                 /* UCS-2 character */
-                *p++ = (Py_UNICODE) chr;
+                WRITE_WSTR(data, i++, chr);
             else if (chr <= 0x10ffff) {
                 /* UCS-4 character. Either store directly, or as
                    surrogate pair. */
 #ifdef Py_UNICODE_WIDE
-                *p++ = chr;
+                WRITE_WSTR(data, i++, chr);
 #else
                 chr -= 0x10000L;
-                *p++ = 0xD800 + (Py_UNICODE) (chr >> 10);
-                *p++ = 0xDC00 + (Py_UNICODE) (chr & 0x03FF);
+                WRITE_WSTR(data, i++, 0xD800 + (Py_UNICODE) (chr >> 10));
+                WRITE_WSTR(data, i++, 0xDC00 + (Py_UNICODE) (chr & 0x03FF));
 #endif
             } else {
                 endinpos = s-starts;
-                outpos = p-PyUnicode_AS_UNICODE(v);
+                p = PyUnicode_AS_UNICODE(v) + i;
                 if (unicode_decode_call_errorhandler(
                         errors, &errorHandler,
                         "unicodeescape", "illegal Unicode character",
                         &starts, &end, &startinpos, &endinpos, &exc, &s,
-                        &v, &outpos, &p))
+                        &v, &i, &p))
                     goto onError;
+                data = PyUnicode_AS_UNICODE(v);
             }
             break;
 
@@ -3947,7 +5059,8 @@
             message = "malformed \\N character escape";
             if (ucnhash_CAPI == NULL) {
                 /* load the unicode data module */
-                ucnhash_CAPI = (_PyUnicode_Name_CAPI *)PyCapsule_Import(PyUnicodeData_CAPSULE_NAME, 1);
+                ucnhash_CAPI = (_PyUnicode_Name_CAPI *)PyCapsule_Import(
+                                                PyUnicodeData_CAPSULE_NAME, 1);
                 if (ucnhash_CAPI == NULL)
                     goto ucnhashError;
             }
@@ -3960,43 +5073,51 @@
                     /* found a name.  look it up in the unicode database */
                     message = "unknown Unicode character name";
                     s++;
-                    if (ucnhash_CAPI->getcode(NULL, start, (int)(s-start-1), &chr))
+                    if (ucnhash_CAPI->getcode(NULL, start, (int)(s-start-1),
+                                              &chr))
                         goto store;
                 }
             }
             endinpos = s-starts;
-            outpos = p-PyUnicode_AS_UNICODE(v);
+            p = PyUnicode_AS_UNICODE(v) + i;
             if (unicode_decode_call_errorhandler(
                     errors, &errorHandler,
                     "unicodeescape", message,
                     &starts, &end, &startinpos, &endinpos, &exc, &s,
-                    &v, &outpos, &p))
+                    &v, &i, &p))
                 goto onError;
+            data = PyUnicode_AS_UNICODE(v);
             break;
 
         default:
             if (s > end) {
+                assert(kind == PyUnicode_WCHAR_KIND);
                 message = "\\ at end of string";
                 s--;
                 endinpos = s-starts;
-                outpos = p-PyUnicode_AS_UNICODE(v);
+                p = PyUnicode_AS_UNICODE(v) + i;
                 if (unicode_decode_call_errorhandler(
                         errors, &errorHandler,
                         "unicodeescape", message,
                         &starts, &end, &startinpos, &endinpos, &exc, &s,
-                        &v, &outpos, &p))
+                        &v, &i, &p))
                     goto onError;
+                data = PyUnicode_AS_UNICODE(v);
             }
             else {
-                *p++ = '\\';
-                *p++ = (unsigned char)s[-1];
+                WRITE_ASCII_OR_WSTR(kind, data, i++, '\\');
+                WRITE_ASCII_OR_WSTR(kind, data, i++, (unsigned char)s[-1]);
             }
             break;
         }
       nextByte:
         ;
     }
-    if (_PyUnicode_Resize(&v, p - PyUnicode_AS_UNICODE(v)) < 0)
+    /* Ensure the length prediction worked in case of ASCII strings */
+    assert(kind == PyUnicode_WCHAR_KIND || i == ascii_length);
+
+    if (kind == PyUnicode_WCHAR_KIND && (_PyUnicode_Resize(&v, i) < 0 ||
+                                         PyUnicode_READY(v) == -1))
         goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
@@ -4019,6 +5140,9 @@
     return NULL;
 }
 
+#undef WRITE_ASCII_OR_WSTR
+#undef WRITE_WSTR
+
 /* Return a Unicode-Escape string version of the Unicode object.
 
    If quotes is true, the string is enclosed in u"" or u'' quotes as
@@ -4026,21 +5150,6 @@
 
 */
 
-Py_LOCAL_INLINE(const Py_UNICODE *) findchar(const Py_UNICODE *s,
-                                             Py_ssize_t size,
-                                             Py_UNICODE ch)
-{
-    /* like wcschr, but doesn't stop at NULL characters */
-
-    while (size-- > 0) {
-        if (*s == ch)
-            return s;
-        s++;
-    }
-
-    return NULL;
-}
-
 static const char *hexdigits = "0123456789abcdef";
 
 PyObject *
@@ -4309,6 +5418,10 @@
         goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
+    if (PyUnicode_READY(v) == -1) {
+        Py_DECREF(v);
+        return NULL;
+    }
     return (PyObject *)v;
 
   onError:
@@ -4447,7 +5560,9 @@
     v = _PyUnicode_New((size+Py_UNICODE_SIZE-1)/ Py_UNICODE_SIZE);
     if (v == NULL)
         goto onError;
-    if (PyUnicode_GetSize((PyObject *)v) == 0)
+    /* Intentionally PyUnicode_GET_SIZE instead of PyUnicode_GET_LENGTH
+       as string was created with the old API. */
+    if (PyUnicode_GET_SIZE(v) == 0)
         return (PyObject *)v;
     p = PyUnicode_AS_UNICODE(v);
     end = s + size;
@@ -4491,6 +5606,10 @@
         goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
+    if (PyUnicode_READY(v) == -1) {
+        Py_DECREF(v);
+        return NULL;
+    }
     return (PyObject *)v;
 
   onError:
@@ -4507,41 +5626,8 @@
 		       Py_ssize_t size,
 		       const char *errors)
 {
-    PyUnicodeObject *v;
-    Py_UNICODE *p;
-    const char *e, *unrolled_end;
-
     /* Latin-1 is equivalent to the first 256 ordinals in Unicode. */
-    if (size == 1) {
-        Py_UNICODE r = *(unsigned char*)s;
-        return PyUnicode_FromUnicode(&r, 1);
-    }
-
-    v = _PyUnicode_New(size);
-    if (v == NULL)
-        goto onError;
-    if (size == 0)
-        return (PyObject *)v;
-    p = PyUnicode_AS_UNICODE(v);
-    e = s + size;
-    /* Unrolling the copy makes it much faster by reducing the looping
-       overhead. This is similar to what many memcpy() implementations do. */
-    unrolled_end = e - 4;
-    while (s < unrolled_end) {
-        p[0] = (unsigned char) s[0];
-        p[1] = (unsigned char) s[1];
-        p[2] = (unsigned char) s[2];
-        p[3] = (unsigned char) s[3];
-        s += 4;
-        p += 4;
-    }
-    while (s < e)
-        *p++ = (unsigned char) *s++;
-    return (PyObject *)v;
-
-  onError:
-    Py_XDECREF(v);
-    return NULL;
+    return PyUnicode_FromUCS1((unsigned char*)s, size);
 }
 
 /* create or adjust a UnicodeEncodeError */
@@ -4849,15 +5935,30 @@
 }
 
 PyObject *
-PyUnicode_AsLatin1String(PyObject *unicode)
+_PyUnicode_AsLatin1String(PyObject *unicode, const char *errors)
 {
     if (!PyUnicode_Check(unicode)) {
         PyErr_BadArgument();
         return NULL;
     }
+    if (PyUnicode_READY(unicode) == -1)
+        return NULL;
+    /* Fast path: if it is a one-byte string, construct
+       bytes object directly. */
+    if (PyUnicode_KIND(unicode) == PyUnicode_1BYTE_KIND)
+        return PyBytes_FromStringAndSize(PyUnicode_DATA(unicode),
+                                         PyUnicode_GET_LENGTH(unicode));
+    /* Non-Latin-1 characters present. Defer to above function to
+       raise the exception. */
     return PyUnicode_EncodeLatin1(PyUnicode_AS_UNICODE(unicode),
                                   PyUnicode_GET_SIZE(unicode),
-                                  NULL);
+                                  errors);
+}
+
+PyObject*
+PyUnicode_AsLatin1String(PyObject *unicode)
+{
+    return _PyUnicode_AsLatin1String(unicode, NULL);
 }
 
 /* --- 7-bit ASCII Codec -------------------------------------------------- */
@@ -4874,14 +5975,32 @@
     Py_ssize_t endinpos;
     Py_ssize_t outpos;
     const char *e;
+    unsigned char* d;
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
+    Py_ssize_t i;
 
     /* ASCII is equivalent to the first 128 ordinals in Unicode. */
-    if (size == 1 && *(unsigned char*)s < 128) {
-        Py_UNICODE r = *(unsigned char*)s;
-        return PyUnicode_FromUnicode(&r, 1);
-    }
+    if (size == 1 && *(unsigned char*)s < 128)
+        return PyUnicode_FromOrdinal(*(unsigned char*)s);
+
+    /* Fast path. Assume the input actually *is* ASCII, and allocate
+       a single-block Unicode object with that assumption. If there is
+       an error, drop the object and start over. */
+    v = (PyUnicodeObject*)PyUnicode_New(size, 127);
+    if (v == NULL)
+        goto onError;
+    d = PyUnicode_1BYTE_DATA(v);
+    for (i = 0; i < size; i++) {
+        unsigned char ch = ((unsigned char*)s)[i];
+        if (ch < 128)
+            d[i] = ch;
+        else
+            break;
+    }
+    if (i == size)
+        return (PyObject*)v;
+    Py_DECREF(v); /* start over */
 
     v = _PyUnicode_New(size);
     if (v == NULL)
@@ -4913,6 +6032,10 @@
             goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
+    if (PyUnicode_READY(v) == -1) {
+        Py_DECREF(v);
+        return NULL;
+    }
     return (PyObject *)v;
 
   onError:
@@ -4931,15 +6054,28 @@
 }
 
 PyObject *
-PyUnicode_AsASCIIString(PyObject *unicode)
+_PyUnicode_AsASCIIString(PyObject *unicode, const char *errors)
 {
     if (!PyUnicode_Check(unicode)) {
         PyErr_BadArgument();
         return NULL;
     }
+    if (PyUnicode_READY(unicode) == -1)
+        return NULL;
+    /* Fast path: if it is an ASCII-only string, construct bytes object
+       directly. Else defer to above function to raise the exception. */
+    if (PyUnicode_MAX_CHAR_VALUE(unicode) < 128)
+        return PyBytes_FromStringAndSize(PyUnicode_DATA(unicode),
+                                         PyUnicode_GET_LENGTH(unicode));
     return PyUnicode_EncodeASCII(PyUnicode_AS_UNICODE(unicode),
                                  PyUnicode_GET_SIZE(unicode),
-                                 NULL);
+                                 errors);
+}
+
+PyObject *
+PyUnicode_AsASCIIString(PyObject *unicode)
+{
+    return _PyUnicode_AsASCIIString(unicode, NULL);
 }
 
 #ifdef HAVE_MBCS
@@ -5090,7 +6226,10 @@
         goto retry;
     }
 #endif
-
+    if (PyUnicode_READY(v) == -1) {
+        Py_DECREF(v);
+        return NULL;
+    }
     return (PyObject *)v;
 }
 
@@ -5386,6 +6525,10 @@
             goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
+    if (PyUnicode_READY(v) == -1) {
+        Py_DECREF(v);
+        return NULL;
+    }
     return (PyObject *)v;
 
   onError:
@@ -5471,7 +6614,6 @@
 PyObject*
 PyUnicode_BuildEncodingMap(PyObject* string)
 {
-    Py_UNICODE *decode;
     PyObject *result;
     struct encoding_map *mresult;
     int i;
@@ -5480,35 +6622,36 @@
     unsigned char level2[512];
     unsigned char *mlevel1, *mlevel2, *mlevel3;
     int count2 = 0, count3 = 0;
-
-    if (!PyUnicode_Check(string) || PyUnicode_GetSize(string) != 256) {
+    int kind;
+    void *data;
+    Py_UCS4 ch;
+
+    if (!PyUnicode_Check(string) || PyUnicode_GET_LENGTH(string) != 256) {
         PyErr_BadArgument();
         return NULL;
     }
-    decode = PyUnicode_AS_UNICODE(string);
+    kind = PyUnicode_KIND(string);
+    data = PyUnicode_DATA(string);
     memset(level1, 0xFF, sizeof level1);
     memset(level2, 0xFF, sizeof level2);
 
     /* If there isn't a one-to-one mapping of NULL to \0,
        or if there are non-BMP characters, we need to use
        a mapping dictionary. */
-    if (decode[0] != 0)
+    if (PyUnicode_READ(kind, data, 0) != 0)
         need_dict = 1;
     for (i = 1; i < 256; i++) {
         int l1, l2;
-        if (decode[i] == 0
-#ifdef Py_UNICODE_WIDE
-            || decode[i] > 0xFFFF
-#endif
-            ) {
+        ch = PyUnicode_READ(kind, data, i);
+        if (ch == 0 || ch > 0xFFFF) {
             need_dict = 1;
             break;
         }
-        if (decode[i] == 0xFFFE)
+        if (ch == 0xFFFE)
             /* unmapped character */
             continue;
-        l1 = decode[i] >> 11;
-        l2 = decode[i] >> 7;
+        l1 = ch >> 11;
+        l2 = ch >> 7;
         if (level1[l1] == 0xFF)
             level1[l1] = count2++;
         if (level2[l2] == 0xFF)
@@ -5524,7 +6667,7 @@
         if (!result)
             return NULL;
         for (i = 0; i < 256; i++) {
-            key = PyLong_FromLong(decode[i]);
+            key = PyLong_FromLong(PyUnicode_READ(kind, data, i));
             value = PyLong_FromLong(i);
             if (!key || !value)
                 goto failed1;
@@ -5559,15 +6702,15 @@
     count3 = 0;
     for (i = 1; i < 256; i++) {
         int o1, o2, o3, i2, i3;
-        if (decode[i] == 0xFFFE)
+        if (PyUnicode_READ(kind, data, i) == 0xFFFE)
             /* unmapped character */
             continue;
-        o1 = decode[i]>>11;
-        o2 = (decode[i]>>7) & 0xF;
+        o1 = PyUnicode_READ(kind, data, i)>>11;
+        o2 = (PyUnicode_READ(kind, data, i)>>7) & 0xF;
         i2 = 16*mlevel1[o1] + o2;
         if (mlevel2[i2] == 0xFF)
             mlevel2[i2] = count3++;
-        o3 = decode[i] & 0x7F;
+        o3 = PyUnicode_READ(kind, data, i) & 0x7F;
         i3 = 128*mlevel2[i2] + o3;
         mlevel3[i3] = i;
     }
@@ -5951,13 +7094,13 @@
 /* create or adjust a UnicodeTranslateError */
 static void
 make_translate_exception(PyObject **exceptionObject,
-                         const Py_UNICODE *unicode, Py_ssize_t size,
+                         PyObject *unicode,
                          Py_ssize_t startpos, Py_ssize_t endpos,
                          const char *reason)
 {
     if (*exceptionObject == NULL) {
-        *exceptionObject = PyUnicodeTranslateError_Create(
-            unicode, size, startpos, endpos, reason);
+        *exceptionObject = _PyUnicodeTranslateError_Create(
+            unicode, startpos, endpos, reason);
     }
     else {
         if (PyUnicodeTranslateError_SetStart(*exceptionObject, startpos))
@@ -5976,12 +7119,12 @@
 /* raises a UnicodeTranslateError */
 static void
 raise_translate_exception(PyObject **exceptionObject,
-                          const Py_UNICODE *unicode, Py_ssize_t size,
+                          PyObject *unicode,
                           Py_ssize_t startpos, Py_ssize_t endpos,
                           const char *reason)
 {
     make_translate_exception(exceptionObject,
-                             unicode, size, startpos, endpos, reason);
+                             unicode, startpos, endpos, reason);
     if (*exceptionObject != NULL)
         PyCodec_StrictErrors(*exceptionObject);
 }
@@ -5994,7 +7137,7 @@
 unicode_translate_call_errorhandler(const char *errors,
                                     PyObject **errorHandler,
                                     const char *reason,
-                                    const Py_UNICODE *unicode, Py_ssize_t size, PyObject **exceptionObject,
+                                    PyObject *unicode, PyObject **exceptionObject,
                                     Py_ssize_t startpos, Py_ssize_t endpos,
                                     Py_ssize_t *newpos)
 {
@@ -6011,7 +7154,7 @@
     }
 
     make_translate_exception(exceptionObject,
-                             unicode, size, startpos, endpos, reason);
+                             unicode, startpos, endpos, reason);
     if (*exceptionObject == NULL)
         return NULL;
 
@@ -6030,10 +7173,10 @@
         return NULL;
     }
     if (i_newpos<0)
-        *newpos = size+i_newpos;
+        *newpos = PyUnicode_GET_LENGTH(unicode)+i_newpos;
     else
         *newpos = i_newpos;
-    if (*newpos<0 || *newpos>size) {
+    if (*newpos<0 || *newpos>PyUnicode_GET_LENGTH(unicode)) {
         PyErr_Format(PyExc_IndexError, "position %zd from error handler out of bounds", *newpos);
         Py_DECREF(restuple);
         return NULL;
@@ -6047,7 +7190,7 @@
    which must be decrefed by the caller.
    Return 0 on success, -1 on error */
 static int
-charmaptranslate_lookup(Py_UNICODE c, PyObject *mapping, PyObject **result)
+charmaptranslate_lookup(Py_UCS4 c, PyObject *mapping, PyObject **result)
 {
     PyObject *w = PyLong_FromLong((long)c);
     PyObject *x;
@@ -6097,19 +7240,18 @@
    if not reallocate and adjust various state variables.
    Return 0 on success, -1 on error */
 static int
-charmaptranslate_makespace(PyObject **outobj, Py_UNICODE **outp,
+charmaptranslate_makespace(Py_UCS4 **outobj, Py_ssize_t *psize,
                                Py_ssize_t requiredsize)
 {
-    Py_ssize_t oldsize = PyUnicode_GET_SIZE(*outobj);
+    Py_ssize_t oldsize = *psize;
     if (requiredsize > oldsize) {
-        /* remember old output position */
-        Py_ssize_t outpos = *outp-PyUnicode_AS_UNICODE(*outobj);
         /* exponentially overallocate to minimize reallocations */
         if (requiredsize < 2 * oldsize)
             requiredsize = 2 * oldsize;
-        if (PyUnicode_Resize(outobj, requiredsize) < 0)
+        *outobj = PyMem_Realloc(*outobj, requiredsize * sizeof(Py_UCS4));
+        if (*outobj == 0)
             return -1;
-        *outp = PyUnicode_AS_UNICODE(*outobj) + outpos;
+        *psize = requiredsize;
     }
     return 0;
 }
@@ -6120,37 +7262,43 @@
    The called must decref result.
    Return 0 on success, -1 on error. */
 static int
-charmaptranslate_output(const Py_UNICODE *startinp, const Py_UNICODE *curinp,
-                        Py_ssize_t insize, PyObject *mapping, PyObject **outobj, Py_UNICODE **outp,
+charmaptranslate_output(PyObject *input, Py_ssize_t ipos,
+                        PyObject *mapping, Py_UCS4 **output,
+                        Py_ssize_t *osize, Py_ssize_t *opos,
                         PyObject **res)
 {
-    if (charmaptranslate_lookup(*curinp, mapping, res))
+    Py_UCS4 curinp = PyUnicode_READ_CHAR(input, ipos);
+    if (charmaptranslate_lookup(curinp, mapping, res))
         return -1;
     if (*res==NULL) {
         /* not found => default to 1:1 mapping */
-        *(*outp)++ = *curinp;
+        (*output)[(*opos)++] = curinp;
     }
     else if (*res==Py_None)
         ;
     else if (PyLong_Check(*res)) {
         /* no overflow check, because we know that the space is enough */
-        *(*outp)++ = (Py_UNICODE)PyLong_AS_LONG(*res);
+        (*output)[(*opos)++] = (Py_UCS4)PyLong_AS_LONG(*res);
     }
     else if (PyUnicode_Check(*res)) {
-        Py_ssize_t repsize = PyUnicode_GET_SIZE(*res);
+        Py_ssize_t repsize;
+        if (PyUnicode_READY(*res) == -1)
+            return -1;
+        repsize = PyUnicode_GET_LENGTH(*res);
         if (repsize==1) {
             /* no overflow check, because we know that the space is enough */
-            *(*outp)++ = *PyUnicode_AS_UNICODE(*res);
+            (*output)[(*opos)++] = PyUnicode_READ_CHAR(*res, 0);
         }
         else if (repsize!=0) {
             /* more than one character */
-            Py_ssize_t requiredsize = (*outp-PyUnicode_AS_UNICODE(*outobj)) +
-                (insize - (curinp-startinp)) +
+            Py_ssize_t requiredsize = *opos +
+                (PyUnicode_GET_LENGTH(input) - ipos) +
                 repsize - 1;
-            if (charmaptranslate_makespace(outobj, outp, requiredsize))
+            Py_ssize_t i;
+            if (charmaptranslate_makespace(output, osize, requiredsize))
                 return -1;
-            memcpy(*outp, PyUnicode_AS_UNICODE(*res), sizeof(Py_UNICODE)*repsize);
-            *outp += repsize;
+            for(i = 0; i < repsize; i++)
+                (*output)[(*opos)++] = PyUnicode_READ_CHAR(*res, i);
         }
     }
     else
@@ -6159,20 +7307,20 @@
 }
 
 PyObject *
-PyUnicode_TranslateCharmap(const Py_UNICODE *p,
-                           Py_ssize_t size,
-                           PyObject *mapping,
-                           const char *errors)
-{
-    /* output object */
-    PyObject *res = NULL;
-    /* pointers to the beginning and end+1 of input */
-    const Py_UNICODE *startp = p;
-    const Py_UNICODE *endp = p + size;
-    /* pointer into the output */
-    Py_UNICODE *str;
+_PyUnicode_TranslateCharmap(PyObject *input,
+                            PyObject *mapping,
+                            const char *errors)
+{
+    /* input object */
+    char *idata;
+    Py_ssize_t size, i;
+    int kind;
+    /* output buffer */
+    Py_UCS4 *output = NULL;
+    Py_ssize_t osize;
+    PyObject *res;
     /* current output position */
-    Py_ssize_t respos = 0;
+    Py_ssize_t opos;
     char *reason = "character maps to <undefined>";
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
@@ -6186,38 +7334,52 @@
         return NULL;
     }
 
+    if (PyUnicode_READY(input) == -1)
+        return NULL;
+    idata = (char*)PyUnicode_DATA(input);
+    kind = PyUnicode_KIND(input);
+    size = PyUnicode_GET_LENGTH(input);
+    i = 0;
+
+    if (size == 0) {
+        Py_INCREF(input);
+        return input;
+    }
+
     /* allocate enough for a simple 1:1 translation without
        replacements, if we need more, we'll resize */
-    res = PyUnicode_FromUnicode(NULL, size);
-    if (res == NULL)
+    osize = size;
+    output = PyMem_Malloc(osize * sizeof(Py_UCS4));
+    opos = 0;
+    if (output == NULL) {
+        PyErr_NoMemory();
         goto onError;
-    if (size == 0)
-        return res;
-    str = PyUnicode_AS_UNICODE(res);
-
-    while (p<endp) {
+    }
+
+    while (i<size) {
         /* try to encode it */
         PyObject *x = NULL;
-        if (charmaptranslate_output(startp, p, size, mapping, &res, &str, &x)) {
+        if (charmaptranslate_output(input, i, mapping,
+                                    &output, &osize, &opos, &x)) {
             Py_XDECREF(x);
             goto onError;
         }
         Py_XDECREF(x);
         if (x!=Py_None) /* it worked => adjust input pointer */
-            ++p;
+            ++i;
         else { /* untranslatable character */
             PyObject *repunicode = NULL; /* initialize to prevent gcc warning */
             Py_ssize_t repsize;
             Py_ssize_t newpos;
-            Py_UNICODE *uni2;
+            Py_ssize_t uni2;
             /* startpos for collecting untranslatable chars */
-            const Py_UNICODE *collstart = p;
-            const Py_UNICODE *collend = p+1;
-            const Py_UNICODE *coll;
+            Py_ssize_t collstart = i;
+            Py_ssize_t collend = i+1;
+            Py_ssize_t coll;
 
             /* find all untranslatable characters */
-            while (collend < endp) {
-                if (charmaptranslate_lookup(*collend, mapping, &x))
+            while (collend < size) {
+                if (charmaptranslate_lookup(PyUnicode_READ(kind,idata, collend), mapping, &x))
                     goto onError;
                 Py_XDECREF(x);
                 if (x!=Py_None)
@@ -6240,67 +7402,79 @@
             }
             switch (known_errorHandler) {
             case 1: /* strict */
-                raise_translate_exception(&exc, startp, size, collstart-startp, collend-startp, reason);
+                raise_translate_exception(&exc, input, collstart,
+                                          collend, reason);
                 goto onError;
             case 2: /* replace */
                 /* No need to check for space, this is a 1:1 replacement */
-                for (coll = collstart; coll<collend; ++coll)
-                    *str++ = '?';
+                for (coll = collstart; coll<collend; coll++)
+                    output[opos++] = '?';
                 /* fall through */
             case 3: /* ignore */
-                p = collend;
+                i = collend;
                 break;
             case 4: /* xmlcharrefreplace */
-                /* generate replacement (temporarily (mis)uses p) */
-                for (p = collstart; p < collend; ++p) {
+                /* generate replacement (temporarily (mis)uses i) */
+                for (i = collstart; i < collend; ++i) {
                     char buffer[2+29+1+1];
                     char *cp;
-                    sprintf(buffer, "&#%d;", (int)*p);
-                    if (charmaptranslate_makespace(&res, &str,
-                                                   (str-PyUnicode_AS_UNICODE(res))+strlen(buffer)+(endp-collend)))
+                    sprintf(buffer, "&#%d;", PyUnicode_READ(kind, idata, i));
+                    if (charmaptranslate_makespace(&output, &osize,
+                                                   opos+strlen(buffer)+(size-collend)))
                         goto onError;
                     for (cp = buffer; *cp; ++cp)
-                        *str++ = *cp;
+                        output[opos++] = *cp;
                 }
-                p = collend;
+                i = collend;
                 break;
             default:
                 repunicode = unicode_translate_call_errorhandler(errors, &errorHandler,
-                                                                 reason, startp, size, &exc,
-                                                                 collstart-startp, collend-startp, &newpos);
-                if (repunicode == NULL)
+                                                                 reason, input, &exc,
+                                                                 collstart, collend, &newpos);
+                if (repunicode == NULL || PyUnicode_READY(repunicode) == -1)
                     goto onError;
                 /* generate replacement  */
-                repsize = PyUnicode_GET_SIZE(repunicode);
-                if (charmaptranslate_makespace(&res, &str,
-                                               (str-PyUnicode_AS_UNICODE(res))+repsize+(endp-collend))) {
+                repsize = PyUnicode_GET_LENGTH(repunicode);
+                if (charmaptranslate_makespace(&output, &osize,
+                                               opos+repsize+(size-collend))) {
                     Py_DECREF(repunicode);
                     goto onError;
                 }
-                for (uni2 = PyUnicode_AS_UNICODE(repunicode); repsize-->0; ++uni2)
-                    *str++ = *uni2;
-                p = startp + newpos;
+                for (uni2 = 0; repsize-->0; ++uni2)
+                    output[opos++] = PyUnicode_READ_CHAR(repunicode, uni2);
+                i = newpos;
                 Py_DECREF(repunicode);
             }
         }
     }
-    /* Resize if we allocated to much */
-    respos = str-PyUnicode_AS_UNICODE(res);
-    if (respos<PyUnicode_GET_SIZE(res)) {
-        if (PyUnicode_Resize(&res, respos) < 0)
-            goto onError;
-    }
+    res = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, output, opos);
+    if (!res)
+        goto onError;
+    PyMem_Free(output);
     Py_XDECREF(exc);
     Py_XDECREF(errorHandler);
     return res;
 
   onError:
-    Py_XDECREF(res);
+    PyMem_Free(output);
     Py_XDECREF(exc);
     Py_XDECREF(errorHandler);
     return NULL;
 }
 
+/* Deprecated. Use PyUnicode_Translate instead. */
+PyObject *
+PyUnicode_TranslateCharmap(const Py_UNICODE *p,
+                           Py_ssize_t size,
+                           PyObject *mapping,
+                           const char *errors)
+{
+    PyObject *unicode = PyUnicode_FromUnicode(p, size);
+    if (!unicode)
+        return NULL;
+    return _PyUnicode_TranslateCharmap(unicode, mapping, errors);
+}
+
 PyObject *
 PyUnicode_Translate(PyObject *str,
                     PyObject *mapping,
@@ -6311,10 +7485,7 @@
     str = PyUnicode_FromObject(str);
     if (str == NULL)
         goto onError;
-    result = PyUnicode_TranslateCharmap(PyUnicode_AS_UNICODE(str),
-                                        PyUnicode_GET_SIZE(str),
-                                        mapping,
-                                        errors);
+    result = _PyUnicode_TranslateCharmap(str, mapping, errors);
     Py_DECREF(str);
     return result;
 
@@ -6323,6 +7494,60 @@
     return NULL;
 }
 
+static Py_UCS4
+fix_decimal_and_space_to_ascii(PyUnicodeObject *self)
+{
+    /* No need to call PyUnicode_READY(self) because this function is only
+       called as a callback from fixup() which does it already. */
+    const Py_ssize_t len = PyUnicode_GET_LENGTH(self);
+    const int kind = PyUnicode_KIND(self);
+    void *data = PyUnicode_DATA(self);
+    Py_UCS4 maxchar = 0, ch, fixed;
+    Py_ssize_t i;
+
+    for (i = 0; i < len; ++i) {
+        ch = PyUnicode_READ(kind, data, i);
+        fixed = 0;
+        if (ch > 127) {
+            if (Py_UNICODE_ISSPACE(ch))
+                fixed = ' ';
+            else {
+                const int decimal = Py_UNICODE_TODECIMAL(ch);
+                if (decimal >= 0)
+                    fixed = '0' + decimal;
+            }
+            if (fixed != 0) {
+                if (fixed > maxchar)
+                    maxchar = fixed;
+                PyUnicode_WRITE(kind, data, i, fixed);
+            }
+            else if (ch > maxchar)
+                maxchar = ch;
+        }
+        else if (ch > maxchar)
+            maxchar = ch;
+    }
+
+    return maxchar;
+}
+
+PyObject *
+_PyUnicode_TransformDecimalAndSpaceToASCII(PyObject *unicode)
+{
+    if (!PyUnicode_Check(unicode)) {
+        PyErr_BadInternalCall();
+        return NULL;
+    }
+    if (PyUnicode_READY(unicode) == -1)
+        return NULL;
+    if (PyUnicode_MAX_CHAR_VALUE(unicode) <= 127) {
+        /* If the string is already ASCII, just return the same string */
+        Py_INCREF(unicode);
+        return unicode;
+    }
+    return fixup((PyUnicodeObject *)unicode, fix_decimal_and_space_to_ascii);
+}
+
 PyObject *
 PyUnicode_TransformDecimalToASCII(Py_UNICODE *s,
                                   Py_ssize_t length)
@@ -6345,6 +7570,10 @@
                 p[i] = '0' + decimal;
         }
     }
+    if (PyUnicode_READY((PyUnicodeObject*)result) == -1) {
+        Py_DECREF(result);
+        return NULL;
+    }
     return result;
 }
 /* --- Decimal Encoder ---------------------------------------------------- */
@@ -6487,17 +7716,123 @@
 
 /* --- Helpers ------------------------------------------------------------ */
 
+#include "stringlib/ucs1lib.h"
+#include "stringlib/fastsearch.h"
+#include "stringlib/partition.h"
+#include "stringlib/split.h"
+#include "stringlib/count.h"
+#include "stringlib/find.h"
+#include "stringlib/localeutil.h"
+#include "stringlib/undef.h"
+
+#include "stringlib/ucs2lib.h"
+#include "stringlib/fastsearch.h"
+#include "stringlib/partition.h"
+#include "stringlib/split.h"
+#include "stringlib/count.h"
+#include "stringlib/find.h"
+#include "stringlib/localeutil.h"
+#include "stringlib/undef.h"
+
+#include "stringlib/ucs4lib.h"
+#include "stringlib/fastsearch.h"
+#include "stringlib/partition.h"
+#include "stringlib/split.h"
+#include "stringlib/count.h"
+#include "stringlib/find.h"
+#include "stringlib/localeutil.h"
+#include "stringlib/undef.h"
+
+static Py_ssize_t
+any_find_slice(Py_ssize_t Py_LOCAL_CALLBACK(ucs1)(const Py_UCS1*, Py_ssize_t,
+                                  const Py_UCS1*, Py_ssize_t,
+                                  Py_ssize_t, Py_ssize_t),
+               Py_ssize_t Py_LOCAL_CALLBACK(ucs2)(const Py_UCS2*, Py_ssize_t,
+                                  const Py_UCS2*, Py_ssize_t,
+                                  Py_ssize_t, Py_ssize_t),
+               Py_ssize_t Py_LOCAL_CALLBACK(ucs4)(const Py_UCS4*, Py_ssize_t,
+                                  const Py_UCS4*, Py_ssize_t,
+                                  Py_ssize_t, Py_ssize_t),
+               PyObject* s1, PyObject* s2,
+               Py_ssize_t start,
+               Py_ssize_t end)
+{
+    int kind1, kind2, kind;
+    void *buf1, *buf2;
+    Py_ssize_t len1, len2, result;
+
+    kind1 = PyUnicode_KIND(s1);
+    kind2 = PyUnicode_KIND(s2);
+    kind = kind1 > kind2 ? kind1 : kind2;
+    buf1 = PyUnicode_DATA(s1);
+    buf2 = PyUnicode_DATA(s2);
+    if (kind1 != kind)
+        buf1 = _PyUnicode_AsKind(s1, kind);
+    if (!buf1)
+        return -2;
+    if (kind2 != kind)
+        buf2 = _PyUnicode_AsKind(s2, kind);
+    if (!buf2) {
+        if (kind1 != kind) PyMem_Free(buf1);
+        return -2;
+    }
+    len1 = PyUnicode_GET_LENGTH(s1);
+    len2 = PyUnicode_GET_LENGTH(s2);
+
+    switch(kind) {
+    case PyUnicode_1BYTE_KIND:
+        result = ucs1(buf1, len1, buf2, len2, start, end);
+        break;
+    case PyUnicode_2BYTE_KIND:
+        result = ucs2(buf1, len1, buf2, len2, start, end);
+        break;
+    case PyUnicode_4BYTE_KIND:
+        result = ucs4(buf1, len1, buf2, len2, start, end);
+        break;
+    default:
+        assert(0); result = -2;
+    }
+
+    if (kind1 != kind)
+        PyMem_Free(buf1);
+    if (kind2 != kind)
+        PyMem_Free(buf2);
+
+    return result;
+}
+
+Py_ssize_t
+_PyUnicode_InsertThousandsGrouping(int kind, void *data,
+                                   Py_ssize_t n_buffer,
+                                   void *digits, Py_ssize_t n_digits,
+                                   Py_ssize_t min_width,
+                                   const char *grouping,
+                                   const char *thousands_sep)
+{
+    switch(kind) {
+    case PyUnicode_1BYTE_KIND:
+        return _PyUnicode_ucs1_InsertThousandsGrouping(
+            (Py_UCS1*)data, n_buffer, (Py_UCS1*)digits, n_digits,
+            min_width, grouping, thousands_sep);
+    case PyUnicode_2BYTE_KIND:
+        return _PyUnicode_ucs2_InsertThousandsGrouping(
+            (Py_UCS2*)data, n_buffer, (Py_UCS2*)digits, n_digits,
+            min_width, grouping, thousands_sep);
+    case PyUnicode_4BYTE_KIND:
+        return _PyUnicode_ucs4_InsertThousandsGrouping(
+            (Py_UCS4*)data, n_buffer, (Py_UCS4*)digits, n_digits,
+            min_width, grouping, thousands_sep);
+    }
+    assert(0);
+    return -1;
+}
+
+
 #include "stringlib/unicodedefs.h"
 #include "stringlib/fastsearch.h"
 
 #include "stringlib/count.h"
 #include "stringlib/find.h"
-#include "stringlib/partition.h"
-#include "stringlib/split.h"
-
-#define _Py_InsertThousandsGrouping _PyUnicode_InsertThousandsGrouping
-#define _Py_InsertThousandsGroupingLocale _PyUnicode_InsertThousandsGroupingLocale
-#include "stringlib/localeutil.h"
 
 /* helper macro to fixup start/end slice values */
 #define ADJUST_INDICES(start, end, len)         \
@@ -6514,28 +7849,6 @@
             start = 0;                          \
     }
 
-/* _Py_UNICODE_NEXT is a private macro used to retrieve the character pointed
- * by 'ptr', possibly combining surrogate pairs on narrow builds.
- * 'ptr' and 'end' must be Py_UNICODE*, with 'ptr' pointing at the character
- * that should be returned and 'end' pointing to the end of the buffer.
- * ('end' is used on narrow builds to detect a lone surrogate at the
- * end of the buffer that should be returned unchanged.)
- * The ptr and end arguments should be side-effect free and ptr must an lvalue.
- * The type of the returned char is always Py_UCS4.
- *
- * Note: the macro advances ptr to next char, so it might have side-effects
- *       (especially if used with other macros).
- */
-#ifdef Py_UNICODE_WIDE
-#define _Py_UNICODE_NEXT(ptr, end) *(ptr)++
-#else
-#define _Py_UNICODE_NEXT(ptr, end)                                     \
-     (((Py_UNICODE_IS_HIGH_SURROGATE(*(ptr)) && (ptr) < (end)) &&      \
-        Py_UNICODE_IS_LOW_SURROGATE((ptr)[1])) ?                       \
-       ((ptr) += 2,Py_UNICODE_JOIN_SURROGATES((ptr)[-2], (ptr)[-1])) : \
-       (Py_UCS4)*(ptr)++)
-#endif
-
 Py_ssize_t
 PyUnicode_Count(PyObject *str,
                 PyObject *substr,
@@ -6545,26 +7858,76 @@
     Py_ssize_t result;
     PyUnicodeObject* str_obj;
     PyUnicodeObject* sub_obj;
+    int kind1, kind2, kind;
+    void *buf1 = NULL, *buf2 = NULL;
+    Py_ssize_t len1, len2;
 
     str_obj = (PyUnicodeObject*) PyUnicode_FromObject(str);
-    if (!str_obj)
+    if (!str_obj || PyUnicode_READY(str_obj) == -1)
         return -1;
     sub_obj = (PyUnicodeObject*) PyUnicode_FromObject(substr);
-    if (!sub_obj) {
+    if (!sub_obj || PyUnicode_READY(str_obj) == -1) {
         Py_DECREF(str_obj);
         return -1;
     }
 
-    ADJUST_INDICES(start, end, str_obj->length);
-    result = stringlib_count(
-        str_obj->str + start, end - start, sub_obj->str, sub_obj->length,
-        PY_SSIZE_T_MAX
-        );
+    kind1 = PyUnicode_KIND(str_obj);
+    kind2 = PyUnicode_KIND(sub_obj);
+    kind = kind1 > kind2 ? kind1 : kind2;
+    buf1 = PyUnicode_DATA(str_obj);
+    if (kind1 != kind)
+        buf1 = _PyUnicode_AsKind((PyObject*)str_obj, kind);
+    if (!buf1)
+        goto onError;
+    buf2 = PyUnicode_DATA(sub_obj);
+    if (kind2 != kind)
+        buf2 = _PyUnicode_AsKind((PyObject*)sub_obj, kind);
+    if (!buf2)
+        goto onError;
+    len1 = PyUnicode_GET_LENGTH(str_obj);
+    len2 = PyUnicode_GET_LENGTH(sub_obj);
+
+    ADJUST_INDICES(start, end, len1);
+    switch(kind) {
+    case PyUnicode_1BYTE_KIND:
+        result = ucs1lib_count(
+            ((Py_UCS1*)buf1) + start, end - start,
+            buf2, len2, PY_SSIZE_T_MAX
+            );
+        break;
+    case PyUnicode_2BYTE_KIND:
+        result = ucs2lib_count(
+            ((Py_UCS2*)buf1) + start, end - start,
+            buf2, len2, PY_SSIZE_T_MAX
+            );
+        break;
+    case PyUnicode_4BYTE_KIND:
+        result = ucs4lib_count(
+            ((Py_UCS4*)buf1) + start, end - start,
+            buf2, len2, PY_SSIZE_T_MAX
+            );
+        break;
+    default:
+        assert(0); result = 0;
+    }
 
     Py_DECREF(sub_obj);
     Py_DECREF(str_obj);
 
+    if (kind1 != kind)
+        PyMem_Free(buf1);
+    if (kind2 != kind)
+        PyMem_Free(buf2);
+
     return result;
+  onError:
+    Py_DECREF(sub_obj);
+    Py_DECREF(str_obj);
+    if (kind1 != kind && buf1)
+        PyMem_Free(buf1);
+    if (kind2 != kind && buf2)
+        PyMem_Free(buf2);
+    return -1;
 }
 
 Py_ssize_t
@@ -6577,25 +7940,23 @@
     Py_ssize_t result;
 
     str = PyUnicode_FromObject(str);
-    if (!str)
+    if (!str || PyUnicode_READY(str) == -1)
         return -2;
     sub = PyUnicode_FromObject(sub);
-    if (!sub) {
+    if (!sub || PyUnicode_READY(sub) == -1) {
         Py_DECREF(str);
         return -2;
     }
 
     if (direction > 0)
-        result = stringlib_find_slice(
-            PyUnicode_AS_UNICODE(str), PyUnicode_GET_SIZE(str),
-            PyUnicode_AS_UNICODE(sub), PyUnicode_GET_SIZE(sub),
-            start, end
+        result = any_find_slice(
+            ucs1lib_find_slice, ucs2lib_find_slice, ucs4lib_find_slice,
+            str, sub, start, end
             );
     else
-        result = stringlib_rfind_slice(
-            PyUnicode_AS_UNICODE(str), PyUnicode_GET_SIZE(str),
-            PyUnicode_AS_UNICODE(sub), PyUnicode_GET_SIZE(sub),
-            start, end
+        result = any_find_slice(
+            ucs1lib_rfind_slice, ucs2lib_rfind_slice, ucs4lib_rfind_slice,
+            str, sub, start, end
             );
 
     Py_DECREF(str);
@@ -6604,6 +7965,27 @@
     return result;
 }
 
+Py_ssize_t
+PyUnicode_FindChar(PyObject *str, Py_UCS4 ch,
+                   Py_ssize_t start, Py_ssize_t end,
+                   int direction)
+{
+    char *result;
+    int kind;
+    if (PyUnicode_READY(str) == -1)
+        return -2;
+    if (end > PyUnicode_GET_LENGTH(str))
+        end = PyUnicode_GET_LENGTH(str);
+    kind = PyUnicode_KIND(str);
+    result = findchar(PyUnicode_1BYTE_DATA(str)
+                      + PyUnicode_KIND_SIZE(kind, start),
+                      kind,
+                      end-start, ch, direction);
+    if (!result)
+        return -1;
+    return (result-(char*)PyUnicode_DATA(str)) >> (kind-1);
+}
+
 static int
 tailmatch(PyUnicodeObject *self,
           PyUnicodeObject *substring,
@@ -6611,20 +7993,62 @@
           Py_ssize_t end,
           int direction)
 {
-    if (substring->length == 0)
+    int kind_self;
+    int kind_sub;
+    void *data_self;
+    void *data_sub;
+    Py_ssize_t offset;
+    Py_ssize_t i;
+    Py_ssize_t end_sub;
+
+    if (PyUnicode_READY(self) == -1 ||
+        PyUnicode_READY(substring) == -1)
+        return 0;
+
+    if (PyUnicode_GET_LENGTH(substring) == 0)
         return 1;
 
-    ADJUST_INDICES(start, end, self->length);
-    end -= substring->length;
+    ADJUST_INDICES(start, end, PyUnicode_GET_LENGTH(self));
+    end -= PyUnicode_GET_LENGTH(substring);
     if (end < start)
         return 0;
 
-    if (direction > 0) {
-        if (Py_UNICODE_MATCH(self, end, substring))
+    kind_self = PyUnicode_KIND(self);
+    data_self = PyUnicode_DATA(self);
+    kind_sub = PyUnicode_KIND(substring);
+    data_sub = PyUnicode_DATA(substring);
+    end_sub = PyUnicode_GET_LENGTH(substring) - 1;
+
+    if (direction > 0)
+        offset = end;
+    else
+        offset = start;
+
+    if (PyUnicode_READ(kind_self, data_self, offset) ==
+        PyUnicode_READ(kind_sub, data_sub, 0) &&
+        PyUnicode_READ(kind_self, data_self, offset + end_sub) ==
+        PyUnicode_READ(kind_sub, data_sub, end_sub)) {
+        /* If both are of the same kind, memcmp is sufficient */
+        if (kind_self == kind_sub) {
+            return ! memcmp((char *)data_self +
+                                (offset * PyUnicode_CHARACTER_SIZE(substring)),
+                            data_sub,
+                            PyUnicode_GET_LENGTH(substring) *
+                                PyUnicode_CHARACTER_SIZE(substring));
+        }
+        /* otherwise we have to compare each character by first accesing it */
+        else {
+            /* We do not need to compare 0 and len(substring)-1 because
+               the if statement above ensured already that they are equal
+               when we end up here. */
+            // TODO: honor direction and do a forward or backwards search
+            for (i = 1; i < end_sub; ++i) {
+                if (PyUnicode_READ(kind_self, data_self, offset + i) !=
+                    PyUnicode_READ(kind_sub, data_sub, i))
+                    return 0;
+            }
             return 1;
-    } else {
-        if (Py_UNICODE_MATCH(self, start, substring))
-            return 1;
+        }
     }
 
     return 0;
@@ -6661,18 +8085,39 @@
 
 static PyObject *
 fixup(PyUnicodeObject *self,
-      int (*fixfct)(PyUnicodeObject *s))
-{
-
-    PyUnicodeObject *u;
-
-    u = (PyUnicodeObject*) PyUnicode_FromUnicode(NULL, self->length);
+      Py_UCS4 (*fixfct)(PyUnicodeObject *s))
+{
+    PyObject *u;
+    Py_UCS4 maxchar_old, maxchar_new = 0;
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+    maxchar_old = PyUnicode_MAX_CHAR_VALUE(self);
+    u = PyUnicode_New(PyUnicode_GET_LENGTH(self),
+                      maxchar_old);
     if (u == NULL)
         return NULL;
 
-    Py_UNICODE_COPY(u->str, self->str, self->length);
-
-    if (!fixfct(u) && PyUnicode_CheckExact(self)) {
+    Py_MEMCPY(PyUnicode_1BYTE_DATA(u), PyUnicode_1BYTE_DATA(self),
+              PyUnicode_GET_LENGTH(u) * PyUnicode_CHARACTER_SIZE(u));
+
+    /* fix functions return the new maximum character in a string,
+       if the kind of the resulting unicode object does not change,
+       everything is fine.  Otherwise we need to change the string kind
+       and re-run the fix function. */
+    maxchar_new = fixfct((PyUnicodeObject*)u);
+    if (maxchar_new == 0)
+        /* do nothing, keep maxchar_new at 0 which means no changes. */;
+    else if (maxchar_new <= 127)
+        maxchar_new = 127;
+    else if (maxchar_new <= 255)
+        maxchar_new = 255;
+    else if (maxchar_new <= 65535)
+        maxchar_new = 65535;
+    else
+        maxchar_new = 1114111; /* 0x10ffff */
+
+    if (!maxchar_new && PyUnicode_CheckExact(self)) {
         /* fixfct should return TRUE if it modified the buffer. If
            FALSE, return a reference to the original buffer instead
            (to save space, not time) */
@@ -6680,123 +8125,207 @@
         Py_DECREF(u);
         return (PyObject*) self;
     }
-    return (PyObject*) u;
-}
-
-static int
+    else if (maxchar_new == maxchar_old) {
+        return u;
+    }
+    else {
+        /* In case the maximum character changed, we need to
+           convert the string to the new category. */
+        PyObject *v = PyUnicode_New(
+                             PyUnicode_GET_LENGTH(self), maxchar_new);
+        if (v == NULL) {
+            Py_DECREF(u);
+            return NULL;
+        }
+        if (maxchar_new > maxchar_old) {
+            /* If the maxchar increased so that the kind changed, not all
+               characters are representable anymore and we need to fix the
+               string again. This only happens in very few cases. */
+            PyUnicode_CopyCharacters(v, 0, (PyObject*)self, 0, PyUnicode_GET_LENGTH(self));
+            maxchar_old = fixfct((PyUnicodeObject*)v);
+            assert(maxchar_old > 0 && maxchar_old <= maxchar_new);
+        }
+        else
+            PyUnicode_CopyCharacters(v, 0, u, 0, PyUnicode_GET_LENGTH(self));
+
+        Py_DECREF(u);
+        return v;
+    }
+}
+
+static Py_UCS4
 fixupper(PyUnicodeObject *self)
 {
-    Py_ssize_t len = self->length;
-    Py_UNICODE *s = self->str;
-    int status = 0;
-
-    while (len-- > 0) {
-        register Py_UNICODE ch;
-
-        ch = Py_UNICODE_TOUPPER(*s);
-        if (ch != *s) {
-            status = 1;
-            *s = ch;
-        }
-        s++;
-    }
-
-    return status;
-}
-
-static int
+    /* No need to call PyUnicode_READY(self) because this function is only
+       called as a callback from fixup() which does it already. */
+    const Py_ssize_t len = PyUnicode_GET_LENGTH(self);
+    const int kind = PyUnicode_KIND(self);
+    void *data = PyUnicode_DATA(self);
+    int touched = 0;
+    Py_UCS4 maxchar = 0;
+    Py_ssize_t i;
+
+    for (i = 0; i < len; ++i) {
+        const Py_UCS4 ch = PyUnicode_READ(kind, data, i);
+        const Py_UCS4 up = Py_UNICODE_TOUPPER(ch);
+        if (up != ch) {
+            if (up > maxchar)
+                maxchar = up;
+            PyUnicode_WRITE(kind, data, i, up);
+            touched = 1;
+        }
+        else if (ch > maxchar)
+            maxchar = ch;
+    }
+
+    if (touched)
+        return maxchar;
+    else
+        return 0;
+}
+
+static Py_UCS4
 fixlower(PyUnicodeObject *self)
 {
-    Py_ssize_t len = self->length;
-    Py_UNICODE *s = self->str;
-    int status = 0;
-
-    while (len-- > 0) {
-        register Py_UNICODE ch;
-
-        ch = Py_UNICODE_TOLOWER(*s);
-        if (ch != *s) {
-            status = 1;
-            *s = ch;
-        }
-        s++;
-    }
-
-    return status;
-}
-
-static int
+    /* No need to call PyUnicode_READY(self) because fixup() which does it. */
+    const Py_ssize_t len = PyUnicode_GET_LENGTH(self);
+    const int kind = PyUnicode_KIND(self);
+    void *data = PyUnicode_DATA(self);
+    int touched = 0;
+    Py_UCS4 maxchar = 0;
+    Py_ssize_t i;
+
+    for(i = 0; i < len; ++i) {
+        const Py_UCS4 ch = PyUnicode_READ(kind, data, i);
+        const Py_UCS4 lo = Py_UNICODE_TOLOWER(ch);
+        if (lo != ch) {
+            if (lo > maxchar)
+                maxchar = lo;
+            PyUnicode_WRITE(kind, data, i, lo);
+            touched = 1;
+        }
+        else if (ch > maxchar)
+            maxchar = ch;
+    }
+
+    if (touched)
+        return maxchar;
+    else
+        return 0;
+}
+
+static Py_UCS4
 fixswapcase(PyUnicodeObject *self)
 {
-    Py_ssize_t len = self->length;
-    Py_UNICODE *s = self->str;
-    int status = 0;
-
-    while (len-- > 0) {
-        if (Py_UNICODE_ISUPPER(*s)) {
-            *s = Py_UNICODE_TOLOWER(*s);
-            status = 1;
-        } else if (Py_UNICODE_ISLOWER(*s)) {
-            *s = Py_UNICODE_TOUPPER(*s);
-            status = 1;
-        }
-        s++;
-    }
-
-    return status;
-}
-
-static int
+    /* No need to call PyUnicode_READY(self) because fixup() which does it. */
+    const Py_ssize_t len = PyUnicode_GET_LENGTH(self);
+    const int kind = PyUnicode_KIND(self);
+    void *data = PyUnicode_DATA(self);
+    int touched = 0;
+    Py_UCS4 maxchar = 0;
+    Py_ssize_t i;
+
+    for(i = 0; i < len; ++i) {
+        const Py_UCS4 ch = PyUnicode_READ(kind, data, i);
+        Py_UCS4 nu = 0;
+
+        if (Py_UNICODE_ISUPPER(ch))
+            nu = Py_UNICODE_TOLOWER(ch);
+        else if (Py_UNICODE_ISLOWER(ch))
+            nu = Py_UNICODE_TOUPPER(ch);
+
+        if (nu != 0) {
+            if (nu > maxchar)
+                maxchar = nu;
+            PyUnicode_WRITE(kind, data, i, nu);
+            touched = 1;
+        }
+        else if (ch > maxchar)
+            maxchar = ch;
+    }
+
+    if (touched)
+        return maxchar;
+    else
+        return 0;
+}
+
+static Py_UCS4
 fixcapitalize(PyUnicodeObject *self)
 {
-    Py_ssize_t len = self->length;
-    Py_UNICODE *s = self->str;
-    int status = 0;
+    /* No need to call PyUnicode_READY(self) because fixup() which does it. */
+    const Py_ssize_t len = PyUnicode_GET_LENGTH(self);
+    const int kind = PyUnicode_KIND(self);
+    void *data = PyUnicode_DATA(self);
+    int touched = 0;
+    Py_UCS4 maxchar = 0;
+    Py_ssize_t i = 0;
+    Py_UCS4 ch;
 
     if (len == 0)
         return 0;
-    if (!Py_UNICODE_ISUPPER(*s)) {
-        *s = Py_UNICODE_TOUPPER(*s);
-        status = 1;
-    }
-    s++;
-    while (--len > 0) {
-        if (!Py_UNICODE_ISLOWER(*s)) {
-            *s = Py_UNICODE_TOLOWER(*s);
-            status = 1;
-        }
-        s++;
-    }
-    return status;
-}
-
-static int
+
+    ch = PyUnicode_READ(kind, data, i);
+    if (!Py_UNICODE_ISUPPER(ch)) {
+        maxchar = Py_UNICODE_TOUPPER(ch);
+        PyUnicode_WRITE(kind, data, i, maxchar);
+        touched = 1;
+    }
+    ++i;
+    for(; i < len; ++i) {
+        ch = PyUnicode_READ(kind, data, i);
+        if (!Py_UNICODE_ISLOWER(ch)) {
+            const Py_UCS4 lo = Py_UNICODE_TOLOWER(ch);
+            if (lo > maxchar)
+                maxchar = lo;
+            PyUnicode_WRITE(kind, data, i, lo);
+            touched = 1;
+        }
+        else if (ch > maxchar)
+            maxchar = ch;
+    }
+
+    if (touched)
+        return maxchar;
+    else
+        return 0;
+}
+
+static Py_UCS4
 fixtitle(PyUnicodeObject *self)
 {
-    register Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
-    register Py_UNICODE *e;
+    /* No need to call PyUnicode_READY(self) because fixup() which does it. */
+    const Py_ssize_t len = PyUnicode_GET_LENGTH(self);
+    const int kind = PyUnicode_KIND(self);
+    void *data = PyUnicode_DATA(self);
+    Py_UCS4 maxchar = 0;
+    Py_ssize_t i = 0;
     int previous_is_cased;
 
     /* Shortcut for single character strings */
-    if (PyUnicode_GET_SIZE(self) == 1) {
-        Py_UNICODE ch = Py_UNICODE_TOTITLE(*p);
-        if (*p != ch) {
-            *p = ch;
-            return 1;
+    if (len == 1) {
+        const Py_UCS4 ch = PyUnicode_READ(kind, data, i);
+        const Py_UCS4 ti = Py_UNICODE_TOTITLE(ch);
+        if (ti != ch) {
+            PyUnicode_WRITE(kind, data, i, ti);
+            return ti;
         }
         else
             return 0;
     }
-
-    e = p + PyUnicode_GET_SIZE(self);
     previous_is_cased = 0;
-    for (; p < e; p++) {
-        register const Py_UNICODE ch = *p;
+    for(; i < len; ++i) {
+        const Py_UCS4 ch = PyUnicode_READ(kind, data, i);
+        Py_UCS4 nu;
 
         if (previous_is_cased)
-            *p = Py_UNICODE_TOLOWER(ch);
+            nu = Py_UNICODE_TOLOWER(ch);
         else
-            *p = Py_UNICODE_TOTITLE(ch);
+            nu = Py_UNICODE_TOTITLE(ch);
+
+        if (nu > maxchar)
+            maxchar = nu;
+        PyUnicode_WRITE(kind, data, i, nu);
 
         if (Py_UNICODE_ISLOWER(ch) ||
             Py_UNICODE_ISUPPER(ch) ||
@@ -6805,22 +8334,22 @@
         else
             previous_is_cased = 0;
     }
-    return 1;
+    return maxchar;
 }
 
 PyObject *
 PyUnicode_Join(PyObject *separator, PyObject *seq)
 {
-    const Py_UNICODE blank = ' ';
-    const Py_UNICODE *sep = &blank;
+    PyObject *sep = NULL;
     Py_ssize_t seplen = 1;
-    PyUnicodeObject *res = NULL; /* the result */
-    Py_UNICODE *res_p;       /* pointer to free byte in res's string area */
+    PyObject *res = NULL; /* the result */
     PyObject *fseq;          /* PySequence_Fast(seq) */
     Py_ssize_t seqlen;       /* len(fseq) -- number of items in sequence */
     PyObject **items;
     PyObject *item;
-    Py_ssize_t sz, i;
+    Py_ssize_t sz, i, res_offset;
+    Py_UCS4 maxchar = 0;
+    Py_UCS4 item_maxchar;
 
     fseq = PySequence_Fast(seq, "");
     if (fseq == NULL) {
@@ -6834,7 +8363,7 @@
     seqlen = PySequence_Fast_GET_SIZE(fseq);
     /* If empty sequence, return u"". */
     if (seqlen == 0) {
-        res = _PyUnicode_New(0);  /* empty sequence; return u"" */
+        res = PyUnicode_New(0, 0);
         goto Done;
     }
     items = PySequence_Fast_ITEMS(fseq);
@@ -6843,15 +8372,17 @@
         item = items[0];
         if (PyUnicode_CheckExact(item)) {
             Py_INCREF(item);
-            res = (PyUnicodeObject *)item;
+            res = item;
             goto Done;
         }
     }
     else {
         /* Set up sep and seplen */
         if (separator == NULL) {
-            sep = &blank;
-            seplen = 1;
+            /* fall back to a blank space separator */
+            sep = PyUnicode_FromOrdinal(' ');
+            if (!sep || PyUnicode_READY(sep) == -1)
+                goto onError;
         }
         else {
             if (!PyUnicode_Check(separator)) {
@@ -6861,8 +8392,14 @@
                              Py_TYPE(separator)->tp_name);
                 goto onError;
             }
-            sep = PyUnicode_AS_UNICODE(separator);
-            seplen = PyUnicode_GET_SIZE(separator);
+            if (PyUnicode_READY(separator) == -1)
+                goto onError;
+            sep = separator;
+            seplen = PyUnicode_GET_LENGTH(separator);
+            maxchar = PyUnicode_MAX_CHAR_VALUE(separator);
+            /* inc refcount to keep this code path symetric with the
+               above case of a blank separator */
+            Py_INCREF(sep);
         }
     }
 
@@ -6882,7 +8419,12 @@
                          i, Py_TYPE(item)->tp_name);
             goto onError;
         }
-        sz += PyUnicode_GET_SIZE(item);
+        if (PyUnicode_READY(item) == -1)
+            goto onError;
+        sz += PyUnicode_GET_LENGTH(item);
+        item_maxchar = PyUnicode_MAX_CHAR_VALUE(item);
+        if (item_maxchar > maxchar)
+            maxchar = item_maxchar;
         if (i != 0)
             sz += seplen;
         if (sz < old_sz || sz > PY_SSIZE_T_MAX) {
@@ -6892,42 +8434,70 @@
         }
     }
 
-    res = _PyUnicode_New(sz);
+    res = PyUnicode_New(sz, maxchar);
     if (res == NULL)
         goto onError;
 
     /* Catenate everything. */
-    res_p = PyUnicode_AS_UNICODE(res);
-    for (i = 0; i < seqlen; ++i) {
+    for (i = 0, res_offset = 0; i < seqlen; ++i) {
         Py_ssize_t itemlen;
         item = items[i];
-        itemlen = PyUnicode_GET_SIZE(item);
+        itemlen = PyUnicode_GET_LENGTH(item);
         /* Copy item, and maybe the separator. */
         if (i) {
-            Py_UNICODE_COPY(res_p, sep, seplen);
-            res_p += seplen;
-        }
-        Py_UNICODE_COPY(res_p, PyUnicode_AS_UNICODE(item), itemlen);
-        res_p += itemlen;
-    }
+            PyUnicode_CopyCharacters(res, res_offset,
+                                     sep, 0, seplen);
+            res_offset += seplen;
+        }
+        PyUnicode_CopyCharacters(res, res_offset,
+                                 item, 0, itemlen);
+        res_offset += itemlen;
+    }
+    assert(res_offset == PyUnicode_GET_LENGTH(res));
 
   Done:
     Py_DECREF(fseq);
-    return (PyObject *)res;
+    Py_XDECREF(sep);
+    return res;
 
   onError:
     Py_DECREF(fseq);
+    Py_XDECREF(sep);
     Py_XDECREF(res);
     return NULL;
 }
 
+#define FILL(kind, data, value, start, length) \
+    do { \
+        Py_ssize_t i_ = 0; \
+        assert(kind != PyUnicode_WCHAR_KIND); \
+        switch ((kind)) { \
+        case PyUnicode_1BYTE_KIND: { \
+            unsigned char * to_ = (unsigned char *)((data)) + (start); \
+            memset(to_, (unsigned char)value, length); \
+            break; \
+        } \
+        case PyUnicode_2BYTE_KIND: { \
+            Py_UCS2 * to_ = (Py_UCS2 *)((data)) + (start); \
+            for (; i_ < (length); ++i_, ++to_) *to_ = (value); \
+            break; \
+        } \
+        default: { \
+            Py_UCS4 * to_ = (Py_UCS4 *)((data)) + (start); \
+            for (; i_ < (length); ++i_, ++to_) *to_ = (value); \
+            break; \
+        } \
+        } \
+    } while (0)
+
 static PyUnicodeObject *
 pad(PyUnicodeObject *self,
     Py_ssize_t left,
     Py_ssize_t right,
-    Py_UNICODE fill)
-{
-    PyUnicodeObject *u;
+    Py_UCS4 fill)
+{
+    PyObject *u;
+    Py_UCS4 maxchar;
 
     if (left < 0)
         left = 0;
@@ -6939,22 +8509,28 @@
         return self;
     }
 
-    if (left > PY_SSIZE_T_MAX - self->length ||
-        right > PY_SSIZE_T_MAX - (left + self->length)) {
+    if (left > PY_SSIZE_T_MAX - _PyUnicode_LENGTH(self) ||
+        right > PY_SSIZE_T_MAX - (left + _PyUnicode_LENGTH(self))) {
         PyErr_SetString(PyExc_OverflowError, "padded string is too long");
         return NULL;
     }
-    u = _PyUnicode_New(left + self->length + right);
+    maxchar = PyUnicode_MAX_CHAR_VALUE(self);
+    if (fill > maxchar)
+        maxchar = fill;
+    u = PyUnicode_New(left + _PyUnicode_LENGTH(self) + right, maxchar);
     if (u) {
+        int kind = PyUnicode_KIND(u);
+        void *data = PyUnicode_DATA(u);
         if (left)
-            Py_UNICODE_FILL(u->str, fill, left);
-        Py_UNICODE_COPY(u->str + left, self->str, self->length);
+            FILL(kind, data, fill, 0, left);
         if (right)
-            Py_UNICODE_FILL(u->str + left + self->length, fill, right);
-    }
-
-    return u;
-}
+            FILL(kind, data, fill, left + _PyUnicode_LENGTH(self), right);
+        PyUnicode_CopyCharacters(u, left, (PyObject*)self, 0, _PyUnicode_LENGTH(self));
+    }
+
+    return (PyUnicodeObject*)u;
+}
+#undef FILL
 
 PyObject *
 PyUnicode_Splitlines(PyObject *string, int keepends)
@@ -6962,13 +8538,29 @@
     PyObject *list;
 
     string = PyUnicode_FromObject(string);
-    if (string == NULL)
-        return NULL;
-
-    list = stringlib_splitlines(
-        (PyObject*) string, PyUnicode_AS_UNICODE(string),
-        PyUnicode_GET_SIZE(string), keepends);
-
+    if (string == NULL || PyUnicode_READY(string) == -1)
+        return NULL;
+
+    switch(PyUnicode_KIND(string)) {
+    case PyUnicode_1BYTE_KIND:
+        list = ucs1lib_splitlines(
+            (PyObject*) string, PyUnicode_1BYTE_DATA(string),
+            PyUnicode_GET_LENGTH(string), keepends);
+        break;
+    case PyUnicode_2BYTE_KIND:
+        list = ucs2lib_splitlines(
+            (PyObject*) string, PyUnicode_2BYTE_DATA(string),
+            PyUnicode_GET_LENGTH(string), keepends);
+        break;
+    case PyUnicode_4BYTE_KIND:
+        list = ucs4lib_splitlines(
+            (PyObject*) string, PyUnicode_4BYTE_DATA(string),
+            PyUnicode_GET_LENGTH(string), keepends);
+        break;
+    default:
+        assert(0);
+        list = 0;
+    }
     Py_DECREF(string);
     return list;
 }
@@ -6978,19 +8570,81 @@
       PyUnicodeObject *substring,
       Py_ssize_t maxcount)
 {
+    int kind1, kind2, kind;
+    void *buf1, *buf2;
+    Py_ssize_t len1, len2;
+    PyObject* out;
+
     if (maxcount < 0)
         maxcount = PY_SSIZE_T_MAX;
 
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+
     if (substring == NULL)
-        return stringlib_split_whitespace(
-            (PyObject*) self,  self->str, self->length, maxcount
-            );
-
-    return stringlib_split(
-        (PyObject*) self,  self->str, self->length,
-        substring->str, substring->length,
-        maxcount
-        );
+        switch(PyUnicode_KIND(self)) {
+        case PyUnicode_1BYTE_KIND:
+            return ucs1lib_split_whitespace(
+                (PyObject*) self,  PyUnicode_1BYTE_DATA(self),
+                PyUnicode_GET_LENGTH(self), maxcount
+                );
+        case PyUnicode_2BYTE_KIND:
+            return ucs2lib_split_whitespace(
+                (PyObject*) self,  PyUnicode_2BYTE_DATA(self),
+                PyUnicode_GET_LENGTH(self), maxcount
+                );
+        case PyUnicode_4BYTE_KIND:
+            return ucs4lib_split_whitespace(
+                (PyObject*) self,  PyUnicode_4BYTE_DATA(self),
+                PyUnicode_GET_LENGTH(self), maxcount
+                );
+        default:
+            assert(0);
+            return NULL;
+        }
+
+    if (PyUnicode_READY(substring) == -1)
+        return NULL;
+
+    kind1 = PyUnicode_KIND(self);
+    kind2 = PyUnicode_KIND(substring);
+    kind = kind1 > kind2 ? kind1 : kind2;
+    buf1 = PyUnicode_DATA(self);
+    buf2 = PyUnicode_DATA(substring);
+    if (kind1 != kind)
+        buf1 = _PyUnicode_AsKind((PyObject*)self, kind);
+    if (!buf1)
+        return NULL;
+    if (kind2 != kind)
+        buf2 = _PyUnicode_AsKind((PyObject*)substring, kind);
+    if (!buf2) {
+        if (kind1 != kind) PyMem_Free(buf1);
+        return NULL;
+    }
+    len1 = PyUnicode_GET_LENGTH(self);
+    len2 = PyUnicode_GET_LENGTH(substring);
+
+    switch(kind) {
+    case PyUnicode_1BYTE_KIND:
+        out = ucs1lib_split(
+            (PyObject*) self,  buf1, len1, buf2, len2, maxcount);
+        break;
+    case PyUnicode_2BYTE_KIND:
+        out = ucs2lib_split(
+            (PyObject*) self,  buf1, len1, buf2, len2, maxcount);
+        break;
+    case PyUnicode_4BYTE_KIND:
+        out = ucs4lib_split(
+            (PyObject*) self,  buf1, len1, buf2, len2, maxcount);
+        break;
+    default:
+        out = NULL;
+    }
+    if (kind1 != kind)
+        PyMem_Free(buf1);
+    if (kind2 != kind)
+        PyMem_Free(buf2);
+    return out;
 }
 
 static PyObject *
@@ -6998,159 +8652,366 @@
        PyUnicodeObject *substring,
        Py_ssize_t maxcount)
 {
+    int kind1, kind2, kind;
+    void *buf1, *buf2;
+    Py_ssize_t len1, len2;
+    PyObject* out;
+
     if (maxcount < 0)
         maxcount = PY_SSIZE_T_MAX;
 
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+
     if (substring == NULL)
-        return stringlib_rsplit_whitespace(
-            (PyObject*) self,  self->str, self->length, maxcount
-            );
-
-    return stringlib_rsplit(
-        (PyObject*) self,  self->str, self->length,
-        substring->str, substring->length,
-        maxcount
-        );
+        switch(PyUnicode_KIND(self)) {
+        case PyUnicode_1BYTE_KIND:
+            return ucs1lib_rsplit_whitespace(
+                (PyObject*) self,  PyUnicode_1BYTE_DATA(self),
+                PyUnicode_GET_LENGTH(self), maxcount
+                );
+        case PyUnicode_2BYTE_KIND:
+            return ucs2lib_rsplit_whitespace(
+                (PyObject*) self,  PyUnicode_2BYTE_DATA(self),
+                PyUnicode_GET_LENGTH(self), maxcount
+                );
+        case PyUnicode_4BYTE_KIND:
+            return ucs4lib_rsplit_whitespace(
+                (PyObject*) self,  PyUnicode_4BYTE_DATA(self),
+                PyUnicode_GET_LENGTH(self), maxcount
+                );
+        default:
+            assert(0);
+            return NULL;
+        }
+
+    if (PyUnicode_READY(substring) == -1)
+        return NULL;
+
+    kind1 = PyUnicode_KIND(self);
+    kind2 = PyUnicode_KIND(substring);
+    kind = kind1 > kind2 ? kind1 : kind2;
+    buf1 = PyUnicode_DATA(self);
+    buf2 = PyUnicode_DATA(substring);
+    if (kind1 != kind)
+        buf1 = _PyUnicode_AsKind((PyObject*)self, kind);
+    if (!buf1)
+        return NULL;
+    if (kind2 != kind)
+        buf2 = _PyUnicode_AsKind((PyObject*)substring, kind);
+    if (!buf2) {
+        if (kind1 != kind) PyMem_Free(buf1);
+        return NULL;
+    }
+    len1 = PyUnicode_GET_LENGTH(self);
+    len2 = PyUnicode_GET_LENGTH(substring);
+
+    switch(kind) {
+    case PyUnicode_1BYTE_KIND:
+        out = ucs1lib_rsplit(
+            (PyObject*) self,  buf1, len1, buf2, len2, maxcount);
+        break;
+    case PyUnicode_2BYTE_KIND:
+        out = ucs2lib_rsplit(
+            (PyObject*) self,  buf1, len1, buf2, len2, maxcount);
+        break;
+    case PyUnicode_4BYTE_KIND:
+        out = ucs4lib_rsplit(
+            (PyObject*) self,  buf1, len1, buf2, len2, maxcount);
+        break;
+    default:
+        out = NULL;
+    }
+    if (kind1 != kind)
+        PyMem_Free(buf1);
+    if (kind2 != kind)
+        PyMem_Free(buf2);
+    return out;
+}
+
+static Py_ssize_t
+anylib_find(int kind, void *buf1, Py_ssize_t len1,
+            void *buf2, Py_ssize_t len2, Py_ssize_t offset)
+{
+    switch(kind) {
+    case PyUnicode_1BYTE_KIND:
+        return ucs1lib_find(buf1, len1, buf2, len2, offset);
+    case PyUnicode_2BYTE_KIND:
+        return ucs2lib_find(buf1, len1, buf2, len2, offset);
+    case PyUnicode_4BYTE_KIND:
+        return ucs4lib_find(buf1, len1, buf2, len2, offset);
+    }
+    assert(0);
+    return -1;
+}
+
+static Py_ssize_t
+anylib_count(int kind, void* sbuf, Py_ssize_t slen,
+             void *buf1, Py_ssize_t len1, Py_ssize_t maxcount)
+{
+        switch(kind) {
+        case PyUnicode_1BYTE_KIND:
+            return ucs1lib_count(sbuf, slen, buf1, len1, maxcount);
+        case PyUnicode_2BYTE_KIND:
+            return ucs2lib_count(sbuf, slen, buf1, len1, maxcount);
+        case PyUnicode_4BYTE_KIND:
+            return ucs4lib_count(sbuf, slen, buf1, len1, maxcount);
+        }
+        assert(0);
+        return 0;
 }
 
 static PyObject *
-replace(PyUnicodeObject *self,
-        PyUnicodeObject *str1,
-        PyUnicodeObject *str2,
-        Py_ssize_t maxcount)
-{
-    PyUnicodeObject *u;
+replace(PyObject *self, PyObject *str1,
+        PyObject *str2, Py_ssize_t maxcount)
+{
+    PyObject *u;
+    char *sbuf = PyUnicode_DATA(self);
+    char *buf1 = PyUnicode_DATA(str1);
+    char *buf2 = PyUnicode_DATA(str2);
+    int srelease = 0, release1 = 0, release2 = 0;
+    int skind = PyUnicode_KIND(self);
+    int kind1 = PyUnicode_KIND(str1);
+    int kind2 = PyUnicode_KIND(str2);
+    Py_ssize_t slen = PyUnicode_GET_LENGTH(self);
+    Py_ssize_t len1 = PyUnicode_GET_LENGTH(str1);
+    Py_ssize_t len2 = PyUnicode_GET_LENGTH(str2);
 
     if (maxcount < 0)
         maxcount = PY_SSIZE_T_MAX;
-    else if (maxcount == 0 || self->length == 0)
+    else if (maxcount == 0 || slen == 0)
         goto nothing;
 
-    if (str1->length == str2->length) {
+    if (skind < kind1)
+        /* substring too wide to be present */
+        goto nothing;
+
+    if (len1 == len2) {
         Py_ssize_t i;
         /* same length */
-        if (str1->length == 0)
+        if (len1 == 0)
             goto nothing;
-        if (str1->length == 1) {
+        if (len1 == 1) {
             /* replace characters */
-            Py_UNICODE u1, u2;
-            if (!findchar(self->str, self->length, str1->str[0]))
+            Py_UCS4 u1, u2, maxchar;
+            int mayshrink, rkind;
+            u1 = PyUnicode_READ_CHAR(str1, 0);
+            if (!findchar(sbuf, PyUnicode_KIND(self),
+                          slen, u1, 1))
                 goto nothing;
-            u = (PyUnicodeObject*) PyUnicode_FromUnicode(NULL, self->length);
+            u2 = PyUnicode_READ_CHAR(str2, 0);
+            maxchar = PyUnicode_MAX_CHAR_VALUE(self);
+            /* Replacing u1 with u2 may cause a maxchar reduction in the
+               result string. */
+            mayshrink = maxchar > 127;
+            if (u2 > maxchar) {
+                maxchar = u2;
+                mayshrink = 0;
+            }
+            u = PyUnicode_New(slen, maxchar);
             if (!u)
-                return NULL;
-            Py_UNICODE_COPY(u->str, self->str, self->length);
-            u1 = str1->str[0];
-            u2 = str2->str[0];
-            for (i = 0; i < u->length; i++)
-                if (u->str[i] == u1) {
+                goto error;
+            PyUnicode_CopyCharacters(u, 0,
+                                     (PyObject*)self, 0, slen);
+            rkind = PyUnicode_KIND(u);
+            for (i = 0; i < PyUnicode_GET_LENGTH(u); i++)
+                if (PyUnicode_READ(rkind, PyUnicode_DATA(u), i) == u1) {
                     if (--maxcount < 0)
                         break;
-                    u->str[i] = u2;
+                    PyUnicode_WRITE(rkind, PyUnicode_DATA(u), i, u2);
                 }
+            if (mayshrink) {
+                PyObject *tmp = u;
+                u = PyUnicode_FromKindAndData(rkind, PyUnicode_DATA(tmp),
+                                              PyUnicode_GET_LENGTH(tmp));
+                Py_DECREF(tmp);
+            }
         } else {
-            i = stringlib_find(
-                self->str, self->length, str1->str, str1->length, 0
-                );
+            int rkind = skind;
+            char *res;
+            if (kind1 < rkind) {
+                /* widen substring */
+                buf1 = _PyUnicode_AsKind(str1, rkind);
+                if (!buf1) goto error;
+                release1 = 1;
+            }
+            i = anylib_find(rkind, sbuf, slen, buf1, len1, 0);
             if (i < 0)
                 goto nothing;
-            u = (PyUnicodeObject*) PyUnicode_FromUnicode(NULL, self->length);
-            if (!u)
-                return NULL;
-            Py_UNICODE_COPY(u->str, self->str, self->length);
-
+            if (rkind > kind2) {
+                /* widen replacement */
+                buf2 = _PyUnicode_AsKind(str2, rkind);
+                if (!buf2) goto error;
+                release2 = 1;
+            }
+            else if (rkind < kind2) {
+                /* widen self and buf1 */
+                rkind = kind2;
+                if (release1) PyMem_Free(buf1);
+                sbuf = _PyUnicode_AsKind(self, rkind);
+                if (!sbuf) goto error;
+                srelease = 1;
+                buf1 = _PyUnicode_AsKind(str1, rkind);
+                if (!buf1) goto error;
+                release1 = 1;
+            }
+            res = PyMem_Malloc(PyUnicode_KIND_SIZE(rkind, slen));
+            if (!res) {
+                PyErr_NoMemory();
+                goto error;
+            }
+            memcpy(res, sbuf, PyUnicode_KIND_SIZE(rkind, slen));
             /* change everything in-place, starting with this one */
-            Py_UNICODE_COPY(u->str+i, str2->str, str2->length);
-            i += str1->length;
+            memcpy(res + PyUnicode_KIND_SIZE(rkind, i),
+                   buf2,
+                   PyUnicode_KIND_SIZE(rkind, len2));
+            i += len1;
 
             while ( --maxcount > 0) {
-                i = stringlib_find(self->str+i, self->length-i,
-                                   str1->str, str1->length,
-                                   i);
+                i = anylib_find(rkind, sbuf+PyUnicode_KIND_SIZE(rkind, i),
+                                slen-i,
+                                buf1, len1, i);
                 if (i == -1)
                     break;
-                Py_UNICODE_COPY(u->str+i, str2->str, str2->length);
-                i += str1->length;
-            }
+                memcpy(res + PyUnicode_KIND_SIZE(rkind, i),
+                       buf2,
+                       PyUnicode_KIND_SIZE(rkind, len2));
+                i += len1;
+            }
+
+            u = PyUnicode_FromKindAndData(rkind, res, slen);
+            PyMem_Free(res);
+            if (!u) goto error;
         }
     } else {
 
-        Py_ssize_t n, i, j;
-        Py_ssize_t product, new_size, delta;
-        Py_UNICODE *p;
-
-        /* replace strings */
-        n = stringlib_count(self->str, self->length, str1->str, str1->length,
-                            maxcount);
+        Py_ssize_t n, i, j, ires;
+        Py_ssize_t product, new_size;
+        int rkind = skind;
+        char *res;
+
+        if (kind1 < rkind) {
+            buf1 = _PyUnicode_AsKind(str1, rkind);
+            if (!buf1) goto error;
+            release1 = 1;
+        }
+        n = anylib_count(rkind, sbuf, slen, buf1, len1, maxcount);
         if (n == 0)
             goto nothing;
-        /* new_size = self->length + n * (str2->length - str1->length)); */
-        delta = (str2->length - str1->length);
-        if (delta == 0) {
-            new_size = self->length;
-        } else {
-            product = n * (str2->length - str1->length);
-            if ((product / (str2->length - str1->length)) != n) {
+        if (kind2 < rkind) {
+            buf2 = _PyUnicode_AsKind(str2, rkind);
+            if (!buf2) goto error;
+            release2 = 1;
+        }
+        else if (kind2 > rkind) {
+            rkind = kind2;
+            sbuf = _PyUnicode_AsKind(self, rkind);
+            if (!sbuf) goto error;
+            srelease = 1;
+            if (release1) PyMem_Free(buf1);
+            buf1 = _PyUnicode_AsKind(str1, rkind);
+            if (!buf1) goto error;
+            release1 = 1;
+        }
+        /* new_size = PyUnicode_GET_LENGTH(self) + n * (PyUnicode_GET_LENGTH(str2) -
+           PyUnicode_GET_LENGTH(str1))); */
+        product = n * (len2-len1);
+        if ((product / (len2-len1)) != n) {
                 PyErr_SetString(PyExc_OverflowError,
                                 "replace string is too long");
-                return NULL;
-            }
-            new_size = self->length + product;
-            if (new_size < 0) {
-                PyErr_SetString(PyExc_OverflowError,
-                                "replace string is too long");
-                return NULL;
-            }
-        }
-        u = _PyUnicode_New(new_size);
-        if (!u)
-            return NULL;
-        i = 0;
-        p = u->str;
-        if (str1->length > 0) {
+                goto error;
+        }
+        new_size = slen + product;
+        if (new_size < 0 || new_size > (PY_SSIZE_T_MAX >> (rkind-1))) {
+            PyErr_SetString(PyExc_OverflowError,
+                            "replace string is too long");
+            goto error;
+        }
+        res = PyMem_Malloc(PyUnicode_KIND_SIZE(rkind, new_size));
+        if (!res)
+            goto error;
+        ires = i = 0;
+        if (len1 > 0) {
             while (n-- > 0) {
                 /* look for next match */
-                j = stringlib_find(self->str+i, self->length-i,
-                                   str1->str, str1->length,
-                                   i);
+                j = anylib_find(rkind,
+                                sbuf + PyUnicode_KIND_SIZE(rkind, i),
+                                slen-i, buf1, len1, i);
                 if (j == -1)
                     break;
                 else if (j > i) {
                     /* copy unchanged part [i:j] */
-                    Py_UNICODE_COPY(p, self->str+i, j-i);
-                    p += j - i;
+                    memcpy(res + PyUnicode_KIND_SIZE(rkind, ires),
+                           sbuf + PyUnicode_KIND_SIZE(rkind, i),
+                           PyUnicode_KIND_SIZE(rkind, j-i));
+                    ires += j - i;
                 }
                 /* copy substitution string */
-                if (str2->length > 0) {
-                    Py_UNICODE_COPY(p, str2->str, str2->length);
-                    p += str2->length;
+                if (len2 > 0) {
+                    memcpy(res + PyUnicode_KIND_SIZE(rkind, ires),
+                           buf2,
+                           PyUnicode_KIND_SIZE(rkind, len2));
+                    ires += len2;
                 }
-                i = j + str1->length;
-            }
-            if (i < self->length)
+                i = j + len1;
+            }
+            if (i < slen)
                 /* copy tail [i:] */
-                Py_UNICODE_COPY(p, self->str+i, self->length-i);
+                memcpy(res + PyUnicode_KIND_SIZE(rkind, ires),
+                       sbuf + PyUnicode_KIND_SIZE(rkind, i),
+                       PyUnicode_KIND_SIZE(rkind, slen-i));
         } else {
             /* interleave */
             while (n > 0) {
-                Py_UNICODE_COPY(p, str2->str, str2->length);
-                p += str2->length;
+                memcpy(res + PyUnicode_KIND_SIZE(rkind, ires),
+                       buf2,
+                       PyUnicode_KIND_SIZE(rkind, len2));
+                ires += len2;
                 if (--n <= 0)
                     break;
-                *p++ = self->str[i++];
-            }
-            Py_UNICODE_COPY(p, self->str+i, self->length-i);
-        }
-    }
-    return (PyObject *) u;
+                memcpy(res + PyUnicode_KIND_SIZE(rkind, ires),
+                       sbuf + PyUnicode_KIND_SIZE(rkind, i),
+                       PyUnicode_KIND_SIZE(rkind, 1));
+                ires++;
+                i++;
+            }
+            memcpy(res + PyUnicode_KIND_SIZE(rkind, ires),
+                   sbuf + PyUnicode_KIND_SIZE(rkind, i),
+                   PyUnicode_KIND_SIZE(rkind, slen-i));
+        }
+        u = PyUnicode_FromKindAndData(rkind, res, new_size);
+    }
+    if (srelease)
+        PyMem_FREE(sbuf);
+    if (release1)
+        PyMem_FREE(buf1);
+    if (release2)
+        PyMem_FREE(buf2);
+    return u;
 
   nothing:
     /* nothing to replace; return original string (when possible) */
+    if (srelease)
+        PyMem_FREE(sbuf);
+    if (release1)
+        PyMem_FREE(buf1);
+    if (release2)
+        PyMem_FREE(buf2);
     if (PyUnicode_CheckExact(self)) {
         Py_INCREF(self);
         return (PyObject *) self;
     }
-    return PyUnicode_FromUnicode(self->str, self->length);
+    return PyUnicode_FromKindAndData(PyUnicode_KIND(self),
+                                     PyUnicode_DATA(self),
+                                     PyUnicode_GET_LENGTH(self));
+  error:
+    if (srelease && sbuf)
+        PyMem_FREE(sbuf);
+    if (release1 && buf1)
+        PyMem_FREE(buf1);
+    if (release2 && buf2)
+        PyMem_FREE(buf2);
+    return NULL;
 }
 
 /* --- Unicode Object Methods --------------------------------------------- */
@@ -7222,9 +9083,8 @@
 static int
 convert_uc(PyObject *obj, void *addr)
 {
-    Py_UNICODE *fillcharloc = (Py_UNICODE *)addr;
+    Py_UCS4 *fillcharloc = (Py_UCS4 *)addr;
     PyObject *uniobj;
-    Py_UNICODE *unistr;
 
     uniobj = PyUnicode_FromObject(obj);
     if (uniobj == NULL) {
@@ -7232,14 +9092,17 @@
                         "The fill character cannot be converted to Unicode");
         return 0;
     }
-    if (PyUnicode_GET_SIZE(uniobj) != 1) {
+    if (PyUnicode_GET_LENGTH(uniobj) != 1) {
         PyErr_SetString(PyExc_TypeError,
                         "The fill character must be exactly one character long");
         Py_DECREF(uniobj);
         return 0;
     }
-    unistr = PyUnicode_AS_UNICODE(uniobj);
-    *fillcharloc = unistr[0];
+    if (PyUnicode_READY(uniobj)) {
+        Py_DECREF(uniobj);
+        return 0;
+    }
+    *fillcharloc = PyUnicode_READ_CHAR(uniobj, 0);
     Py_DECREF(uniobj);
     return 1;
 }
@@ -7255,17 +9118,20 @@
 {
     Py_ssize_t marg, left;
     Py_ssize_t width;
-    Py_UNICODE fillchar = ' ';
+    Py_UCS4 fillchar = ' ';
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
 
     if (!PyArg_ParseTuple(args, "n|O&:center", &width, convert_uc, &fillchar))
         return NULL;
 
-    if (self->length >= width && PyUnicode_CheckExact(self)) {
+    if (_PyUnicode_LENGTH(self) >= width && PyUnicode_CheckExact(self)) {
         Py_INCREF(self);
         return (PyObject*) self;
     }
 
-    marg = width - self->length;
+    marg = width - _PyUnicode_LENGTH(self);
     left = marg / 2 + (marg & width & 1);
 
     return (PyObject*) pad(self, left, marg - left, fillchar);
@@ -7297,8 +9163,8 @@
     Py_UNICODE *s1 = str1->str;
     Py_UNICODE *s2 = str2->str;
 
-    len1 = str1->length;
-    len2 = str2->length;
+    len1 = str1->_base._base.length;
+    len2 = str2->_base._base.length;
 
     while (len1 > 0 && len2 > 0) {
         Py_UNICODE c1, c2;
@@ -7323,27 +9189,29 @@
 
 #else
 
+/* This function assumes that str1 and str2 are readied by the caller. */
+
 static int
 unicode_compare(PyUnicodeObject *str1, PyUnicodeObject *str2)
 {
-    register Py_ssize_t len1, len2;
-
-    Py_UNICODE *s1 = str1->str;
-    Py_UNICODE *s2 = str2->str;
-
-    len1 = str1->length;
-    len2 = str2->length;
-
-    while (len1 > 0 && len2 > 0) {
-        Py_UNICODE c1, c2;
-
-        c1 = *s1++;
-        c2 = *s2++;
+    int kind1, kind2;
+    void *data1, *data2;
+    Py_ssize_t len1, len2, i;
+
+    kind1 = PyUnicode_KIND(str1);
+    kind2 = PyUnicode_KIND(str2);
+    data1 = PyUnicode_DATA(str1);
+    data2 = PyUnicode_DATA(str2);
+    len1 = PyUnicode_GET_LENGTH(str1);
+    len2 = PyUnicode_GET_LENGTH(str2);
+
+    for (i = 0; i < len1 && i < len2; ++i) {
+        Py_UCS4 c1, c2;
+        c1 = PyUnicode_READ(kind1, data1, i);
+        c2 = PyUnicode_READ(kind2, data2, i);
 
         if (c1 != c2)
             return (c1 < c2) ? -1 : 1;
-
-        len1--; len2--;
     }
 
     return (len1 < len2) ? -1 : (len1 != len2);
@@ -7354,9 +9222,13 @@
 int
 PyUnicode_Compare(PyObject *left, PyObject *right)
 {
-    if (PyUnicode_Check(left) && PyUnicode_Check(right))
+    if (PyUnicode_Check(left) && PyUnicode_Check(right)) {
+        if (PyUnicode_READY(left) == -1 ||
+            PyUnicode_READY(right) == -1)
+            return -1;
         return unicode_compare((PyUnicodeObject *)left,
                                (PyUnicodeObject *)right);
+    }
     PyErr_Format(PyExc_TypeError,
                  "Can't compare %.100s and %.100s",
                  left->ob_type->tp_name,
@@ -7367,17 +9239,23 @@
 int
 PyUnicode_CompareWithASCIIString(PyObject* uni, const char* str)
 {
-    int i;
-    Py_UNICODE *id;
+    Py_ssize_t i;
+    int kind;
+    void *data;
+    Py_UCS4 chr;
+
     assert(PyUnicode_Check(uni));
-    id = PyUnicode_AS_UNICODE(uni);
+    if (PyUnicode_READY(uni) == -1)
+        return -1;
+    kind = PyUnicode_KIND(uni);
+    data = PyUnicode_DATA(uni);
     /* Compare Unicode string and source character set string */
-    for (i = 0; id[i] && str[i]; i++)
-        if (id[i] != str[i])
-            return ((int)id[i] < (int)str[i]) ? -1 : 1;
+    for (i = 0; (chr = PyUnicode_READ(kind, data, i)) && str[i]; i++)
+        if (chr != str[i])
+            return (chr < (unsigned char)(str[i])) ? -1 : 1;
     /* This check keeps Python strings that end in '\0' from comparing equal
      to C strings identical up to that point. */
-    if (PyUnicode_GET_SIZE(uni) != i || id[i])
+    if (PyUnicode_GET_LENGTH(uni) != i || chr)
         return 1; /* uni is longer */
     if (str[i])
         return -1; /* str is longer */
@@ -7395,7 +9273,11 @@
 
     if (PyUnicode_Check(left) && PyUnicode_Check(right)) {
         PyObject *v;
-        if (PyUnicode_GET_SIZE(left) != PyUnicode_GET_SIZE(right)) {
+        if (PyUnicode_READY(left) == -1 ||
+            PyUnicode_READY(right) == -1)
+            return NULL;
+        if (PyUnicode_GET_LENGTH(left) != PyUnicode_GET_LENGTH(right) ||
+            PyUnicode_KIND(left) != PyUnicode_KIND(right)) {
             if (op == Py_EQ) {
                 Py_INCREF(Py_False);
                 return Py_False;
@@ -7446,6 +9328,9 @@
 PyUnicode_Contains(PyObject *container, PyObject *element)
 {
     PyObject *str, *sub;
+    int kind1, kind2, kind;
+    void *buf1, *buf2;
+    Py_ssize_t len1, len2;
     int result;
 
     /* Coerce the two arguments */
@@ -7456,18 +9341,59 @@
                      element->ob_type->tp_name);
         return -1;
     }
+    if (PyUnicode_READY(sub) == -1)
+        return -1;
 
     str = PyUnicode_FromObject(container);
-    if (!str) {
+    if (!str || PyUnicode_READY(container) == -1) {
         Py_DECREF(sub);
         return -1;
     }
 
-    result = stringlib_contains_obj(str, sub);
+    kind1 = PyUnicode_KIND(str);
+    kind2 = PyUnicode_KIND(sub);
+    kind = kind1 > kind2 ? kind1 : kind2;
+    buf1 = PyUnicode_DATA(str);
+    buf2 = PyUnicode_DATA(sub);
+    if (kind1 != kind)
+        buf1 = _PyUnicode_AsKind((PyObject*)str, kind);
+    if (!buf1) {
+        Py_DECREF(sub);
+        return -1;
+    }
+    if (kind2 != kind)
+        buf2 = _PyUnicode_AsKind((PyObject*)sub, kind);
+    if (!buf2) {
+        Py_DECREF(sub);
+        if (kind1 != kind) PyMem_Free(buf1);
+        return -1;
+    }
+    len1 = PyUnicode_GET_LENGTH(str);
+    len2 = PyUnicode_GET_LENGTH(sub);
+
+    switch(kind) {
+    case PyUnicode_1BYTE_KIND:
+        result = ucs1lib_find(buf1, len1, buf2, len2, 0) != -1;
+        break;
+    case PyUnicode_2BYTE_KIND:
+        result = ucs2lib_find(buf1, len1, buf2, len2, 0) != -1;
+        break;
+    case PyUnicode_4BYTE_KIND:
+        result = ucs4lib_find(buf1, len1, buf2, len2, 0) != -1;
+        break;
+    default:
+        result = -1;
+        assert(0);
+    }
 
     Py_DECREF(str);
     Py_DECREF(sub);
 
+    if (kind1 != kind)
+        PyMem_Free(buf1);
+    if (kind2 != kind)
+        PyMem_Free(buf2);
+
     return result;
 }
 
@@ -7476,36 +9402,46 @@
 PyObject *
 PyUnicode_Concat(PyObject *left, PyObject *right)
 {
-    PyUnicodeObject *u = NULL, *v = NULL, *w;
+    PyObject *u = NULL, *v = NULL, *w;
+    Py_UCS4 maxchar;
 
     /* Coerce the two arguments */
-    u = (PyUnicodeObject *)PyUnicode_FromObject(left);
+    u = PyUnicode_FromObject(left);
     if (u == NULL)
         goto onError;
-    v = (PyUnicodeObject *)PyUnicode_FromObject(right);
+    v = PyUnicode_FromObject(right);
     if (v == NULL)
         goto onError;
 
     /* Shortcuts */
-    if (v == unicode_empty) {
+    if (v == (PyObject*)unicode_empty) {
         Py_DECREF(v);
-        return (PyObject *)u;
-    }
-    if (u == unicode_empty) {
+        return u;
+    }
+    if (u == (PyObject*)unicode_empty) {
         Py_DECREF(u);
-        return (PyObject *)v;
-    }
+        return v;
+    }
+
+    if (PyUnicode_READY(u) == -1 || PyUnicode_READY(v) == -1)
+        goto onError;
+
+    maxchar = PyUnicode_MAX_CHAR_VALUE(u);
+    if (PyUnicode_MAX_CHAR_VALUE(v) > maxchar)
+        maxchar = PyUnicode_MAX_CHAR_VALUE(v);
 
     /* Concat the two Unicode strings */
-    w = _PyUnicode_New(u->length + v->length);
+    w = PyUnicode_New(
+        PyUnicode_GET_LENGTH(u) + PyUnicode_GET_LENGTH(v),
+        maxchar);
     if (w == NULL)
         goto onError;
-    Py_UNICODE_COPY(w->str, u->str, u->length);
-    Py_UNICODE_COPY(w->str + u->length, v->str, v->length);
-
+    PyUnicode_CopyCharacters(w, 0, u, 0, PyUnicode_GET_LENGTH(u));
+    PyUnicode_CopyCharacters(w, PyUnicode_GET_LENGTH(u), v, 0,
+                             PyUnicode_GET_LENGTH(v));
     Py_DECREF(u);
     Py_DECREF(v);
-    return (PyObject *)w;
+    return w;
 
   onError:
     Py_XDECREF(u);
@@ -7550,17 +9486,65 @@
     Py_ssize_t start = 0;
     Py_ssize_t end = PY_SSIZE_T_MAX;
     PyObject *result;
+    int kind1, kind2, kind;
+    void *buf1, *buf2;
+    Py_ssize_t len1, len2, iresult;
 
     if (!stringlib_parse_args_finds_unicode("count", args, &substring,
                                             &start, &end))
         return NULL;
 
-    ADJUST_INDICES(start, end, self->length);
-    result = PyLong_FromSsize_t(
-        stringlib_count(self->str + start, end - start,
-                        substring->str, substring->length,
-                        PY_SSIZE_T_MAX)
-        );
+    kind1 = PyUnicode_KIND(self);
+    kind2 = PyUnicode_KIND(substring);
+    kind = kind1 > kind2 ? kind1 : kind2;
+    buf1 = PyUnicode_DATA(self);
+    buf2 = PyUnicode_DATA(substring);
+    if (kind1 != kind)
+        buf1 = _PyUnicode_AsKind((PyObject*)self, kind);
+    if (!buf1) {
+        Py_DECREF(substring);
+        return NULL;
+    }
+    if (kind2 != kind)
+        buf2 = _PyUnicode_AsKind((PyObject*)substring, kind);
+    if (!buf2) {
+        Py_DECREF(substring);
+        if (kind1 != kind) PyMem_Free(buf1);
+        return NULL;
+    }
+    len1 = PyUnicode_GET_LENGTH(self);
+    len2 = PyUnicode_GET_LENGTH(substring);
+
+    ADJUST_INDICES(start, end, len1);
+    switch(kind) {
+    case PyUnicode_1BYTE_KIND:
+        iresult = ucs1lib_count(
+            ((Py_UCS1*)buf1) + start, end - start,
+            buf2, len2, PY_SSIZE_T_MAX
+            );
+        break;
+    case PyUnicode_2BYTE_KIND:
+        iresult = ucs2lib_count(
+            ((Py_UCS2*)buf1) + start, end - start,
+            buf2, len2, PY_SSIZE_T_MAX
+            );
+        break;
+    case PyUnicode_4BYTE_KIND:
+        iresult = ucs4lib_count(
+            ((Py_UCS4*)buf1) + start, end - start,
+            buf2, len2, PY_SSIZE_T_MAX
+            );
+        break;
+    default:
+        assert(0); iresult = 0;
+    }
+
+    result = PyLong_FromSsize_t(iresult);
+
+    if (kind1 != kind)
+        PyMem_Free(buf1);
+    if (kind2 != kind)
+        PyMem_Free(buf2);
 
     Py_DECREF(substring);
 
@@ -7603,18 +9587,21 @@
     Py_UNICODE *p;
     Py_UNICODE *q;
     Py_UNICODE *qe;
-    Py_ssize_t i, j, incr;
+    Py_ssize_t i, j, incr, wstr_length;
     PyUnicodeObject *u;
     int tabsize = 8;
 
     if (!PyArg_ParseTuple(args, "|i:expandtabs", &tabsize))
         return NULL;
 
+    if (PyUnicode_AsUnicodeAndSize((PyObject *)self, &wstr_length) == NULL)
+        return NULL;
+
     /* First pass: determine size of output string */
     i = 0; /* chars up to and including most recent \n or \r */
     j = 0; /* chars since most recent \n or \r (use in tab calculations) */
-    e = self->str + self->length; /* end of input */
-    for (p = self->str; p < e; p++)
+    e = _PyUnicode_WSTR(self) + wstr_length; /* end of input */
+    for (p = _PyUnicode_WSTR(self); p < e; p++)
         if (*p == '\t') {
             if (tabsize > 0) {
                 incr = tabsize - (j % tabsize); /* cannot overflow */
@@ -7644,10 +9631,10 @@
         return NULL;
 
     j = 0; /* same as in first pass */
-    q = u->str; /* next output char */
-    qe = u->str + u->length; /* end of output */
-
-    for (p = self->str; p < e; p++)
+    q = _PyUnicode_WSTR(u); /* next output char */
+    qe = _PyUnicode_WSTR(u) + PyUnicode_GET_SIZE(u); /* end of output */
+
+    for (p = _PyUnicode_WSTR(self); p < e; p++)
         if (*p == '\t') {
             if (tabsize > 0) {
                 i = tabsize - (j % tabsize);
@@ -7668,6 +9655,10 @@
                 j = 0;
         }
 
+    if (PyUnicode_READY(u) == -1) {
+        Py_DECREF(u);
+        return NULL;
+    }
     return (PyObject*) u;
 
   overflow2:
@@ -7687,7 +9678,7 @@
 Return -1 on failure.");
 
 static PyObject *
-unicode_find(PyUnicodeObject *self, PyObject *args)
+unicode_find(PyObject *self, PyObject *args)
 {
     PyUnicodeObject *substring;
     Py_ssize_t start;
@@ -7698,26 +9689,38 @@
                                             &start, &end))
         return NULL;
 
-    result = stringlib_find_slice(
-        PyUnicode_AS_UNICODE(self), PyUnicode_GET_SIZE(self),
-        PyUnicode_AS_UNICODE(substring), PyUnicode_GET_SIZE(substring),
-        start, end
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+    if (PyUnicode_READY(substring) == -1)
+        return NULL;
+
+    result = any_find_slice(
+        ucs1lib_find_slice, ucs2lib_find_slice, ucs4lib_find_slice,
+        self, (PyObject*)substring, start, end
         );
 
     Py_DECREF(substring);
 
+    if (result == -2)
+        return NULL;
+
     return PyLong_FromSsize_t(result);
 }
 
 static PyObject *
 unicode_getitem(PyUnicodeObject *self, Py_ssize_t index)
 {
-    if (index < 0 || index >= self->length) {
+    Py_UCS4 ch;
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+    if (index < 0 || index >= _PyUnicode_LENGTH(self)) {
         PyErr_SetString(PyExc_IndexError, "string index out of range");
         return NULL;
     }
 
-    return (PyObject*) PyUnicode_FromUnicode(&self->str[index], 1);
+    ch = PyUnicode_READ(PyUnicode_KIND(self), PyUnicode_DATA(self), index);
+    return PyUnicode_FromOrdinal(ch);
 }
 
 /* Believe it or not, this produces the same value for ASCII strings
@@ -7726,22 +9729,48 @@
 unicode_hash(PyUnicodeObject *self)
 {
     Py_ssize_t len;
-    Py_UNICODE *p;
     Py_uhash_t x;
 
-    if (self->hash != -1)
-        return self->hash;
-    len = Py_SIZE(self);
-    p = self->str;
-    x = (Py_uhash_t)*p << 7;
-    while (--len >= 0)
-        x = (1000003U*x) ^ (Py_uhash_t)*p++;
-    x ^= (Py_uhash_t)Py_SIZE(self);
+    if (_PyUnicode_HASH(self) != -1)
+        return _PyUnicode_HASH(self);
+    if (PyUnicode_READY(self) == -1)
+        return -1;
+    len = PyUnicode_GET_LENGTH(self);
+
+    /* The hash function as a macro, gets expanded three times below. */
+#define HASH(P) \
+    x = (Py_uhash_t)*P << 7; \
+    while (--len >= 0) \
+        x = (1000003*x) ^ (Py_uhash_t)*P++;
+
+    switch (PyUnicode_KIND(self)) {
+    case PyUnicode_1BYTE_KIND: {
+        const unsigned char *c = PyUnicode_1BYTE_DATA(self);
+        HASH(c);
+        break;
+    }
+    case PyUnicode_2BYTE_KIND: {
+        const Py_UCS2 *s = PyUnicode_2BYTE_DATA(self);
+        HASH(s);
+        break;
+    }
+    default: {
+        Py_UCS4 *l;
+        assert(PyUnicode_KIND(self) == PyUnicode_4BYTE_KIND &&
+               "Impossible switch case in unicode_hash");
+        l = PyUnicode_4BYTE_DATA(self);
+        HASH(l);
+        break;
+    }
+    }
+    x ^= (Py_uhash_t)PyUnicode_GET_LENGTH(self);
+
     if (x == -1)
         x = -2;
-    self->hash = x;
+    _PyUnicode_HASH(self) = x;
     return x;
 }
+#undef HASH
 
 PyDoc_STRVAR(index__doc__,
              "S.index(sub[, start[, end]]) -> int\n\
@@ -7749,7 +9778,7 @@
 Like S.find() but raise ValueError when the substring is not found.");
 
 static PyObject *
-unicode_index(PyUnicodeObject *self, PyObject *args)
+unicode_index(PyObject *self, PyObject *args)
 {
     Py_ssize_t result;
     PyUnicodeObject *substring;
@@ -7760,14 +9789,21 @@
                                             &start, &end))
         return NULL;
 
-    result = stringlib_find_slice(
-        PyUnicode_AS_UNICODE(self), PyUnicode_GET_SIZE(self),
-        PyUnicode_AS_UNICODE(substring), PyUnicode_GET_SIZE(substring),
-        start, end
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+    if (PyUnicode_READY(substring) == -1)
+        return NULL;
+
+    result = any_find_slice(
+        ucs1lib_find_slice, ucs2lib_find_slice, ucs4lib_find_slice,
+        self, (PyObject*)substring, start, end
         );
 
     Py_DECREF(substring);
 
+    if (result == -2)
+        return NULL;
+
     if (result < 0) {
         PyErr_SetString(PyExc_ValueError, "substring not found");
         return NULL;
@@ -7785,22 +9821,29 @@
 static PyObject*
 unicode_islower(PyUnicodeObject *self)
 {
-    register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
-    register const Py_UNICODE *e;
+    Py_ssize_t i, length;
+    int kind;
+    void *data;
     int cased;
 
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+    length = PyUnicode_GET_LENGTH(self);
+    kind = PyUnicode_KIND(self);
+    data = PyUnicode_DATA(self);
+
     /* Shortcut for single character strings */
-    if (PyUnicode_GET_SIZE(self) == 1)
-        return PyBool_FromLong(Py_UNICODE_ISLOWER(*p));
+    if (length == 1)
+        return PyBool_FromLong(
+            Py_UNICODE_ISLOWER(PyUnicode_READ(kind, data, 0)));
 
     /* Special case for empty strings */
-    if (PyUnicode_GET_SIZE(self) == 0)
+    if (length == 0)
         return PyBool_FromLong(0);
 
-    e = p + PyUnicode_GET_SIZE(self);
     cased = 0;
-    while (p < e) {
-        const Py_UCS4 ch = _Py_UNICODE_NEXT(p, e);
+    for (i = 0; i < length; i++) {
+        const Py_UCS4 ch = PyUnicode_READ(kind, data, i);
 
         if (Py_UNICODE_ISUPPER(ch) || Py_UNICODE_ISTITLE(ch))
             return PyBool_FromLong(0);
@@ -7819,22 +9862,29 @@
 static PyObject*
 unicode_isupper(PyUnicodeObject *self)
 {
-    register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
-    register const Py_UNICODE *e;
+    Py_ssize_t i, length;
+    int kind;
+    void *data;
     int cased;
 
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+    length = PyUnicode_GET_LENGTH(self);
+    kind = PyUnicode_KIND(self);
+    data = PyUnicode_DATA(self);
+
     /* Shortcut for single character strings */
-    if (PyUnicode_GET_SIZE(self) == 1)
-        return PyBool_FromLong(Py_UNICODE_ISUPPER(*p) != 0);
+    if (length == 1)
+        return PyBool_FromLong(
+            Py_UNICODE_ISUPPER(PyUnicode_READ(kind, data, 0)) != 0);
 
     /* Special case for empty strings */
-    if (PyUnicode_GET_SIZE(self) == 0)
+    if (length == 0)
         return PyBool_FromLong(0);
 
-    e = p + PyUnicode_GET_SIZE(self);
     cased = 0;
-    while (p < e) {
-        const Py_UCS4 ch = _Py_UNICODE_NEXT(p, e);
+    for (i = 0; i < length; i++) {
+        const Py_UCS4 ch = PyUnicode_READ(kind, data, i);
 
         if (Py_UNICODE_ISLOWER(ch) || Py_UNICODE_ISTITLE(ch))
             return PyBool_FromLong(0);
@@ -7855,24 +9905,32 @@
 static PyObject*
 unicode_istitle(PyUnicodeObject *self)
 {
-    register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
-    register const Py_UNICODE *e;
+    Py_ssize_t i, length;
+    int kind;
+    void *data;
     int cased, previous_is_cased;
 
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+    length = PyUnicode_GET_LENGTH(self);
+    kind = PyUnicode_KIND(self);
+    data = PyUnicode_DATA(self);
+
     /* Shortcut for single character strings */
-    if (PyUnicode_GET_SIZE(self) == 1)
-        return PyBool_FromLong((Py_UNICODE_ISTITLE(*p) != 0) ||
-                               (Py_UNICODE_ISUPPER(*p) != 0));
+    if (length == 1) {
+        Py_UCS4 ch = PyUnicode_READ(kind, data, 0);
+        return PyBool_FromLong((Py_UNICODE_ISTITLE(ch) != 0) ||
+                               (Py_UNICODE_ISUPPER(ch) != 0));
+    }
 
     /* Special case for empty strings */
-    if (PyUnicode_GET_SIZE(self) == 0)
+    if (length == 0)
         return PyBool_FromLong(0);
 
-    e = p + PyUnicode_GET_SIZE(self);
     cased = 0;
     previous_is_cased = 0;
-    while (p < e) {
-        const Py_UCS4 ch = _Py_UNICODE_NEXT(p, e);
+    for (i = 0; i < length; i++) {
+        const Py_UCS4 ch = PyUnicode_READ(kind, data, i);
 
         if (Py_UNICODE_ISUPPER(ch) || Py_UNICODE_ISTITLE(ch)) {
             if (previous_is_cased)
@@ -7901,21 +9959,27 @@
 static PyObject*
 unicode_isspace(PyUnicodeObject *self)
 {
-    register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
-    register const Py_UNICODE *e;
+    Py_ssize_t i, length;
+    int kind;
+    void *data;
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+    length = PyUnicode_GET_LENGTH(self);
+    kind = PyUnicode_KIND(self);
+    data = PyUnicode_DATA(self);
 
     /* Shortcut for single character strings */
-    if (PyUnicode_GET_SIZE(self) == 1 &&
-        Py_UNICODE_ISSPACE(*p))
-        return PyBool_FromLong(1);
+    if (length == 1)
+        return PyBool_FromLong(
+            Py_UNICODE_ISSPACE(PyUnicode_READ(kind, data, 0)));
 
     /* Special case for empty strings */
-    if (PyUnicode_GET_SIZE(self) == 0)
+    if (length == 0)
         return PyBool_FromLong(0);
 
-    e = p + PyUnicode_GET_SIZE(self);
-    while (p < e) {
-        const Py_UCS4 ch = _Py_UNICODE_NEXT(p, e);
+    for (i = 0; i < length; i++) {
+        const Py_UCS4 ch = PyUnicode_READ(kind, data, i);
         if (!Py_UNICODE_ISSPACE(ch))
             return PyBool_FromLong(0);
     }
@@ -7931,21 +9995,27 @@
 static PyObject*
 unicode_isalpha(PyUnicodeObject *self)
 {
-    register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
-    register const Py_UNICODE *e;
+    Py_ssize_t i, length;
+    int kind;
+    void *data;
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+    length = PyUnicode_GET_LENGTH(self);
+    kind = PyUnicode_KIND(self);
+    data = PyUnicode_DATA(self);
 
     /* Shortcut for single character strings */
-    if (PyUnicode_GET_SIZE(self) == 1 &&
-        Py_UNICODE_ISALPHA(*p))
-        return PyBool_FromLong(1);
+    if (length == 1)
+        return PyBool_FromLong(
+            Py_UNICODE_ISALPHA(PyUnicode_READ(kind, data, 0)));
 
     /* Special case for empty strings */
-    if (PyUnicode_GET_SIZE(self) == 0)
+    if (length == 0)
         return PyBool_FromLong(0);
 
-    e = p + PyUnicode_GET_SIZE(self);
-    while (p < e) {
-        if (!Py_UNICODE_ISALPHA(_Py_UNICODE_NEXT(p, e)))
+    for (i = 0; i < length; i++) {
+        if (!Py_UNICODE_ISALPHA(PyUnicode_READ(kind, data, i)))
             return PyBool_FromLong(0);
     }
     return PyBool_FromLong(1);
@@ -7960,21 +10030,29 @@
 static PyObject*
 unicode_isalnum(PyUnicodeObject *self)
 {
-    register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
-    register const Py_UNICODE *e;
+    int kind;
+    void *data;
+    Py_ssize_t len, i;
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+
+    kind = PyUnicode_KIND(self);
+    data = PyUnicode_DATA(self);
+    len = PyUnicode_GET_LENGTH(self);
 
     /* Shortcut for single character strings */
-    if (PyUnicode_GET_SIZE(self) == 1 &&
-        Py_UNICODE_ISALNUM(*p))
-        return PyBool_FromLong(1);
+    if (len == 1) {
+        const Py_UCS4 ch = PyUnicode_READ(kind, data, 0);
+        return PyBool_FromLong(Py_UNICODE_ISALNUM(ch));
+    }
 
     /* Special case for empty strings */
-    if (PyUnicode_GET_SIZE(self) == 0)
+    if (len == 0)
         return PyBool_FromLong(0);
 
-    e = p + PyUnicode_GET_SIZE(self);
-    while (p < e) {
-        const Py_UCS4 ch = _Py_UNICODE_NEXT(p, e);
+    for (i = 0; i < len; i++) {
+        const Py_UCS4 ch = PyUnicode_READ(kind, data, i);
         if (!Py_UNICODE_ISALNUM(ch))
             return PyBool_FromLong(0);
     }
@@ -7990,21 +10068,27 @@
 static PyObject*
 unicode_isdecimal(PyUnicodeObject *self)
 {
-    register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
-    register const Py_UNICODE *e;
+    Py_ssize_t i, length;
+    int kind;
+    void *data;
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+    length = PyUnicode_GET_LENGTH(self);
+    kind = PyUnicode_KIND(self);
+    data = PyUnicode_DATA(self);
 
     /* Shortcut for single character strings */
-    if (PyUnicode_GET_SIZE(self) == 1 &&
-        Py_UNICODE_ISDECIMAL(*p))
-        return PyBool_FromLong(1);
+    if (length == 1)
+        return PyBool_FromLong(
+            Py_UNICODE_ISDECIMAL(PyUnicode_READ(kind, data, 0)));
 
     /* Special case for empty strings */
-    if (PyUnicode_GET_SIZE(self) == 0)
+    if (length == 0)
         return PyBool_FromLong(0);
 
-    e = p + PyUnicode_GET_SIZE(self);
-    while (p < e) {
-        if (!Py_UNICODE_ISDECIMAL(_Py_UNICODE_NEXT(p, e)))
+    for (i = 0; i < length; i++) {
+        if (!Py_UNICODE_ISDECIMAL(PyUnicode_READ(kind, data, i)))
             return PyBool_FromLong(0);
     }
     return PyBool_FromLong(1);
@@ -8019,21 +10103,28 @@
 static PyObject*
 unicode_isdigit(PyUnicodeObject *self)
 {
-    register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
-    register const Py_UNICODE *e;
+    Py_ssize_t i, length;
+    int kind;
+    void *data;
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+    length = PyUnicode_GET_LENGTH(self);
+    kind = PyUnicode_KIND(self);
+    data = PyUnicode_DATA(self);
 
     /* Shortcut for single character strings */
-    if (PyUnicode_GET_SIZE(self) == 1 &&
-        Py_UNICODE_ISDIGIT(*p))
-        return PyBool_FromLong(1);
+    if (length == 1) {
+        const Py_UCS4 ch = PyUnicode_READ(kind, data, 0);
+        return PyBool_FromLong(Py_UNICODE_ISDIGIT(ch));
+    }
 
     /* Special case for empty strings */
-    if (PyUnicode_GET_SIZE(self) == 0)
+    if (length == 0)
         return PyBool_FromLong(0);
 
-    e = p + PyUnicode_GET_SIZE(self);
-    while (p < e) {
-        if (!Py_UNICODE_ISDIGIT(_Py_UNICODE_NEXT(p, e)))
+    for (i = 0; i < length; i++) {
+        if (!Py_UNICODE_ISDIGIT(PyUnicode_READ(kind, data, i)))
             return PyBool_FromLong(0);
     }
     return PyBool_FromLong(1);
@@ -8048,21 +10139,27 @@
 static PyObject*
 unicode_isnumeric(PyUnicodeObject *self)
 {
-    register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
-    register const Py_UNICODE *e;
+    Py_ssize_t i, length;
+    int kind;
+    void *data;
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+    length = PyUnicode_GET_LENGTH(self);
+    kind = PyUnicode_KIND(self);
+    data = PyUnicode_DATA(self);
 
     /* Shortcut for single character strings */
-    if (PyUnicode_GET_SIZE(self) == 1 &&
-        Py_UNICODE_ISNUMERIC(*p))
-        return PyBool_FromLong(1);
+    if (length == 1)
+        return PyBool_FromLong(
+            Py_UNICODE_ISNUMERIC(PyUnicode_READ(kind, data, 0)));
 
     /* Special case for empty strings */
-    if (PyUnicode_GET_SIZE(self) == 0)
+    if (length == 0)
         return PyBool_FromLong(0);
 
-    e = p + PyUnicode_GET_SIZE(self);
-    while (p < e) {
-        if (!Py_UNICODE_ISNUMERIC(_Py_UNICODE_NEXT(p, e)))
+    for (i = 0; i < length; i++) {
+        if (!Py_UNICODE_ISNUMERIC(PyUnicode_READ(kind, data, i)))
             return PyBool_FromLong(0);
     }
     return PyBool_FromLong(1);
@@ -8071,13 +10168,21 @@
 int
 PyUnicode_IsIdentifier(PyObject *self)
 {
-    const Py_UNICODE *p = PyUnicode_AS_UNICODE((PyUnicodeObject*)self);
-    const Py_UNICODE *e;
+    int kind;
+    void *data;
+    Py_ssize_t i;
     Py_UCS4 first;
 
+    if (PyUnicode_READY(self) == -1) {
+        Py_FatalError("identifier not ready");
+        return 0;
+    }
+
     /* Special case for empty strings */
-    if (PyUnicode_GET_SIZE(self) == 0)
+    if (PyUnicode_GET_LENGTH(self) == 0)
         return 0;
+    kind = PyUnicode_KIND(self);
+    data = PyUnicode_DATA(self);
 
     /* PEP 3131 says that the first character must be in
        XID_Start and subsequent characters in XID_Continue,
@@ -8087,13 +10192,12 @@
        definition of XID_Start and XID_Continue, it is sufficient
        to check just for these, except that _ must be allowed
        as starting an identifier.  */
-    e = p + PyUnicode_GET_SIZE(self);
-    first = _Py_UNICODE_NEXT(p, e);
+    first = PyUnicode_READ(kind, data, 0);
     if (!_PyUnicode_IsXidStart(first) && first != 0x5F /* LOW LINE */)
         return 0;
 
-    while (p < e)
-        if (!_PyUnicode_IsXidContinue(_Py_UNICODE_NEXT(p, e)))
+    for (i = 0; i < PyUnicode_GET_LENGTH(self); i++)
+        if (!_PyUnicode_IsXidContinue(PyUnicode_READ(kind, data, i)))
             return 0;
     return 1;
 }
@@ -8119,17 +10223,23 @@
 static PyObject*
 unicode_isprintable(PyObject *self)
 {
-    register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
-    register const Py_UNICODE *e;
+    Py_ssize_t i, length;
+    int kind;
+    void *data;
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+    length = PyUnicode_GET_LENGTH(self);
+    kind = PyUnicode_KIND(self);
+    data = PyUnicode_DATA(self);
 
     /* Shortcut for single character strings */
-    if (PyUnicode_GET_SIZE(self) == 1 && Py_UNICODE_ISPRINTABLE(*p)) {
-        Py_RETURN_TRUE;
-    }
-
-    e = p + PyUnicode_GET_SIZE(self);
-    while (p < e) {
-        if (!Py_UNICODE_ISPRINTABLE(_Py_UNICODE_NEXT(p, e))) {
+    if (length == 1)
+        return PyBool_FromLong(
+            Py_UNICODE_ISPRINTABLE(PyUnicode_READ(kind, data, 0)));
+
+    for (i = 0; i < length; i++) {
+        if (!Py_UNICODE_ISPRINTABLE(PyUnicode_READ(kind, data, i))) {
             Py_RETURN_FALSE;
         }
     }
@@ -8151,7 +10261,9 @@
 static Py_ssize_t
 unicode_length(PyUnicodeObject *self)
 {
-    return self->length;
+    if (PyUnicode_READY(self) == -1)
+        return -1;
+    return PyUnicode_GET_LENGTH(self);
 }
 
 PyDoc_STRVAR(ljust__doc__,
@@ -8164,17 +10276,20 @@
 unicode_ljust(PyUnicodeObject *self, PyObject *args)
 {
     Py_ssize_t width;
-    Py_UNICODE fillchar = ' ';
+    Py_UCS4 fillchar = ' ';
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
 
     if (!PyArg_ParseTuple(args, "n|O&:ljust", &width, convert_uc, &fillchar))
         return NULL;
 
-    if (self->length >= width && PyUnicode_CheckExact(self)) {
+    if (_PyUnicode_LENGTH(self) >= width && PyUnicode_CheckExact(self)) {
         Py_INCREF(self);
         return (PyObject*) self;
     }
 
-    return (PyObject*) pad(self, 0, width - self->length, fillchar);
+    return (PyObject*) pad(self, 0, width - _PyUnicode_LENGTH(self), fillchar);
 }
 
 PyDoc_STRVAR(lower__doc__,
@@ -8201,17 +10316,25 @@
 PyObject *
 _PyUnicode_XStrip(PyUnicodeObject *self, int striptype, PyObject *sepobj)
 {
-    Py_UNICODE *s = PyUnicode_AS_UNICODE(self);
-    Py_ssize_t len = PyUnicode_GET_SIZE(self);
-    Py_UNICODE *sep = PyUnicode_AS_UNICODE(sepobj);
-    Py_ssize_t seplen = PyUnicode_GET_SIZE(sepobj);
-    Py_ssize_t i, j;
-
-    BLOOM_MASK sepmask = make_bloom_mask(sep, seplen);
+    void *data;
+    int kind;
+    Py_ssize_t i, j, len;
+    BLOOM_MASK sepmask;
+
+    if (PyUnicode_READY(self) == -1 || PyUnicode_READY(sepobj) == -1)
+        return NULL;
+
+    kind = PyUnicode_KIND(self);
+    data = PyUnicode_DATA(self);
+    len = PyUnicode_GET_LENGTH(self);
+    sepmask = make_bloom_mask(PyUnicode_KIND(sepobj),
+                              PyUnicode_DATA(sepobj),
+                              PyUnicode_GET_LENGTH(sepobj));
 
     i = 0;
     if (striptype != RIGHTSTRIP) {
-        while (i < len && BLOOM_MEMBER(sepmask, s[i], sep, seplen)) {
+        while (i < len &&
+               BLOOM_MEMBER(sepmask, PyUnicode_READ(kind, data, i), sepobj)) {
             i++;
         }
     }
@@ -8220,7 +10343,8 @@
     if (striptype != LEFTSTRIP) {
         do {
             j--;
-        } while (j >= i && BLOOM_MEMBER(sepmask, s[j], sep, seplen));
+        } while (j >= i &&
+                 BLOOM_MEMBER(sepmask, PyUnicode_READ(kind, data, j), sepobj));
         j++;
     }
 
@@ -8229,19 +10353,85 @@
         return (PyObject*)self;
     }
     else
-        return PyUnicode_FromUnicode(s+i, j-i);
-}
-
+        return PyUnicode_Substring((PyObject*)self, i, j);
+}
+
+/* Assumes an already ready self string. */
+
+static PyObject *
+substring(PyUnicodeObject *self, Py_ssize_t start, Py_ssize_t len)
+{
+    const int kind = PyUnicode_KIND(self);
+    void *data = PyUnicode_DATA(self);
+    Py_UCS4 maxchar = 0;
+    Py_ssize_t i;
+    PyObject *unicode;
+
+    if (start < 0 || len < 0 || (start + len) > PyUnicode_GET_LENGTH(self)) {
+        PyErr_BadInternalCall();
+        return NULL;
+    }
+
+    if (len == PyUnicode_GET_LENGTH(self) && PyUnicode_CheckExact(self)) {
+        Py_INCREF(self);
+        return (PyObject*)self;
+    }
+
+    for (i = 0; i < len; ++i) {
+        const Py_UCS4 ch = PyUnicode_READ(kind, data, start + i);
+        if (ch > maxchar)
+            maxchar = ch;
+    }
+
+    unicode = PyUnicode_New(len, maxchar);
+    if (unicode == NULL)
+        return NULL;
+    PyUnicode_CopyCharacters(unicode, 0,
+                             (PyObject*)self, start, len);
+    return unicode;
+}
+
+PyObject*
+PyUnicode_Substring(PyObject *self, Py_ssize_t start, Py_ssize_t end)
+{
+    unsigned char *data;
+    int kind;
+
+    if (start == 0 && end == PyUnicode_GET_LENGTH(self)
+        && PyUnicode_CheckExact(self))
+    {
+        Py_INCREF(self);
+        return (PyObject *)self;
+    }
+
+    if ((end - start) == 1)
+        return unicode_getitem((PyUnicodeObject*)self, start);
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+    kind = PyUnicode_KIND(self);
+    data = PyUnicode_1BYTE_DATA(self);
+    return PyUnicode_FromKindAndData(kind, data + PyUnicode_KIND_SIZE(kind, start),
+                                     end-start);
+}
 
 static PyObject *
 do_strip(PyUnicodeObject *self, int striptype)
 {
-    Py_UNICODE *s = PyUnicode_AS_UNICODE(self);
-    Py_ssize_t len = PyUnicode_GET_SIZE(self), i, j;
+    int kind;
+    void *data;
+    Py_ssize_t len, i, j;
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+
+    kind = PyUnicode_KIND(self);
+    data = PyUnicode_DATA(self);
+    len = PyUnicode_GET_LENGTH(self);
 
     i = 0;
     if (striptype != RIGHTSTRIP) {
-        while (i < len && Py_UNICODE_ISSPACE(s[i])) {
+        while (i < len && Py_UNICODE_ISSPACE(PyUnicode_READ(kind, data, i))) {
             i++;
         }
     }
@@ -8250,7 +10440,7 @@
     if (striptype != LEFTSTRIP) {
         do {
             j--;
-        } while (j >= i && Py_UNICODE_ISSPACE(s[j]));
+        } while (j >= i && Py_UNICODE_ISSPACE(PyUnicode_READ(kind, data, j)));
         j++;
     }
 
@@ -8259,7 +10449,7 @@
         return (PyObject*)self;
     }
     else
-        return PyUnicode_FromUnicode(s+i, j-i);
+        return substring(self, i, j-i);
 }
 
 
@@ -8339,9 +10529,8 @@
 unicode_repeat(PyUnicodeObject *str, Py_ssize_t len)
 {
     PyUnicodeObject *u;
-    Py_UNICODE *p;
-    Py_ssize_t nchars;
-    size_t nbytes;
+    Py_ssize_t nchars, n;
+    size_t nbytes, char_size;
 
     if (len < 1) {
         Py_INCREF(unicode_empty);
@@ -8354,35 +10543,46 @@
         return (PyObject*) str;
     }
 
+    if (PyUnicode_READY(str) == -1)
+        return NULL;
+
     /* ensure # of chars needed doesn't overflow int and # of bytes
      * needed doesn't overflow size_t
      */
-    nchars = len * str->length;
-    if (nchars / len != str->length) {
+    nchars = len * PyUnicode_GET_LENGTH(str);
+    if (nchars / len != PyUnicode_GET_LENGTH(str)) {
         PyErr_SetString(PyExc_OverflowError,
                         "repeated string is too long");
         return NULL;
     }
-    nbytes = (nchars + 1) * sizeof(Py_UNICODE);
-    if (nbytes / sizeof(Py_UNICODE) != (size_t)(nchars + 1)) {
+    char_size = PyUnicode_CHARACTER_SIZE(str);
+    nbytes = (nchars + 1) * char_size;
+    if (nbytes / char_size != (size_t)(nchars + 1)) {
         PyErr_SetString(PyExc_OverflowError,
                         "repeated string is too long");
         return NULL;
     }
-    u = _PyUnicode_New(nchars);
+    u = (PyUnicodeObject *)PyUnicode_New(nchars, PyUnicode_MAX_CHAR_VALUE(str));
     if (!u)
         return NULL;
 
-    p = u->str;
-
-    if (str->length == 1) {
-        Py_UNICODE_FILL(p, str->str[0], len);
-    } else {
-        Py_ssize_t done = str->length; /* number of characters copied this far */
-        Py_UNICODE_COPY(p, str->str, str->length);
+    if (PyUnicode_GET_LENGTH(str) == 1) {
+        const int kind = PyUnicode_KIND(str);
+        const Py_UCS4 fill_char = PyUnicode_READ(kind, PyUnicode_DATA(str), 0);
+        void *to = PyUnicode_DATA(u);
+        for (n = 0; n < len; ++n)
+            PyUnicode_WRITE(kind, to, n, fill_char);
+    }
+    else {
+        /* number of characters copied this far */
+        Py_ssize_t done = PyUnicode_GET_LENGTH(str);
+        const Py_ssize_t char_size = PyUnicode_CHARACTER_SIZE(str);
+        char *to = (char *) PyUnicode_DATA(u);
+        Py_MEMCPY(to, PyUnicode_DATA(str),
+                  PyUnicode_GET_LENGTH(str) * char_size);
         while (done < nchars) {
-            Py_ssize_t n = (done <= nchars-done) ? done : nchars-done;
-            Py_UNICODE_COPY(p+done, p, n);
+            n = (done <= nchars-done) ? done : nchars-done;
+            Py_MEMCPY(to + (done * char_size), to, n * char_size);
             done += n;
         }
     }
@@ -8402,23 +10602,20 @@
     PyObject *result;
 
     self = PyUnicode_FromObject(obj);
-    if (self == NULL)
+    if (self == NULL || PyUnicode_READY(obj) == -1)
         return NULL;
     str1 = PyUnicode_FromObject(subobj);
-    if (str1 == NULL) {
+    if (str1 == NULL || PyUnicode_READY(obj) == -1) {
         Py_DECREF(self);
         return NULL;
     }
     str2 = PyUnicode_FromObject(replobj);
-    if (str2 == NULL) {
+    if (str2 == NULL || PyUnicode_READY(obj)) {
         Py_DECREF(self);
         Py_DECREF(str1);
         return NULL;
     }
-    result = replace((PyUnicodeObject *)self,
-                     (PyUnicodeObject *)str1,
-                     (PyUnicodeObject *)str2,
-                     maxcount);
+    result = replace(self, str1, str2, maxcount);
     Py_DECREF(self);
     Py_DECREF(str1);
     Py_DECREF(str2);
@@ -8433,20 +10630,22 @@
 given, only the first count occurrences are replaced.");
 
 static PyObject*
-unicode_replace(PyUnicodeObject *self, PyObject *args)
-{
-    PyUnicodeObject *str1;
-    PyUnicodeObject *str2;
+unicode_replace(PyObject *self, PyObject *args)
+{
+    PyObject *str1;
+    PyObject *str2;
     Py_ssize_t maxcount = -1;
     PyObject *result;
 
     if (!PyArg_ParseTuple(args, "OO|n:replace", &str1, &str2, &maxcount))
         return NULL;
-    str1 = (PyUnicodeObject *)PyUnicode_FromObject((PyObject *)str1);
-    if (str1 == NULL)
-        return NULL;
-    str2 = (PyUnicodeObject *)PyUnicode_FromObject((PyObject *)str2);
-    if (str2 == NULL) {
+    if (!PyUnicode_READY(self) == -1)
+        return NULL;
+    str1 = PyUnicode_FromObject(str1);
+    if (str1 == NULL || PyUnicode_READY(str1) == -1)
+        return NULL;
+    str2 = PyUnicode_FromObject(str2);
+    if (str2 == NULL || PyUnicode_READY(str1) == -1) {
         Py_DECREF(str1);
         return NULL;
     }
@@ -8462,148 +10661,149 @@
 unicode_repr(PyObject *unicode)
 {
     PyObject *repr;
-    Py_UNICODE *p;
-    Py_UNICODE *s = PyUnicode_AS_UNICODE(unicode);
-    Py_ssize_t size = PyUnicode_GET_SIZE(unicode);
-
-    /* XXX(nnorwitz): rather than over-allocating, it would be
-       better to choose a different scheme.  Perhaps scan the
-       first N-chars of the string and allocate based on that size.
-    */
-    /* Initial allocation is based on the longest-possible unichr
-       escape.
-
-       In wide (UTF-32) builds '\U00xxxxxx' is 10 chars per source
-       unichr, so in this case it's the longest unichr escape. In
-       narrow (UTF-16) builds this is five chars per source unichr
-       since there are two unichrs in the surrogate pair, so in narrow
-       (UTF-16) builds it's not the longest unichr escape.
-
-       In wide or narrow builds '\uxxxx' is 6 chars per source unichr,
-       so in the narrow (UTF-16) build case it's the longest unichr
-       escape.
-    */
-
-    repr = PyUnicode_FromUnicode(NULL,
-                                 2 /* quotes */
-#ifdef Py_UNICODE_WIDE
-                                 + 10*size
-#else
-                                 + 6*size
-#endif
-                                 + 1);
+    Py_ssize_t isize;
+    Py_ssize_t osize, squote, dquote, i, o;
+    Py_UCS4 max, quote;
+    int ikind, okind;
+    void *idata, *odata;
+
+    if (PyUnicode_READY(unicode) == -1)
+        return NULL;
+
+    isize = PyUnicode_GET_LENGTH(unicode);
+    idata = PyUnicode_DATA(unicode);
+
+    /* Compute length of output, quote characters, and
+       maximum character */
+    osize = 2; /* quotes */
+    max = 127;
+    squote = dquote = 0;
+    ikind = PyUnicode_KIND(unicode);
+    for (i = 0; i < isize; i++) {
+        Py_UCS4 ch = PyUnicode_READ(ikind, idata, i);
+        switch (ch) {
+        case '\'': squote++; osize++; break;
+        case '"':  dquote++; osize++; break;
+        case '\\': case '\t': case '\r': case '\n':
+            osize += 2; break;
+        default:
+            /* Fast-path ASCII */
+            if (ch < ' ' || ch == 0x7f)
+                osize += 4; /* \xHH */
+            else if (ch < 0x7f)
+                osize++;
+            else if (Py_UNICODE_ISPRINTABLE(ch)) {
+                osize++;
+                max = ch > max ? ch : max;
+            }
+            else if (ch < 0x100)
+                osize += 4; /* \xHH */
+            else if (ch < 0x10000)
+                osize += 6; /* \uHHHH */
+            else
+                osize += 10; /* \uHHHHHHHH */
+        }
+    }
+
+    quote = '\'';
+    if (squote) {
+        if (dquote)
+            /* Both squote and dquote present. Use squote,
+               and escape them */
+            osize += squote;
+        else
+            quote = '"';
+    }
+
+    repr = PyUnicode_New(osize, max);
     if (repr == NULL)
         return NULL;
-
-    p = PyUnicode_AS_UNICODE(repr);
-
-    /* Add quote */
-    *p++ = (findchar(s, size, '\'') &&
-            !findchar(s, size, '"')) ? '"' : '\'';
-    while (size-- > 0) {
-        Py_UNICODE ch = *s++;
+    okind = PyUnicode_KIND(repr);
+    odata = PyUnicode_DATA(repr);
+
+    PyUnicode_WRITE(okind, odata, 0, quote);
+    PyUnicode_WRITE(okind, odata, osize-1, quote);
+
+    for (i = 0, o = 1; i < isize; i++) {
+        Py_UCS4 ch = PyUnicode_READ(ikind, idata, i);
 
         /* Escape quotes and backslashes */
-        if ((ch == PyUnicode_AS_UNICODE(repr)[0]) || (ch == '\\')) {
-            *p++ = '\\';
-            *p++ = ch;
+        if ((ch == quote) || (ch == '\\')) {
+            PyUnicode_WRITE(okind, odata, o++, '\\');
+            PyUnicode_WRITE(okind, odata, o++, ch);
             continue;
         }
 
         /* Map special whitespace to '\t', \n', '\r' */
         if (ch == '\t') {
-            *p++ = '\\';
-            *p++ = 't';
+            PyUnicode_WRITE(okind, odata, o++, '\\');
+            PyUnicode_WRITE(okind, odata, o++, 't');
         }
         else if (ch == '\n') {
-            *p++ = '\\';
-            *p++ = 'n';
+            PyUnicode_WRITE(okind, odata, o++, '\\');
+            PyUnicode_WRITE(okind, odata, o++, 'n');
         }
         else if (ch == '\r') {
-            *p++ = '\\';
-            *p++ = 'r';
+            PyUnicode_WRITE(okind, odata, o++, '\\');
+            PyUnicode_WRITE(okind, odata, o++, 'r');
         }
 
         /* Map non-printable US ASCII to '\xhh' */
         else if (ch < ' ' || ch == 0x7F) {
-            *p++ = '\\';
-            *p++ = 'x';
-            *p++ = hexdigits[(ch >> 4) & 0x000F];
-            *p++ = hexdigits[ch & 0x000F];
+            PyUnicode_WRITE(okind, odata, o++, '\\');
+            PyUnicode_WRITE(okind, odata, o++, 'x');
+            PyUnicode_WRITE(okind, odata, o++, hexdigits[(ch >> 4) & 0x000F]);
+            PyUnicode_WRITE(okind, odata, o++, hexdigits[ch & 0x000F]);
         }
 
         /* Copy ASCII characters as-is */
         else if (ch < 0x7F) {
-            *p++ = ch;
+            PyUnicode_WRITE(okind, odata, o++, ch);
         }
 
         /* Non-ASCII characters */
         else {
-            Py_UCS4 ucs = ch;
-
-#ifndef Py_UNICODE_WIDE
-            Py_UNICODE ch2 = 0;
-            /* Get code point from surrogate pair */
-            if (size > 0) {
-                ch2 = *s;
-                if (ch >= 0xD800 && ch < 0xDC00 && ch2 >= 0xDC00
-                    && ch2 <= 0xDFFF) {
-                    ucs = (((ch & 0x03FF) << 10) | (ch2 & 0x03FF))
-                        + 0x00010000;
-                    s++;
-                    size--;
-                }
-            }
-#endif
             /* Map Unicode whitespace and control characters
                (categories Z* and C* except ASCII space)
             */
-            if (!Py_UNICODE_ISPRINTABLE(ucs)) {
+            if (!Py_UNICODE_ISPRINTABLE(ch)) {
                 /* Map 8-bit characters to '\xhh' */
-                if (ucs <= 0xff) {
-                    *p++ = '\\';
-                    *p++ = 'x';
-                    *p++ = hexdigits[(ch >> 4) & 0x000F];
-                    *p++ = hexdigits[ch & 0x000F];
+                if (ch <= 0xff) {
+                    PyUnicode_WRITE(okind, odata, o++, '\\');
+                    PyUnicode_WRITE(okind, odata, o++, 'x');
+                    PyUnicode_WRITE(okind, odata, o++, hexdigits[(ch >> 4) & 0x000F]);
+                    PyUnicode_WRITE(okind, odata, o++, hexdigits[ch & 0x000F]);
                 }
                 /* Map 21-bit characters to '\U00xxxxxx' */
-                else if (ucs >= 0x10000) {
-                    *p++ = '\\';
-                    *p++ = 'U';
-                    *p++ = hexdigits[(ucs >> 28) & 0x0000000F];
-                    *p++ = hexdigits[(ucs >> 24) & 0x0000000F];
-                    *p++ = hexdigits[(ucs >> 20) & 0x0000000F];
-                    *p++ = hexdigits[(ucs >> 16) & 0x0000000F];
-                    *p++ = hexdigits[(ucs >> 12) & 0x0000000F];
-                    *p++ = hexdigits[(ucs >> 8) & 0x0000000F];
-                    *p++ = hexdigits[(ucs >> 4) & 0x0000000F];
-                    *p++ = hexdigits[ucs & 0x0000000F];
+                else if (ch >= 0x10000) {
+                    PyUnicode_WRITE(okind, odata, o++, '\\');
+                    PyUnicode_WRITE(okind, odata, o++, 'U');
+                    PyUnicode_WRITE(okind, odata, o++, hexdigits[(ch >> 28) & 0xF]);
+                    PyUnicode_WRITE(okind, odata, o++, hexdigits[(ch >> 24) & 0xF]);
+                    PyUnicode_WRITE(okind, odata, o++, hexdigits[(ch >> 20) & 0xF]);
+                    PyUnicode_WRITE(okind, odata, o++, hexdigits[(ch >> 16) & 0xF]);
+                    PyUnicode_WRITE(okind, odata, o++, hexdigits[(ch >> 12) & 0xF]);
+                    PyUnicode_WRITE(okind, odata, o++, hexdigits[(ch >> 8) & 0xF]);
+                    PyUnicode_WRITE(okind, odata, o++, hexdigits[(ch >> 4) & 0xF]);
+                    PyUnicode_WRITE(okind, odata, o++, hexdigits[ch & 0xF]);
                 }
                 /* Map 16-bit characters to '\uxxxx' */
                 else {
-                    *p++ = '\\';
-                    *p++ = 'u';
-                    *p++ = hexdigits[(ucs >> 12) & 0x000F];
-                    *p++ = hexdigits[(ucs >> 8) & 0x000F];
-                    *p++ = hexdigits[(ucs >> 4) & 0x000F];
-                    *p++ = hexdigits[ucs & 0x000F];
+                    PyUnicode_WRITE(okind, odata, o++, '\\');
+                    PyUnicode_WRITE(okind, odata, o++, 'u');
+                    PyUnicode_WRITE(okind, odata, o++, hexdigits[(ch >> 12) & 0xF]);
+                    PyUnicode_WRITE(okind, odata, o++, hexdigits[(ch >> 8) & 0xF]);
+                    PyUnicode_WRITE(okind, odata, o++, hexdigits[(ch >> 4) & 0xF]);
+                    PyUnicode_WRITE(okind, odata, o++, hexdigits[ch & 0xF]);
                 }
             }
             /* Copy characters as-is */
             else {
-                *p++ = ch;
-#ifndef Py_UNICODE_WIDE
-                if (ucs >= 0x10000)
-                    *p++ = ch2;
-#endif
-            }
-        }
-    }
-    /* Add quote */
-    *p++ = PyUnicode_AS_UNICODE(repr)[0];
-
-    *p = '\0';
-    PyUnicode_Resize(&repr, p - PyUnicode_AS_UNICODE(repr));
+                PyUnicode_WRITE(okind, odata, o++, ch);
+            }
+        }
+    }
+    /* Closing quote already added at the beginning */
     return repr;
 }
 
@@ -8617,7 +10817,7 @@
 Return -1 on failure.");
 
 static PyObject *
-unicode_rfind(PyUnicodeObject *self, PyObject *args)
+unicode_rfind(PyObject *self, PyObject *args)
 {
     PyUnicodeObject *substring;
     Py_ssize_t start;
@@ -8628,14 +10828,21 @@
                                             &start, &end))
         return NULL;
 
-    result = stringlib_rfind_slice(
-        PyUnicode_AS_UNICODE(self), PyUnicode_GET_SIZE(self),
-        PyUnicode_AS_UNICODE(substring), PyUnicode_GET_SIZE(substring),
-        start, end
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+    if (PyUnicode_READY(substring) == -1)
+        return NULL;
+
+    result = any_find_slice(
+        ucs1lib_rfind_slice, ucs2lib_rfind_slice, ucs4lib_rfind_slice,
+        self, (PyObject*)substring, start, end
         );
 
     Py_DECREF(substring);
 
+    if (result == -2)
+        return NULL;
+
     return PyLong_FromSsize_t(result);
 }
 
@@ -8645,7 +10852,7 @@
 Like S.rfind() but raise ValueError when the substring is not found.");
 
 static PyObject *
-unicode_rindex(PyUnicodeObject *self, PyObject *args)
+unicode_rindex(PyObject *self, PyObject *args)
 {
     PyUnicodeObject *substring;
     Py_ssize_t start;
@@ -8656,18 +10863,26 @@
                                             &start, &end))
         return NULL;
 
-    result = stringlib_rfind_slice(
-        PyUnicode_AS_UNICODE(self), PyUnicode_GET_SIZE(self),
-        PyUnicode_AS_UNICODE(substring), PyUnicode_GET_SIZE(substring),
-        start, end
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+    if (PyUnicode_READY(substring) == -1)
+        return NULL;
+
+    result = any_find_slice(
+        ucs1lib_rfind_slice, ucs2lib_rfind_slice, ucs4lib_rfind_slice,
+        self, (PyObject*)substring, start, end
         );
 
     Py_DECREF(substring);
 
+    if (result == -2)
+        return NULL;
+
     if (result < 0) {
         PyErr_SetString(PyExc_ValueError, "substring not found");
         return NULL;
     }
+
     return PyLong_FromSsize_t(result);
 }
 
@@ -8681,17 +10896,20 @@
 unicode_rjust(PyUnicodeObject *self, PyObject *args)
 {
     Py_ssize_t width;
-    Py_UNICODE fillchar = ' ';
+    Py_UCS4 fillchar = ' ';
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
 
     if (!PyArg_ParseTuple(args, "n|O&:rjust", &width, convert_uc, &fillchar))
         return NULL;
 
-    if (self->length >= width && PyUnicode_CheckExact(self)) {
+    if (_PyUnicode_LENGTH(self) >= width && PyUnicode_CheckExact(self)) {
         Py_INCREF(self);
         return (PyObject*) self;
     }
 
-    return (PyObject*) pad(self, width - self->length, 0, fillchar);
+    return (PyObject*) pad(self, width - _PyUnicode_LENGTH(self), 0, fillchar);
 }
 
 PyObject *
@@ -8749,6 +10967,78 @@
     PyObject* str_obj;
     PyObject* sep_obj;
     PyObject* out;
+    int kind1, kind2, kind;
+    void *buf1 = NULL, *buf2 = NULL;
+    Py_ssize_t len1, len2;
+
+    str_obj = PyUnicode_FromObject(str_in);
+    if (!str_obj || PyUnicode_READY(str_in) == -1)
+        return NULL;
+    sep_obj = PyUnicode_FromObject(sep_in);
+    if (!sep_obj || PyUnicode_READY(sep_obj) == -1) {
+        Py_DECREF(str_obj);
+        return NULL;
+    }
+
+    kind1 = PyUnicode_KIND(str_in);
+    kind2 = PyUnicode_KIND(sep_obj);
+    kind = kind1 > kind2 ? kind1 : kind2;
+    buf1 = PyUnicode_DATA(str_in);
+    if (kind1 != kind)
+        buf1 = _PyUnicode_AsKind(str_in, kind);
+    if (!buf1)
+        goto onError;
+    buf2 = PyUnicode_DATA(sep_obj);
+    if (kind2 != kind)
+        buf2 = _PyUnicode_AsKind(sep_obj, kind);
+    if (!buf2)
+        goto onError;
+    len1 = PyUnicode_GET_LENGTH(str_obj);
+    len2 = PyUnicode_GET_LENGTH(sep_obj);
+
+    switch(PyUnicode_KIND(str_in)) {
+    case PyUnicode_1BYTE_KIND:
+        out = ucs1lib_partition(str_obj, buf1, len1, sep_obj, buf2, len2);
+        break;
+    case PyUnicode_2BYTE_KIND:
+        out = ucs2lib_partition(str_obj, buf1, len1, sep_obj, buf2, len2);
+        break;
+    case PyUnicode_4BYTE_KIND:
+        out = ucs4lib_partition(str_obj, buf1, len1, sep_obj, buf2, len2);
+        break;
+    default:
+        assert(0);
+        out = 0;
+    }
+
+    Py_DECREF(sep_obj);
+    Py_DECREF(str_obj);
+    if (kind1 != kind)
+        PyMem_Free(buf1);
+    if (kind2 != kind)
+        PyMem_Free(buf2);
+
+    return out;
+  onError:
+    Py_DECREF(sep_obj);
+    Py_DECREF(str_obj);
+    if (kind1 != kind && buf1)
+        PyMem_Free(buf1);
+    if (kind2 != kind && buf2)
+        PyMem_Free(buf2);
+    return NULL;
+}
+
+
+PyObject *
+PyUnicode_RPartition(PyObject *str_in, PyObject *sep_in)
+{
+    PyObject* str_obj;
+    PyObject* sep_obj;
+    PyObject* out;
+    int kind1, kind2, kind;
+    void *buf1 = NULL, *buf2 = NULL;
+    Py_ssize_t len1, len2;
 
     str_obj = PyUnicode_FromObject(str_in);
     if (!str_obj)
@@ -8759,43 +11049,53 @@
         return NULL;
     }
 
-    out = stringlib_partition(
-        str_obj, PyUnicode_AS_UNICODE(str_obj), PyUnicode_GET_SIZE(str_obj),
-        sep_obj, PyUnicode_AS_UNICODE(sep_obj), PyUnicode_GET_SIZE(sep_obj)
-        );
+    kind1 = PyUnicode_KIND(str_in);
+    kind2 = PyUnicode_KIND(sep_obj);
+    kind = PY_MAX(kind1, kind2);
+    buf1 = PyUnicode_DATA(str_in);
+    if (kind1 != kind)
+        buf1 = _PyUnicode_AsKind(str_in, kind);
+    if (!buf1)
+        goto onError;
+    buf2 = PyUnicode_DATA(sep_obj);
+    if (kind2 != kind)
+        buf2 = _PyUnicode_AsKind(sep_obj, kind);
+    if (!buf2)
+        goto onError;
+    len1 = PyUnicode_GET_LENGTH(str_obj);
+    len2 = PyUnicode_GET_LENGTH(sep_obj);
+
+    switch(PyUnicode_KIND(str_in)) {
+    case PyUnicode_1BYTE_KIND:
+        out = ucs1lib_rpartition(str_obj, buf1, len1, sep_obj, buf2, len2);
+        break;
+    case PyUnicode_2BYTE_KIND:
+        out = ucs2lib_rpartition(str_obj, buf1, len1, sep_obj, buf2, len2);
+        break;
+    case PyUnicode_4BYTE_KIND:
+        out = ucs4lib_rpartition(str_obj, buf1, len1, sep_obj, buf2, len2);
+        break;
+    default:
+        assert(0);
+        out = 0;
+    }
 
     Py_DECREF(sep_obj);
     Py_DECREF(str_obj);
+    if (kind1 != kind)
+        PyMem_Free(buf1);
+    if (kind2 != kind)
+        PyMem_Free(buf2);
 
     return out;
-}
-
-
-PyObject *
-PyUnicode_RPartition(PyObject *str_in, PyObject *sep_in)
-{
-    PyObject* str_obj;
-    PyObject* sep_obj;
-    PyObject* out;
-
-    str_obj = PyUnicode_FromObject(str_in);
-    if (!str_obj)
-        return NULL;
-    sep_obj = PyUnicode_FromObject(sep_in);
-    if (!sep_obj) {
-        Py_DECREF(str_obj);
-        return NULL;
-    }
-
-    out = stringlib_rpartition(
-        str_obj, PyUnicode_AS_UNICODE(str_obj), PyUnicode_GET_SIZE(str_obj),
-        sep_obj, PyUnicode_AS_UNICODE(sep_obj), PyUnicode_GET_SIZE(sep_obj)
-        );
-
+  onError:
     Py_DECREF(sep_obj);
     Py_DECREF(str_obj);
-
-    return out;
+    if (kind1 != kind && buf1)
+        PyMem_Free(buf1);
+    if (kind2 != kind && buf2)
+        PyMem_Free(buf2);
+    return NULL;
 }
 
 PyDoc_STRVAR(partition__doc__,
@@ -8943,22 +11243,28 @@
     if (!new)
         return NULL;
     if (y != NULL) {
+        int x_kind, y_kind, z_kind;
+        void *x_data, *y_data, *z_data;
+
         /* x must be a string too, of equal length */
-        Py_ssize_t ylen = PyUnicode_GET_SIZE(y);
         if (!PyUnicode_Check(x)) {
             PyErr_SetString(PyExc_TypeError, "first maketrans argument must "
                             "be a string if there is a second argument");
             goto err;
         }
-        if (PyUnicode_GET_SIZE(x) != ylen) {
+        if (PyUnicode_GET_LENGTH(x) != PyUnicode_GET_LENGTH(y)) {
             PyErr_SetString(PyExc_ValueError, "the first two maketrans "
                             "arguments must have equal length");
             goto err;
         }
         /* create entries for translating chars in x to those in y */
-        for (i = 0; i < PyUnicode_GET_SIZE(x); i++) {
-            key = PyLong_FromLong(PyUnicode_AS_UNICODE(x)[i]);
-            value = PyLong_FromLong(PyUnicode_AS_UNICODE(y)[i]);
+        x_kind = PyUnicode_KIND(x);
+        y_kind = PyUnicode_KIND(y);
+        x_data = PyUnicode_DATA(x);
+        y_data = PyUnicode_DATA(y);
+        for (i = 0; i < PyUnicode_GET_LENGTH(x); i++) {
+            key = PyLong_FromLong(PyUnicode_READ(x_kind, x_data, i));
+            value = PyLong_FromLong(PyUnicode_READ(y_kind, y_data, i));
             if (!key || !value)
                 goto err;
             res = PyDict_SetItem(new, key, value);
@@ -8969,8 +11275,10 @@
         }
         /* create entries for deleting chars in z */
         if (z != NULL) {
+            z_kind = PyUnicode_KIND(z);
+            z_data = PyUnicode_DATA(z);
             for (i = 0; i < PyUnicode_GET_SIZE(z); i++) {
-                key = PyLong_FromLong(PyUnicode_AS_UNICODE(z)[i]);
+                key = PyLong_FromLong(PyUnicode_READ(z_kind, z_data, i));
                 if (!key)
                     goto err;
                 res = PyDict_SetItem(new, key, Py_None);
@@ -8980,6 +11288,9 @@
             }
         }
     } else {
+        int kind;
+        void *data;
+
         /* x must be a dict */
         if (!PyDict_CheckExact(x)) {
             PyErr_SetString(PyExc_TypeError, "if you give only one argument "
@@ -8996,7 +11307,9 @@
                                     "table must be of length 1");
                     goto err;
                 }
-                newkey = PyLong_FromLong(PyUnicode_AS_UNICODE(key)[0]);
+                kind = PyUnicode_KIND(key);
+                data = PyUnicode_DATA(key);
+                newkey = PyLong_FromLong(PyUnicode_READ(kind, data, 0));
                 if (!newkey)
                     goto err;
                 res = PyDict_SetItem(new, newkey, value);
@@ -9030,9 +11343,9 @@
 are deleted.");
 
 static PyObject*
-unicode_translate(PyUnicodeObject *self, PyObject *table)
-{
-    return PyUnicode_TranslateCharmap(self->str, self->length, table, "ignore");
+unicode_translate(PyObject *self, PyObject *table)
+{
+    return _PyUnicode_TranslateCharmap(self, table, "ignore");
 }
 
 PyDoc_STRVAR(upper__doc__,
@@ -9057,12 +11370,18 @@
 {
     Py_ssize_t fill;
     PyUnicodeObject *u;
-
     Py_ssize_t width;
+    int kind;
+    void *data;
+    Py_UCS4 chr;
+
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+
     if (!PyArg_ParseTuple(args, "n:zfill", &width))
         return NULL;
 
-    if (self->length >= width) {
+    if (PyUnicode_GET_LENGTH(self) >= width) {
         if (PyUnicode_CheckExact(self)) {
             Py_INCREF(self);
             return (PyObject*) self;
@@ -9074,34 +11393,31 @@
                 );
     }
 
-    fill = width - self->length;
+    fill = width - _PyUnicode_LENGTH(self);
 
     u = pad(self, fill, 0, '0');
 
     if (u == NULL)
         return NULL;
 
-    if (u->str[fill] == '+' || u->str[fill] == '-') {
+    kind = PyUnicode_KIND(u);
+    data = PyUnicode_DATA(u);
+    chr = PyUnicode_READ(kind, data, fill);
+
+    if (chr == '+' || chr == '-') {
         /* move sign to beginning of string */
-        u->str[0] = u->str[fill];
-        u->str[fill] = '0';
+        PyUnicode_WRITE(kind, data, 0, chr);
+        PyUnicode_WRITE(kind, data, fill, '0');
     }
 
     return (PyObject*) u;
 }
 
 #if 0
-static PyObject*
-unicode_freelistsize(PyUnicodeObject *self)
-{
-    return PyLong_FromLong(numfree);
-}
-
 static PyObject *
 unicode__decimal2ascii(PyObject *self)
 {
-    return PyUnicode_TransformDecimalToASCII(PyUnicode_AS_UNICODE(self),
-                                             PyUnicode_GET_SIZE(self));
+    return PyUnicode_TransformDecimalAndSpaceToASCII(self);
 }
 #endif
 
@@ -9201,7 +11517,7 @@
     return PyBool_FromLong(result);
 }
 
-#include "stringlib/string_format.h"
+#include "stringlib/unicode_format.h"
 
 PyDoc_STRVAR(format__doc__,
              "S.format(*args, **kwargs) -> str\n\
@@ -9223,9 +11539,8 @@
     if (!PyArg_ParseTuple(args, "U:__format__", &format_spec))
         return NULL;
 
-    return _PyUnicode_FormatAdvanced(self,
-                                     PyUnicode_AS_UNICODE(format_spec),
-                                     PyUnicode_GET_SIZE(format_spec));
+    return _PyUnicode_FormatAdvanced(self, format_spec, 0,
+                                     PyUnicode_GET_LENGTH(format_spec));
 }
 
 PyDoc_STRVAR(p_format__doc__,
@@ -9236,8 +11551,35 @@
 static PyObject *
 unicode__sizeof__(PyUnicodeObject *v)
 {
-    return PyLong_FromSsize_t(sizeof(PyUnicodeObject) +
-                              sizeof(Py_UNICODE) * (v->length + 1));
+    Py_ssize_t size;
+
+    /* If it's a compact object, account for base structure +
+       character data. */
+    if (PyUnicode_IS_COMPACT_ASCII(v))
+        size = sizeof(PyASCIIObject) + PyUnicode_GET_LENGTH(v) + 1;
+    else if (PyUnicode_IS_COMPACT(v))
+        size = sizeof(PyCompactUnicodeObject) +
+            (PyUnicode_GET_LENGTH(v) + 1) * PyUnicode_CHARACTER_SIZE(v);
+    else {
+        /* If it is a two-block object, account for base object, and
+           for character block if present. */
+        size = sizeof(PyUnicodeObject);
+        if (v->data.any)
+            size += (PyUnicode_GET_LENGTH(v) + 1) *
+                PyUnicode_CHARACTER_SIZE(v);
+    }
+    /* If the wstr pointer is present, account for it unless it is shared
+       with the data pointer. Since PyUnicode_DATA will crash if the object
+       is not ready, check whether it's either not ready (in which case the
+       data is entirely in wstr) or if the data is not shared. */
+    if (_PyUnicode_WSTR(v) &&
+        (!PyUnicode_IS_READY(v) ||
+         (PyUnicode_DATA(v) != _PyUnicode_WSTR(v))))
+        size += (PyUnicode_WSTR_LENGTH(v) + 1) * sizeof(wchar_t);
+    if (_PyUnicode_UTF8(v) && _PyUnicode_UTF8(v) != PyUnicode_DATA(v))
+        size += _PyUnicode_UTF8_LENGTH(v) + 1;
+
+    return PyLong_FromSsize_t(size);
 }
 
 PyDoc_STRVAR(sizeof__doc__,
@@ -9246,7 +11588,17 @@
 static PyObject *
 unicode_getnewargs(PyUnicodeObject *v)
 {
-    return Py_BuildValue("(u#)", v->str, v->length);
+    PyObject *copy;
+    unsigned char *data;
+    int kind;
+    if (PyUnicode_READY(v) == -1)
+        return NULL;
+    kind = PyUnicode_KIND(v);
+    data = PyUnicode_1BYTE_DATA(v);
+    copy = PyUnicode_FromKindAndData(kind, data, PyUnicode_GET_LENGTH(v));
+    if (!copy)
+        return NULL;
+    return Py_BuildValue("(N)", copy);
 }
 
 static PyMethodDef unicode_methods[] = {
@@ -9306,7 +11658,6 @@
 
 #if 0
     /* These methods are just used for debugging the implementation. */
-    {"freelistsize", (PyCFunction) unicode_freelistsize, METH_NOARGS},
     {"_decimal2ascii", (PyCFunction) unicode__decimal2ascii, METH_NOARGS},
 #endif
 
@@ -9343,32 +11694,36 @@
 static PyObject*
 unicode_subscript(PyUnicodeObject* self, PyObject* item)
 {
+    if (PyUnicode_READY(self) == -1)
+        return NULL;
+
     if (PyIndex_Check(item)) {
         Py_ssize_t i = PyNumber_AsSsize_t(item, PyExc_IndexError);
         if (i == -1 && PyErr_Occurred())
             return NULL;
         if (i < 0)
-            i += PyUnicode_GET_SIZE(self);
+            i += PyUnicode_GET_LENGTH(self);
         return unicode_getitem(self, i);
     } else if (PySlice_Check(item)) {
         Py_ssize_t start, stop, step, slicelength, cur, i;
-        Py_UNICODE* source_buf;
+        const Py_UNICODE* source_buf;
         Py_UNICODE* result_buf;
         PyObject* result;
 
-        if (PySlice_GetIndicesEx(item, PyUnicode_GET_SIZE(self),
+        if (PySlice_GetIndicesEx(item, PyUnicode_GET_LENGTH(self),
                                  &start, &stop, &step, &slicelength) < 0) {
             return NULL;
         }
 
         if (slicelength <= 0) {
-            return PyUnicode_FromUnicode(NULL, 0);
-        } else if (start == 0 && step == 1 && slicelength == self->length &&
+            return PyUnicode_New(0, 0);
+        } else if (start == 0 && step == 1 &&
+                   slicelength == PyUnicode_GET_LENGTH(self) &&
                    PyUnicode_CheckExact(self)) {
             Py_INCREF(self);
             return (PyObject *)self;
         } else if (step == 1) {
-            return PyUnicode_FromUnicode(self->str + start, slicelength);
+            return substring(self, start, slicelength);
         } else {
             source_buf = PyUnicode_AS_UNICODE((PyObject*)self);
             result_buf = (Py_UNICODE *)PyObject_MALLOC(slicelength*
@@ -9436,7 +11791,7 @@
                               (flags & F_ALT) ? Py_DTSF_ALT : 0, NULL);
     if (p == NULL)
         return NULL;
-    result = PyUnicode_FromStringAndSize(p, strlen(p));
+    result = PyUnicode_DecodeASCII(p, strlen(p), NULL);
     PyMem_Free(p);
     return result;
 }
@@ -9452,37 +11807,23 @@
     str = _PyBytes_FormatLong(val, flags, prec, type, &buf, &len);
     if (!str)
         return NULL;
-    result = PyUnicode_FromStringAndSize(buf, len);
+    result = PyUnicode_DecodeASCII(buf, len, NULL);
     Py_DECREF(str);
     return result;
 }
 
 static int
-formatchar(Py_UNICODE *buf,
+formatchar(Py_UCS4 *buf,
            size_t buflen,
            PyObject *v)
 {
     /* presume that the buffer is at least 3 characters long */
     if (PyUnicode_Check(v)) {
-        if (PyUnicode_GET_SIZE(v) == 1) {
-            buf[0] = PyUnicode_AS_UNICODE(v)[0];
+        if (PyUnicode_GET_LENGTH(v) == 1) {
+            buf[0] = PyUnicode_READ_CHAR(v, 0);
             buf[1] = '\0';
             return 1;
         }
-#ifndef Py_UNICODE_WIDE
-        if (PyUnicode_GET_SIZE(v) == 2) {
-            /* Decode a valid surrogate pair */
-            int c0 = PyUnicode_AS_UNICODE(v)[0];
-            int c1 = PyUnicode_AS_UNICODE(v)[1];
-            if (0xD800 <= c0 && c0 <= 0xDBFF &&
-                0xDC00 <= c1 && c1 <= 0xDFFF) {
-                buf[0] = c0;
-                buf[1] = c1;
-                buf[2] = '\0';
-                return 2;
-            }
-        }
-#endif
         goto onError;
     }
     else {
@@ -9498,15 +11839,7 @@
             return -1;
         }
 
-#ifndef Py_UNICODE_WIDE
-        if (x > 0xffff) {
-            x -= 0x10000;
-            buf[0] = (Py_UNICODE)(0xD800 | (x >> 10));
-            buf[1] = (Py_UNICODE)(0xDC00 | (x & 0x3FF));
-            return 2;
-        }
-#endif
-        buf[0] = (Py_UNICODE) x;
+        buf[0] = (Py_UCS4) x;
         buf[1] = '\0';
         return 1;
     }
@@ -9525,28 +11858,35 @@
 PyObject *
 PyUnicode_Format(PyObject *format, PyObject *args)
 {
-    Py_UNICODE *fmt, *res;
-    Py_ssize_t fmtcnt, rescnt, reslen, arglen, argidx;
+    void *fmt;
+    int fmtkind;
+    PyObject *result;
+    Py_UCS4 *res, *res0;
+    Py_UCS4 max;
+    int kind;
+    Py_ssize_t fmtcnt, fmtpos, rescnt, reslen, arglen, argidx;
     int args_owned = 0;
-    PyUnicodeObject *result = NULL;
     PyObject *dict = NULL;
-    PyObject *uformat;
+    PyUnicodeObject *uformat;
 
     if (format == NULL || args == NULL) {
         PyErr_BadInternalCall();
         return NULL;
     }
-    uformat = PyUnicode_FromObject(format);
-    if (uformat == NULL)
-        return NULL;
-    fmt = PyUnicode_AS_UNICODE(uformat);
-    fmtcnt = PyUnicode_GET_SIZE(uformat);
+    uformat = (PyUnicodeObject*)PyUnicode_FromObject(format);
+    if (uformat == NULL || PyUnicode_READY(uformat) == -1)
+        return NULL;
+    fmt = PyUnicode_DATA(uformat);
+    fmtkind = PyUnicode_KIND(uformat);
+    fmtcnt = PyUnicode_GET_LENGTH(uformat);
+    fmtpos = 0;
 
     reslen = rescnt = fmtcnt + 100;
-    result = _PyUnicode_New(reslen);
-    if (result == NULL)
+    res = res0 = PyMem_Malloc(reslen * sizeof(Py_UCS4));
+    if (res0 == NULL) {
+        PyErr_NoMemory();
         goto onError;
-    res = PyUnicode_AS_UNICODE(result);
+    }
 
     if (PyTuple_Check(args)) {
         arglen = PyTuple_Size(args);
@@ -9561,35 +11901,39 @@
         dict = args;
 
     while (--fmtcnt >= 0) {
-        if (*fmt != '%') {
+        if (PyUnicode_READ(fmtkind, fmt, fmtpos) != '%') {
             if (--rescnt < 0) {
                 rescnt = fmtcnt + 100;
                 reslen += rescnt;
-                if (_PyUnicode_Resize(&result, reslen) < 0)
+                res0 = PyMem_Realloc(res0, reslen*sizeof(Py_UCS4));
+                if (res0 == NULL){
+                    PyErr_NoMemory();
                     goto onError;
-                res = PyUnicode_AS_UNICODE(result) + reslen - rescnt;
+                }
+                res = res0 + reslen - rescnt;
                 --rescnt;
             }
-            *res++ = *fmt++;
+            *res++ = PyUnicode_READ(fmtkind, fmt, fmtpos++);
         }
         else {
             /* Got a format specifier */
             int flags = 0;
             Py_ssize_t width = -1;
             int prec = -1;
-            Py_UNICODE c = '\0';
-            Py_UNICODE fill;
+            Py_UCS4 c = '\0';
+            Py_UCS4 fill;
             int isnumok;
             PyObject *v = NULL;
             PyObject *temp = NULL;
-            Py_UNICODE *pbuf;
+            void *pbuf;
+            Py_ssize_t pindex;
             Py_UNICODE sign;
-            Py_ssize_t len;
-            Py_UNICODE formatbuf[FORMATBUFLEN]; /* For formatchar() */
-
-            fmt++;
-            if (*fmt == '(') {
-                Py_UNICODE *keystart;
+            Py_ssize_t len, len1;
+            Py_UCS4 formatbuf[FORMATBUFLEN]; /* For formatchar() */
+
+            fmtpos++;
+            if (PyUnicode_READ(fmtkind, fmt, fmtpos) == '(') {
+                Py_ssize_t keystart;
                 Py_ssize_t keylen;
                 PyObject *key;
                 int pcount = 1;
@@ -9599,34 +11943,24 @@
                                     "format requires a mapping");
                     goto onError;
                 }
-                ++fmt;
+                ++fmtpos;
                 --fmtcnt;
-                keystart = fmt;
+                keystart = fmtpos;
                 /* Skip over balanced parentheses */
                 while (pcount > 0 && --fmtcnt >= 0) {
-                    if (*fmt == ')')
+                    if (PyUnicode_READ(fmtkind, fmt, fmtpos) == ')')
                         --pcount;
-                    else if (*fmt == '(')
+                    else if (PyUnicode_READ(fmtkind, fmt, fmtpos) == '(')
                         ++pcount;
-                    fmt++;
+                    fmtpos++;
                 }
-                keylen = fmt - keystart - 1;
+                keylen = fmtpos - keystart - 1;
                 if (fmtcnt < 0 || pcount > 0) {
                     PyErr_SetString(PyExc_ValueError,
                                     "incomplete format key");
                     goto onError;
                 }
-#if 0
-                /* keys are converted to strings using UTF-8 and
-                   then looked up since Python uses strings to hold
-                   variables names etc. in its namespaces and we
-                   wouldn't want to break common idioms. */
-                key = PyUnicode_EncodeUTF8(keystart,
-                                           keylen,
-                                           NULL);
-#else
-                key = PyUnicode_FromUnicode(keystart, keylen);
-#endif
+                key = substring(uformat, keystart, keylen);
                 if (key == NULL)
                     goto onError;
                 if (args_owned) {
@@ -9643,7 +11977,7 @@
                 argidx = -2;
             }
             while (--fmtcnt >= 0) {
-                switch (c = *fmt++) {
+                switch (c = PyUnicode_READ(fmtkind, fmt, fmtpos++)) {
                 case '-': flags |= F_LJUST; continue;
                 case '+': flags |= F_SIGN; continue;
                 case ' ': flags |= F_BLANK; continue;
@@ -9669,12 +12003,12 @@
                     width = -width;
                 }
                 if (--fmtcnt >= 0)
-                    c = *fmt++;
+                    c = PyUnicode_READ(fmtkind, fmt, fmtpos++);
             }
             else if (c >= '0' && c <= '9') {
                 width = c - '0';
                 while (--fmtcnt >= 0) {
-                    c = *fmt++;
+                    c = PyUnicode_READ(fmtkind, fmt, fmtpos++);
                     if (c < '0' || c > '9')
                         break;
                     if ((width*10) / 10 != width) {
@@ -9688,7 +12022,7 @@
             if (c == '.') {
                 prec = 0;
                 if (--fmtcnt >= 0)
-                    c = *fmt++;
+                    c = PyUnicode_READ(fmtkind, fmt, fmtpos++);
                 if (c == '*') {
                     v = getnextarg(args, arglen, &argidx);
                     if (v == NULL)
@@ -9704,12 +12038,12 @@
                     if (prec < 0)
                         prec = 0;
                     if (--fmtcnt >= 0)
-                        c = *fmt++;
+                        c = PyUnicode_READ(fmtkind, fmt, fmtpos++);
                 }
                 else if (c >= '0' && c <= '9') {
                     prec = c - '0';
                     while (--fmtcnt >= 0) {
-                        c = *fmt++;
+                        c = PyUnicode_READ(fmtkind, fmt, fmtpos++);
                         if (c < '0' || c > '9')
                             break;
                         if ((prec*10) / 10 != prec) {
@@ -9724,7 +12058,7 @@
             if (fmtcnt >= 0) {
                 if (c == 'h' || c == 'l' || c == 'L') {
                     if (--fmtcnt >= 0)
-                        c = *fmt++;
+                        c = PyUnicode_READ(fmtkind, fmt, fmtpos++);
                 }
             }
             if (fmtcnt < 0) {
@@ -9743,8 +12077,9 @@
 
             case '%':
                 pbuf = formatbuf;
+                kind = PyUnicode_4BYTE_KIND;
                 /* presume that buffer length is at least 1 */
-                pbuf[0] = '%';
+                PyUnicode_WRITE(kind, pbuf, 0, '%');
                 len = 1;
                 break;
 
@@ -9773,8 +12108,13 @@
                         goto onError;
                     }
                 }
-                pbuf = PyUnicode_AS_UNICODE(temp);
-                len = PyUnicode_GET_SIZE(temp);
+                if (PyUnicode_READY(temp) == -1) {
+                    Py_CLEAR(temp);
+                    goto onError;
+                }
+                pbuf = PyUnicode_DATA(temp);
+                kind = PyUnicode_KIND(temp);
+                len = PyUnicode_GET_LENGTH(temp);
                 if (prec >= 0 && len > prec)
                     len = prec;
                 break;
@@ -9803,8 +12143,13 @@
                             Py_DECREF(iobj);
                             if (!temp)
                                 goto onError;
-                            pbuf = PyUnicode_AS_UNICODE(temp);
-                            len = PyUnicode_GET_SIZE(temp);
+                            if (PyUnicode_READY(temp) == -1) {
+                                Py_CLEAR(temp);
+                                goto onError;
+                            }
+                            pbuf = PyUnicode_DATA(temp);
+                            kind = PyUnicode_KIND(temp);
+                            len = PyUnicode_GET_LENGTH(temp);
                             sign = 1;
                         }
                         else {
@@ -9831,8 +12176,13 @@
                 temp = formatfloat(v, flags, prec, c);
                 if (!temp)
                     goto onError;
-                pbuf = PyUnicode_AS_UNICODE(temp);
-                len = PyUnicode_GET_SIZE(temp);
+                if (PyUnicode_READY(temp) == -1) {
+                    Py_CLEAR(temp);
+                    goto onError;
+                }
+                pbuf = PyUnicode_DATA(temp);
+                kind = PyUnicode_KIND(temp);
+                len = PyUnicode_GET_LENGTH(temp);
                 sign = 1;
                 if (flags & F_ZERO)
                     fill = '0';
@@ -9840,6 +12190,7 @@
 
             case 'c':
                 pbuf = formatbuf;
+                kind = PyUnicode_4BYTE_KIND;
                 len = formatchar(pbuf, sizeof(formatbuf)/sizeof(Py_UNICODE), v);
                 if (len < 0)
                     goto onError;
@@ -9851,13 +12202,15 @@
                              "at index %zd",
                              (31<=c && c<=126) ? (char)c : '?',
                              (int)c,
-                             (Py_ssize_t)(fmt - 1 -
-                                          PyUnicode_AS_UNICODE(uformat)));
+                             fmtpos - 1);
                 goto onError;
             }
+            /* pbuf is initialized here. */
+            pindex = 0;
             if (sign) {
-                if (*pbuf == '-' || *pbuf == '+') {
-                    sign = *pbuf++;
+                if (PyUnicode_READ(kind, pbuf, pindex) == '-' ||
+                    PyUnicode_READ(kind, pbuf, pindex) == '+') {
+                    sign = PyUnicode_READ(kind, pbuf, pindex++);
                     len--;
                 }
                 else if (flags & F_SIGN)
@@ -9878,12 +12231,13 @@
                     PyErr_NoMemory();
                     goto onError;
                 }
-                if (_PyUnicode_Resize(&result, reslen) < 0) {
+                res0 = PyMem_Realloc(res0, reslen*sizeof(Py_UCS4));
+                if (res0 == 0) {
+                    PyErr_NoMemory();
                     Py_XDECREF(temp);
                     goto onError;
                 }
-                res = PyUnicode_AS_UNICODE(result)
-                    + reslen - rescnt;
+                res = res0 + reslen - rescnt;
             }
             if (sign) {
                 if (fill != ' ')
@@ -9893,11 +12247,11 @@
                     width--;
             }
             if ((flags & F_ALT) && (c == 'x' || c == 'X' || c == 'o')) {
-                assert(pbuf[0] == '0');
-                assert(pbuf[1] == c);
+                assert(PyUnicode_READ(kind, pbuf, pindex) == '0');
+                assert(PyUnicode_READ(kind, pbuf, pindex+1) == c);
                 if (fill != ' ') {
-                    *res++ = *pbuf++;
-                    *res++ = *pbuf++;
+                    *res++ = PyUnicode_READ(kind, pbuf, pindex++);
+                    *res++ = PyUnicode_READ(kind, pbuf, pindex++);
                 }
                 rescnt -= 2;
                 width -= 2;
@@ -9915,15 +12269,18 @@
                 if (sign)
                     *res++ = sign;
                 if ((flags & F_ALT) && (c == 'x' || c == 'X' || c == 'o')) {
-                    assert(pbuf[0] == '0');
-                    assert(pbuf[1] == c);
-                    *res++ = *pbuf++;
-                    *res++ = *pbuf++;
+                    assert(PyUnicode_READ(kind, pbuf, pindex) == '0');
+                    assert(PyUnicode_READ(kind, pbuf, pindex+1) == c);
+                    *res++ = PyUnicode_READ(kind, pbuf, pindex++);
+                    *res++ = PyUnicode_READ(kind, pbuf, pindex++);
                 }
             }
-            Py_UNICODE_COPY(res, pbuf, len);
-            res += len;
-            rescnt -= len;
+            /* Copy all characters, preserving len */
+            len1 = len;
+            while (len1--) {
+                *res++ = PyUnicode_READ(kind, pbuf, pindex++);
+                rescnt--;
+            }
             while (--width >= len) {
                 --rescnt;
                 *res++ = ' ';
@@ -9943,8 +12300,17 @@
         goto onError;
     }
 
-    if (_PyUnicode_Resize(&result, reslen - rescnt) < 0)
+
+    for (max=0, res = res0; res < res0+reslen-rescnt; res++)
+        if (*res > max)
+            max = *res;
+    result = PyUnicode_New(reslen - rescnt, max);
+    if (!result)
         goto onError;
+    kind = PyUnicode_KIND(result);
+    for (res = res0; res < res0+reslen-rescnt; res++)
+        PyUnicode_WRITE(kind, PyUnicode_DATA(result), res-res0, *res);
+    PyMem_Free(res0);
     if (args_owned) {
         Py_DECREF(args);
     }
@@ -9952,7 +12318,7 @@
     return (PyObject *)result;
 
   onError:
-    Py_XDECREF(result);
+    PyMem_Free(res0);
     Py_DECREF(uformat);
     if (args_owned) {
         Py_DECREF(args);
@@ -9977,7 +12343,7 @@
                                      kwlist, &x, &encoding, &errors))
         return NULL;
     if (x == NULL)
-        return (PyObject *)_PyUnicode_New(0);
+        return (PyObject *)PyUnicode_New(0, 0);
     if (encoding == NULL && errors == NULL)
         return PyObject_Str(x);
     else
@@ -9989,29 +12355,54 @@
 {
     PyUnicodeObject *tmp, *pnew;
     Py_ssize_t n;
+    PyObject *err = NULL;
 
     assert(PyType_IsSubtype(type, &PyUnicode_Type));
     tmp = (PyUnicodeObject *)unicode_new(&PyUnicode_Type, args, kwds);
     if (tmp == NULL)
         return NULL;
     assert(PyUnicode_Check(tmp));
-    pnew = (PyUnicodeObject *) type->tp_alloc(type, n = tmp->length);
+    // TODO: Verify the PyUnicode_GET_SIZE does the right thing.
+    //       it seems kind of strange that tp_alloc gets passed the size
+    //       of the unicode string because there will follow another
+    //       malloc.
+    pnew = (PyUnicodeObject *) type->tp_alloc(type,
+                                              n = PyUnicode_GET_SIZE(tmp));
     if (pnew == NULL) {
         Py_DECREF(tmp);
         return NULL;
     }
-    pnew->str = (Py_UNICODE*) PyObject_MALLOC(sizeof(Py_UNICODE) * (n+1));
-    if (pnew->str == NULL) {
-        _Py_ForgetReference((PyObject *)pnew);
-        PyObject_Del(pnew);
-        Py_DECREF(tmp);
-        return PyErr_NoMemory();
-    }
-    Py_UNICODE_COPY(pnew->str, tmp->str, n+1);
-    pnew->length = n;
-    pnew->hash = tmp->hash;
+    _PyUnicode_WSTR(pnew) = (Py_UNICODE*) PyObject_MALLOC(sizeof(Py_UNICODE) * (n+1));
+    if (_PyUnicode_WSTR(pnew) == NULL) {
+        err = PyErr_NoMemory();
+        goto onError;
+    }
+    Py_UNICODE_COPY(_PyUnicode_WSTR(pnew), PyUnicode_AS_UNICODE(tmp), n+1);
+    _PyUnicode_WSTR_LENGTH(pnew) = n;
+    _PyUnicode_HASH(pnew) = _PyUnicode_HASH(tmp);
+    _PyUnicode_STATE(pnew).interned = 0;
+    _PyUnicode_STATE(pnew).kind = 0;
+    _PyUnicode_STATE(pnew).compact = 0;
+    _PyUnicode_STATE(pnew).ready = 0;
+    _PyUnicode_STATE(pnew).ascii = 0;
+    pnew->data.any = NULL;
+    _PyUnicode_LENGTH(pnew) = 0;
+    pnew->_base.utf8 = NULL;
+    pnew->_base.utf8_length = 0;
+
+    if (PyUnicode_READY(pnew) == -1) {
+        PyObject_FREE(_PyUnicode_WSTR(pnew));
+        goto onError;
+    }
+
     Py_DECREF(tmp);
     return (PyObject *)pnew;
+
+  onError:
+    _Py_ForgetReference((PyObject *)pnew);
+    PyObject_Del(pnew);
+    Py_DECREF(tmp);
+    return err;
 }
 
 PyDoc_STRVAR(unicode_doc,
@@ -10074,7 +12465,7 @@
     int i;
 
     /* XXX - move this array to unicodectype.c ? */
-    Py_UNICODE linebreak[] = {
+    Py_UCS2 linebreak[] = {
         0x000A, /* LINE FEED */
         0x000D, /* CARRIAGE RETURN */
         0x001C, /* FILE SEPARATOR */
@@ -10086,11 +12477,9 @@
     };
 
     /* Init the implementation */
-    free_list = NULL;
-    numfree = 0;
-    unicode_empty = _PyUnicode_New(0);
+    unicode_empty = (PyUnicodeObject *) PyUnicode_New(0, 0);
     if (!unicode_empty)
-        return;
+        Py_FatalError("Can't create empty string");
 
     for (i = 0; i < 256; i++)
         unicode_latin1[i] = NULL;
@@ -10099,8 +12488,8 @@
 
     /* initialize the linebreak bloom filter */
     bloom_linebreak = make_bloom_mask(
-        linebreak, sizeof(linebreak) / sizeof(linebreak[0])
-        );
+        PyUnicode_2BYTE_KIND, linebreak,
+        sizeof(linebreak) / sizeof(linebreak[0]));
 
     PyType_Ready(&EncodingMapType);
 }
@@ -10110,21 +12499,7 @@
 int
 PyUnicode_ClearFreeList(void)
 {
-    int freelist_size = numfree;
-    PyUnicodeObject *u;
-
-    for (u = free_list; u != NULL;) {
-        PyUnicodeObject *v = u;
-        u = *(PyUnicodeObject **)u;
-        if (v->str)
-            PyObject_DEL(v->str);
-        Py_XDECREF(v->defenc);
-        PyObject_Del(v);
-        numfree--;
-    }
-    free_list = NULL;
-    assert(numfree == 0);
-    return freelist_size;
+    return 0;
 }
 
 void
@@ -10158,6 +12533,10 @@
         return;
     if (PyUnicode_CHECK_INTERNED(s))
         return;
+    if (PyUnicode_READY(s) == -1) {
+        assert(0 && "ready fail in intern...");
+        return;
+    }
     if (interned == NULL) {
         interned = PyDict_New();
         if (interned == NULL) {
@@ -10189,15 +12568,17 @@
     /* The two references in interned are not counted by refcnt.
        The deallocator will take care of this */
     Py_REFCNT(s) -= 2;
-    PyUnicode_CHECK_INTERNED(s) = SSTATE_INTERNED_MORTAL;
+    _PyUnicode_STATE(s).interned = SSTATE_INTERNED_MORTAL;
 }
 
 void
 PyUnicode_InternImmortal(PyObject **p)
 {
+    PyUnicodeObject *u = (PyUnicodeObject *)*p;
+
     PyUnicode_InternInPlace(p);
     if (PyUnicode_CHECK_INTERNED(*p) != SSTATE_INTERNED_IMMORTAL) {
-        PyUnicode_CHECK_INTERNED(*p) = SSTATE_INTERNED_IMMORTAL;
+        _PyUnicode_STATE(u).interned = SSTATE_INTERNED_IMMORTAL;
         Py_INCREF(*p);
     }
 }
@@ -10238,22 +12619,24 @@
             n);
     for (i = 0; i < n; i++) {
         s = (PyUnicodeObject *) PyList_GET_ITEM(keys, i);
-        switch (s->state) {
+        if (PyUnicode_READY(s) == -1)
+            fprintf(stderr, "could not ready string\n");
+        switch (PyUnicode_CHECK_INTERNED(s)) {
         case SSTATE_NOT_INTERNED:
             /* XXX Shouldn't happen */
             break;
         case SSTATE_INTERNED_IMMORTAL:
             Py_REFCNT(s) += 1;
-            immortal_size += s->length;
+            immortal_size += PyUnicode_GET_LENGTH(s);
             break;
         case SSTATE_INTERNED_MORTAL:
             Py_REFCNT(s) += 2;
-            mortal_size += s->length;
+            mortal_size += PyUnicode_GET_LENGTH(s);
             break;
         default:
             Py_FatalError("Inconsistent interned string state.");
         }
-        s->state = SSTATE_NOT_INTERNED;
+        _PyUnicode_STATE(s).interned = SSTATE_NOT_INTERNED;
     }
     fprintf(stderr, "total size of all interned strings: "
             "%" PY_FORMAT_SIZE_T "d/%" PY_FORMAT_SIZE_T "d "
@@ -10300,9 +12683,11 @@
         return NULL;
     assert(PyUnicode_Check(seq));
 
-    if (it->it_index < PyUnicode_GET_SIZE(seq)) {
-        item = PyUnicode_FromUnicode(
-            PyUnicode_AS_UNICODE(seq)+it->it_index, 1);
+    if (it->it_index < PyUnicode_GET_LENGTH(seq)) {
+        int kind = PyUnicode_KIND(seq);
+        void *data = PyUnicode_DATA(seq);
+        Py_UCS4 chr = PyUnicode_READ(kind, data, it->it_index);
+        item = PyUnicode_FromOrdinal(chr);
         if (item != NULL)
             ++it->it_index;
         return item;
@@ -10372,6 +12757,8 @@
         PyErr_BadInternalCall();
         return NULL;
     }
+    if (PyUnicode_READY(seq) == -1)
+        return NULL;
     it = PyObject_GC_New(unicodeiterobject, &PyUnicodeIter_Type);
     if (it == NULL)
         return NULL;
@@ -10382,95 +12769,16 @@
     return (PyObject *)it;
 }
 
-size_t
-Py_UNICODE_strlen(const Py_UNICODE *u)
-{
-    int res = 0;
-    while(*u++)
-        res++;
-    return res;
-}
-
-Py_UNICODE*
-Py_UNICODE_strcpy(Py_UNICODE *s1, const Py_UNICODE *s2)
-{
-    Py_UNICODE *u = s1;
-    while ((*u++ = *s2++));
-    return s1;
-}
-
-Py_UNICODE*
-Py_UNICODE_strncpy(Py_UNICODE *s1, const Py_UNICODE *s2, size_t n)
-{
-    Py_UNICODE *u = s1;
-    while ((*u++ = *s2++))
-        if (n-- == 0)
-            break;
-    return s1;
-}
-
-Py_UNICODE*
-Py_UNICODE_strcat(Py_UNICODE *s1, const Py_UNICODE *s2)
-{
-    Py_UNICODE *u1 = s1;
-    u1 += Py_UNICODE_strlen(u1);
-    Py_UNICODE_strcpy(u1, s2);
-    return s1;
-}
-
-int
-Py_UNICODE_strcmp(const Py_UNICODE *s1, const Py_UNICODE *s2)
-{
-    while (*s1 && *s2 && *s1 == *s2)
-        s1++, s2++;
-    if (*s1 && *s2)
-        return (*s1 < *s2) ? -1 : +1;
-    if (*s1)
-        return 1;
-    if (*s2)
-        return -1;
-    return 0;
-}
-
-int
-Py_UNICODE_strncmp(const Py_UNICODE *s1, const Py_UNICODE *s2, size_t n)
-{
-    register Py_UNICODE u1, u2;
-    for (; n != 0; n--) {
-        u1 = *s1;
-        u2 = *s2;
-        if (u1 != u2)
-            return (u1 < u2) ? -1 : +1;
-        if (u1 == '\0')
-            return 0;
-        s1++;
-        s2++;
-    }
-    return 0;
-}
-
-Py_UNICODE*
-Py_UNICODE_strchr(const Py_UNICODE *s, Py_UNICODE c)
-{
-    const Py_UNICODE *p;
-    for (p = s; *p; p++)
-        if (*p == c)
-            return (Py_UNICODE*)p;
-    return NULL;
-}
-
-Py_UNICODE*
-Py_UNICODE_strrchr(const Py_UNICODE *s, Py_UNICODE c)
-{
-    const Py_UNICODE *p;
-    p = s + Py_UNICODE_strlen(s);
-    while (p != s) {
-        p--;
-        if (*p == c)
-            return (Py_UNICODE*)p;
-    }
-    return NULL;
-}
+#define UNIOP(x) Py_UNICODE_##x
+#define UNIOP_t Py_UNICODE
+#include "uniops.h"
+#undef UNIOP
+#undef UNIOP_t
+#define UNIOP(x) Py_UCS4_##x
+#define UNIOP_t Py_UCS4
+#include "uniops.h"
+#undef UNIOP
+#undef UNIOP_t
 
 Py_UNICODE*
 PyUnicode_AsUnicodeCopy(PyObject *object)
@@ -10479,6 +12787,10 @@
     Py_UNICODE *copy;
     Py_ssize_t size;
 
+    if (!PyUnicode_Check(unicode)) {
+        PyErr_BadArgument();
+        return NULL;
+    }
     /* Ensure we won't overflow the size. */
     if (PyUnicode_GET_SIZE(unicode) > ((PY_SSIZE_T_MAX / sizeof(Py_UNICODE)) - 1)) {
         PyErr_NoMemory();
diff --git a/Objects/uniops.h b/Objects/uniops.h
new file mode 100644
--- /dev/null
+++ b/Objects/uniops.h
@@ -0,0 +1,91 @@
+
+size_t
+UNIOP(strlen)(const UNIOP_t *u)
+{
+    int res = 0;
+    while(*u++)
+        res++;
+    return res;
+}
+
+UNIOP_t*
+UNIOP(strcpy)(UNIOP_t *s1, const UNIOP_t *s2)
+{
+    UNIOP_t *u = s1;
+    while ((*u++ = *s2++));
+    return s1;
+}
+
+UNIOP_t*
+UNIOP(strncpy)(UNIOP_t *s1, const UNIOP_t *s2, size_t n)
+{
+    UNIOP_t *u = s1;
+    while ((*u++ = *s2++))
+        if (n-- == 0)
+            break;
+    return s1;
+}
+
+UNIOP_t*
+UNIOP(strcat)(UNIOP_t *s1, const UNIOP_t *s2)
+{
+    UNIOP_t *u1 = s1;
+    u1 += UNIOP(strlen(u1));
+    UNIOP(strcpy(u1, s2));
+    return s1;
+}
+
+int
+UNIOP(strcmp)(const UNIOP_t *s1, const UNIOP_t *s2)
+{
+    while (*s1 && *s2 && *s1 == *s2)
+        s1++, s2++;
+    if (*s1 && *s2)
+        return (*s1 < *s2) ? -1 : +1;
+    if (*s1)
+        return 1;
+    if (*s2)
+        return -1;
+    return 0;
+}
+
+int
+UNIOP(strncmp)(const UNIOP_t *s1, const UNIOP_t *s2, size_t n)
+{
+    register UNIOP_t u1, u2;
+    for (; n != 0; n--) {
+        u1 = *s1;
+        u2 = *s2;
+        if (u1 != u2)
+            return (u1 < u2) ? -1 : +1;
+        if (u1 == '\0')
+            return 0;
+        s1++;
+        s2++;
+    }
+    return 0;
+}
+
+UNIOP_t*
+UNIOP(strchr)(const UNIOP_t *s, UNIOP_t c)
+{
+    const UNIOP_t *p;
+    for (p = s; *p; p++)
+        if (*p == c)
+            return (UNIOP_t*)p;
+    return NULL;
+}
+
+UNIOP_t*
+UNIOP(strrchr)(const UNIOP_t *s, UNIOP_t c)
+{
+    const UNIOP_t *p;
+    p = s + UNIOP(strlen)(s);
+    while (p != s) {
+        p--;
+        if (*p == c)
+            return (UNIOP_t*)p;
+    }
+    return NULL;
+}
+
diff --git a/PC/_subprocess.c b/PC/_subprocess.c
--- a/PC/_subprocess.c
+++ b/PC/_subprocess.c
@@ -329,12 +329,9 @@
 static PyObject*
 getenvironment(PyObject* environment)
 {
-    int i;
-    Py_ssize_t envsize;
-    PyObject* out = NULL;
-    PyObject* keys;
-    PyObject* values;
-    Py_UNICODE* p;
+    Py_ssize_t i, envsize, totalsize;
+    Py_UCS4 *buffer = NULL, *p, *end;
+    PyObject *keys, *values, *res;
 
     /* convert environment dictionary to windows enviroment string */
     if (! PyMapping_Check(environment)) {
@@ -350,14 +347,8 @@
     if (!keys || !values)
         goto error;
 
-    out = PyUnicode_FromUnicode(NULL, 2048);
-    if (! out)
-        goto error;
-
-    p = PyUnicode_AS_UNICODE(out);
-
+    totalsize = 1; /* trailing null character */
     for (i = 0; i < envsize; i++) {
-        Py_ssize_t ksize, vsize, totalsize;
         PyObject* key = PyList_GET_ITEM(keys, i);
         PyObject* value = PyList_GET_ITEM(values, i);
 
@@ -366,36 +357,42 @@
                 "environment can only contain strings");
             goto error;
         }
-        ksize = PyUnicode_GET_SIZE(key);
-        vsize = PyUnicode_GET_SIZE(value);
-        totalsize = (p - PyUnicode_AS_UNICODE(out)) + ksize + 1 +
-                                                     vsize + 1 + 1;
-        if (totalsize > PyUnicode_GET_SIZE(out)) {
-            Py_ssize_t offset = p - PyUnicode_AS_UNICODE(out);
-            PyUnicode_Resize(&out, totalsize + 1024);
-            p = PyUnicode_AS_UNICODE(out) + offset;
-        }
-        Py_UNICODE_COPY(p, PyUnicode_AS_UNICODE(key), ksize);
-        p += ksize;
+        totalsize += PyUnicode_GET_LENGTH(key) + 1;    /* +1 for '=' */
+        totalsize += PyUnicode_GET_LENGTH(value) + 1;  /* +1 for '\0' */
+    }
+
+    buffer = PyMem_Malloc(totalsize * sizeof(Py_UCS4));
+    if (! buffer)
+        goto error;
+    p = buffer;
+    end = buffer + totalsize;
+
+    for (i = 0; i < envsize; i++) {
+        PyObject* key = PyList_GET_ITEM(keys, i);
+        PyObject* value = PyList_GET_ITEM(values, i);
+        if (!PyUnicode_AsUCS4(key, p, end - p, 0))
+            goto error;
+        p += PyUnicode_GET_LENGTH(key);
         *p++ = '=';
-        Py_UNICODE_COPY(p, PyUnicode_AS_UNICODE(value), vsize);
-        p += vsize;
+        if (!PyUnicode_AsUCS4(value, p, end - p, 0))
+            goto error;
+        p += PyUnicode_GET_LENGTH(value);
         *p++ = '\0';
     }
 
     /* add trailing null byte */
     *p++ = '\0';
-    PyUnicode_Resize(&out, p - PyUnicode_AS_UNICODE(out));
-
-    /* PyObject_Print(out, stdout, 0); */
+    assert(p == end);
 
     Py_XDECREF(keys);
     Py_XDECREF(values);
 
-    return out;
+    res = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, buffer, p - buffer);
+    PyMem_Free(buffer);
+    return res;
 
  error:
-    Py_XDECREF(out);
+    PyMem_Free(buffer);
     Py_XDECREF(keys);
     Py_XDECREF(values);
     return NULL;
@@ -609,7 +606,7 @@
     if (! result)
         return PyErr_SetFromWindowsErr(GetLastError());
 
-    return PyUnicode_FromUnicode(filename, Py_UNICODE_strlen(filename));
+    return PyUnicode_FromWideChar(filename, wcslen(filename));
 }
 
 static PyMethodDef sp_functions[] = {
diff --git a/PC/import_nt.c b/PC/import_nt.c
--- a/PC/import_nt.c
+++ b/PC/import_nt.c
@@ -93,7 +93,7 @@
     }
     if (fdp->suffix == NULL)
         return NULL;
-    path = PyUnicode_FromUnicode(pathBuf, wcslen(pathBuf));
+    path = PyUnicode_FromWideChar(pathBuf, wcslen(pathBuf));
     if (path == NULL)
         return NULL;
     fp = _Py_fopen(path, fdp->mode);
diff --git a/PC/msvcrtmodule.c b/PC/msvcrtmodule.c
--- a/PC/msvcrtmodule.c
+++ b/PC/msvcrtmodule.c
@@ -212,7 +212,6 @@
 msvcrt_getwch(PyObject *self, PyObject *args)
 {
     Py_UNICODE ch;
-    Py_UNICODE u[1];
 
     if (!PyArg_ParseTuple(args, ":getwch"))
         return NULL;
@@ -220,8 +219,7 @@
     Py_BEGIN_ALLOW_THREADS
     ch = _getwch();
     Py_END_ALLOW_THREADS
-    u[0] = ch;
-    return PyUnicode_FromUnicode(u, 1);
+    return PyUnicode_FromOrdinal(ch);
 }
 
 PyDoc_STRVAR(getwch_doc,
@@ -257,7 +255,6 @@
 msvcrt_getwche(PyObject *self, PyObject *args)
 {
     Py_UNICODE ch;
-    Py_UNICODE s[1];
 
     if (!PyArg_ParseTuple(args, ":getwche"))
         return NULL;
@@ -265,8 +262,7 @@
     Py_BEGIN_ALLOW_THREADS
     ch = _getwche();
     Py_END_ALLOW_THREADS
-    s[0] = ch;
-    return PyUnicode_FromUnicode(s, 1);
+    return PyUnicode_FromOrdinal(ch);
 }
 
 PyDoc_STRVAR(getwche_doc,
diff --git a/PC/pyconfig.h b/PC/pyconfig.h
--- a/PC/pyconfig.h
+++ b/PC/pyconfig.h
@@ -550,10 +550,6 @@
 /* Define if you want to use the GNU readline library */
 /* #define WITH_READLINE 1 */
 
-/* Define as the size of the unicode type. */
-/* This is enough for unicodeobject.h to do the "right thing" on Windows. */
-#define Py_UNICODE_SIZE 2
-
 /* Use Python's own small-block memory-allocator. */
 #define WITH_PYMALLOC 1
 
diff --git a/PC/winreg.c b/PC/winreg.c
--- a/PC/winreg.c
+++ b/PC/winreg.c
@@ -882,7 +882,7 @@
                     retDataSize -= 2;
                 if (retDataSize <= 0)
                     data = L"";
-                obData = PyUnicode_FromUnicode(data, retDataSize/2);
+                obData = PyUnicode_FromWideChar(data, retDataSize/2);
                 break;
             }
         case REG_MULTI_SZ:
@@ -913,7 +913,7 @@
                     }
                     PyList_SetItem(obData,
                                    index,
-                                   PyUnicode_FromUnicode(str[index], len));
+                                   PyUnicode_FromWideChar(str[index], len));
                 }
                 free(str);
 
@@ -1123,7 +1123,7 @@
     if (rc != ERROR_SUCCESS)
         return PyErr_SetFromWindowsErrWithFunction(rc, "RegEnumKeyEx");
 
-    retStr = PyUnicode_FromUnicode(tmpbuf, len);
+    retStr = PyUnicode_FromWideChar(tmpbuf, len);
     return retStr;  /* can be NULL */
 }
 
@@ -1394,7 +1394,7 @@
                                                    "RegQueryValue");
     }
 
-    retStr = PyUnicode_FromUnicode(retBuf, wcslen(retBuf));
+    retStr = PyUnicode_FromWideChar(retBuf, wcslen(retBuf));
     PyMem_Free(retBuf);
     return retStr;
 }
diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c
--- a/Parser/tokenizer.c
+++ b/Parser/tokenizer.c
@@ -1258,14 +1258,16 @@
 #ifdef PGEN
 #define verify_identifier(tok) 1
 #else
-/* Verify that the identifier follows PEP 3131. */
+/* Verify that the identifier follows PEP 3131.
+   All identifier strings are guaranteed to be "ready" unicode objects.
+ */
 static int
 verify_identifier(struct tok_state *tok)
 {
     PyObject *s;
     int result;
     s = PyUnicode_DecodeUTF8(tok->start, tok->cur - tok->start, NULL);
-    if (s == NULL) {
+    if (s == NULL || PyUnicode_READY(s) == -1) {
         if (PyErr_ExceptionMatches(PyExc_UnicodeDecodeError)) {
             PyErr_Clear();
             tok->done = E_IDENTIFIER;
diff --git a/Python/_warnings.c b/Python/_warnings.c
--- a/Python/_warnings.c
+++ b/Python/_warnings.c
@@ -498,17 +498,19 @@
     *filename = PyDict_GetItemString(globals, "__file__");
     if (*filename != NULL && PyUnicode_Check(*filename)) {
         Py_ssize_t len = PyUnicode_GetSize(*filename);
-        Py_UNICODE *unicode = PyUnicode_AS_UNICODE(*filename);
+        int kind = PyUnicode_KIND(*filename);
+        void *data = PyUnicode_DATA(*filename);
 
         /* if filename.lower().endswith((".pyc", ".pyo")): */
         if (len >= 4 &&
-            unicode[len-4] == '.' &&
-            Py_UNICODE_TOLOWER(unicode[len-3]) == 'p' &&
-            Py_UNICODE_TOLOWER(unicode[len-2]) == 'y' &&
-            (Py_UNICODE_TOLOWER(unicode[len-1]) == 'c' ||
-                Py_UNICODE_TOLOWER(unicode[len-1]) == 'o'))
+            PyUnicode_READ(kind, data, len-4) == '.' &&
+            Py_UNICODE_TOLOWER(PyUnicode_READ(kind, data, len-3)) == 'p' &&
+            Py_UNICODE_TOLOWER(PyUnicode_READ(kind, data, len-2)) == 'y' &&
+            (Py_UNICODE_TOLOWER(PyUnicode_READ(kind, data, len-1)) == 'c' ||
+                Py_UNICODE_TOLOWER(PyUnicode_READ(kind, data, len-1)) == 'o'))
         {
-            *filename = PyUnicode_FromUnicode(unicode, len-1);
+            *filename = PyUnicode_Substring(*filename, 0,
+                                            PyUnicode_GET_LENGTH(*filename)-1);
             if (*filename == NULL)
                 goto handle_error;
         }
diff --git a/Python/ast.c b/Python/ast.c
--- a/Python/ast.c
+++ b/Python/ast.c
@@ -528,26 +528,21 @@
 new_identifier(const char* n, PyArena *arena)
 {
     PyObject* id = PyUnicode_DecodeUTF8(n, strlen(n), NULL);
-    Py_UNICODE *u;
-    if (!id)
+    if (!id || PyUnicode_READY(id) == -1)
         return NULL;
-    u = PyUnicode_AS_UNICODE(id);
     /* Check whether there are non-ASCII characters in the
        identifier; if so, normalize to NFKC. */
-    for (; *u; u++) {
-        if (*u >= 128) {
-            PyObject *m = PyImport_ImportModuleNoBlock("unicodedata");
-            PyObject *id2;
-            if (!m)
-                return NULL;
-            id2 = PyObject_CallMethod(m, "normalize", "sO", "NFKC", id);
-            Py_DECREF(m);
-            if (!id2)
-                return NULL;
-            Py_DECREF(id);
-            id = id2;
-            break;
-        }
+    if (PyUnicode_MAX_CHAR_VALUE((PyUnicodeObject *)id) >= 128) {
+        PyObject *m = PyImport_ImportModuleNoBlock("unicodedata");
+        PyObject *id2;
+        if (!m)
+            return NULL;
+        id2 = PyObject_CallMethod(m, "normalize", "sO", "NFKC", id);
+        Py_DECREF(m);
+        if (!id2)
+            return NULL;
+        Py_DECREF(id);
+        id = id2;
     }
     PyUnicode_InternInPlace(&id);
     PyArena_AddPyObject(arena, id);
@@ -3660,20 +3655,14 @@
 }
 
 static PyObject *
-decode_utf8(struct compiling *c, const char **sPtr, const char *end, char* encoding)
+decode_utf8(struct compiling *c, const char **sPtr, const char *end)
 {
-    PyObject *u, *v;
     char *s, *t;
     t = s = (char *)*sPtr;
     /* while (s < end && *s != '\\') s++; */ /* inefficient for u".." */
     while (s < end && (*s & 0x80)) s++;
     *sPtr = s;
-    u = PyUnicode_DecodeUTF8(t, s - t, NULL);
-    if (u == NULL)
-        return NULL;
-    v = PyUnicode_AsEncodedString(u, encoding, NULL);
-    Py_DECREF(u);
-    return v;
+    return PyUnicode_DecodeUTF8(t, s - t, NULL);
 }
 
 static PyObject *
@@ -3707,22 +3696,20 @@
             }
             if (*s & 0x80) { /* XXX inefficient */
                 PyObject *w;
-                char *r;
-                Py_ssize_t rn, i;
-                w = decode_utf8(c, &s, end, "utf-32-be");
+                int kind;
+                void *data;
+                Py_ssize_t len, i;
+                w = decode_utf8(c, &s, end);
                 if (w == NULL) {
                     Py_DECREF(u);
                     return NULL;
                 }
-                r = PyBytes_AS_STRING(w);
-                rn = Py_SIZE(w);
-                assert(rn % 4 == 0);
-                for (i = 0; i < rn; i += 4) {
-                    sprintf(p, "\\U%02x%02x%02x%02x",
-                            r[i + 0] & 0xFF,
-                            r[i + 1] & 0xFF,
-                            r[i + 2] & 0xFF,
-                            r[i + 3] & 0xFF);
+                kind = PyUnicode_KIND(w);
+                data = PyUnicode_DATA(w);
+                len = PyUnicode_GET_LENGTH(w);
+                for (i = 0; i < len; i++) {
+                    Py_UCS4 chr = PyUnicode_READ(kind, data, i);
+                    sprintf(p, "\\U%08x", chr);
                     p += 10;
                 }
                 /* Should be impossible to overflow */
diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c
--- a/Python/bltinmodule.c
+++ b/Python/bltinmodule.c
@@ -508,8 +508,8 @@
 
     if (PyUnicode_Check(cmd)) {
         cf->cf_flags |= PyCF_IGNORE_COOKIE;
-        cmd = _PyUnicode_AsDefaultEncodedString(cmd);
-        if (cmd == NULL)
+        str = PyUnicode_AsUTF8AndSize(cmd, &size);
+        if (str == NULL)
             return NULL;
     }
     else if (!PyObject_CheckReadBuffer(cmd)) {
@@ -518,9 +518,10 @@
           funcname, what);
         return NULL;
     }
-    if (PyObject_AsReadBuffer(cmd, (const void **)&str, &size) < 0) {
+    else if (PyObject_AsReadBuffer(cmd, (const void **)&str, &size) < 0) {
         return NULL;
     }
+
     if (strlen(str) != size) {
         PyErr_SetString(PyExc_TypeError,
                         "source code string cannot contain null bytes");
@@ -1395,24 +1396,13 @@
         }
     }
     else if (PyUnicode_Check(obj)) {
-        size = PyUnicode_GET_SIZE(obj);
+        if (PyUnicode_READY(obj) == -1)
+            return NULL;
+        size = PyUnicode_GET_LENGTH(obj);
         if (size == 1) {
-            ord = (long)*PyUnicode_AS_UNICODE(obj);
+            ord = (long)PyUnicode_READ_CHAR(obj, 0);
             return PyLong_FromLong(ord);
         }
-#ifndef Py_UNICODE_WIDE
-        if (size == 2) {
-            /* Decode a valid surrogate pair */
-            int c0 = PyUnicode_AS_UNICODE(obj)[0];
-            int c1 = PyUnicode_AS_UNICODE(obj)[1];
-            if (0xD800 <= c0 && c0 <= 0xDBFF &&
-                0xDC00 <= c1 && c1 <= 0xDFFF) {
-                ord = ((((c0 & 0x03FF) << 10) | (c1 & 0x03FF)) +
-                       0x00010000);
-                return PyLong_FromLong(ord);
-            }
-        }
-#endif
     }
     else if (PyByteArray_Check(obj)) {
         /* XXX Hopefully this is temporary */
diff --git a/Python/ceval.c b/Python/ceval.c
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -2054,7 +2054,7 @@
                 /* Inline the PyDict_GetItem() calls.
                    WARNING: this is an extreme speed hack.
                    Do not try this at home. */
-                Py_hash_t hash = ((PyUnicodeObject *)w)->hash;
+                Py_hash_t hash = ((PyASCIIObject *)w)->hash;
                 if (hash != -1) {
                     PyDictObject *d;
                     PyDictEntry *e;
@@ -4456,7 +4456,8 @@
         }
         if (skip_leading_underscores &&
             PyUnicode_Check(name) &&
-            PyUnicode_AS_UNICODE(name)[0] == '_')
+            PyUnicode_READY(name) != -1 &&
+            PyUnicode_READ_CHAR(name, 0) == '_')
         {
             Py_DECREF(name);
             continue;
@@ -4520,6 +4521,14 @@
 {
     /* This function implements 'variable += expr' when both arguments
        are (Unicode) strings. */
+
+    w = PyUnicode_Concat(v, w);
+    Py_DECREF(v);
+    return w;
+
+    /* XXX: This optimization is currently disabled as unicode objects in the
+       new flexible representation are not in-place resizable anymore. */
+#if 0
     Py_ssize_t v_len = PyUnicode_GET_SIZE(v);
     Py_ssize_t w_len = PyUnicode_GET_SIZE(w);
     Py_ssize_t new_len = v_len + w_len;
@@ -4570,7 +4579,8 @@
         }
     }
 
-    if (Py_REFCNT(v) == 1 && !PyUnicode_CHECK_INTERNED(v)) {
+    if (Py_REFCNT(v) == 1 && !PyUnicode_CHECK_INTERNED(v) &&
+        !PyUnicode_IS_COMPACT((PyUnicodeObject *)v)) {
         /* Now we own the last reference to 'v', so we can resize it
          * in-place.
          */
@@ -4594,6 +4604,7 @@
         Py_DECREF(v);
         return w;
     }
+#endif
 }
 
 #ifdef DYNAMIC_EXECUTION_PROFILE
diff --git a/Python/codecs.c b/Python/codecs.c
--- a/Python/codecs.c
+++ b/Python/codecs.c
@@ -513,27 +513,25 @@
 
 PyObject *PyCodec_ReplaceErrors(PyObject *exc)
 {
-    PyObject *restuple;
-    Py_ssize_t start;
-    Py_ssize_t end;
-    Py_ssize_t i;
+    Py_ssize_t start, end, i, len;
 
     if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
         PyObject *res;
-        Py_UNICODE *p;
+        int kind;
+        void *data;
         if (PyUnicodeEncodeError_GetStart(exc, &start))
             return NULL;
         if (PyUnicodeEncodeError_GetEnd(exc, &end))
             return NULL;
-        res = PyUnicode_FromUnicode(NULL, end-start);
+        len = end - start;
+        res = PyUnicode_New(len, '?');
         if (res == NULL)
             return NULL;
-        for (p = PyUnicode_AS_UNICODE(res), i = start;
-            i<end; ++p, ++i)
-            *p = '?';
-        restuple = Py_BuildValue("(On)", res, end);
-        Py_DECREF(res);
-        return restuple;
+        kind = PyUnicode_KIND(res);
+        data = PyUnicode_DATA(res);
+        for (i = 0; i < len; ++i)
+            PyUnicode_WRITE(kind, data, i, '?');
+        return Py_BuildValue("(Nn)", res, end);
     }
     else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
         Py_UNICODE res = Py_UNICODE_REPLACEMENT_CHARACTER;
@@ -543,20 +541,21 @@
     }
     else if (PyObject_IsInstance(exc, PyExc_UnicodeTranslateError)) {
         PyObject *res;
-        Py_UNICODE *p;
+        int kind;
+        void *data;
         if (PyUnicodeTranslateError_GetStart(exc, &start))
             return NULL;
         if (PyUnicodeTranslateError_GetEnd(exc, &end))
             return NULL;
-        res = PyUnicode_FromUnicode(NULL, end-start);
+        len = end - start;
+        res = PyUnicode_New(len, Py_UNICODE_REPLACEMENT_CHARACTER);
         if (res == NULL)
             return NULL;
-        for (p = PyUnicode_AS_UNICODE(res), i = start;
-            i<end; ++p, ++i)
-            *p = Py_UNICODE_REPLACEMENT_CHARACTER;
-        restuple = Py_BuildValue("(On)", res, end);
-        Py_DECREF(res);
-        return restuple;
+        kind = PyUnicode_KIND(res);
+        data = PyUnicode_DATA(res);
+        for (i=0; i < len; i++)
+            PyUnicode_WRITE(kind, data, i, Py_UNICODE_REPLACEMENT_CHARACTER);
+        return Py_BuildValue("(Nn)", res, end);
     }
     else {
         wrong_exception_type(exc);
@@ -671,10 +670,7 @@
     }
 }
 
-static Py_UNICODE hexdigits[] = {
-    '0', '1', '2', '3', '4', '5', '6', '7',
-    '8', '9', 'a', 'b', 'c', 'd', 'e', 'f'
-};
+static const char *hexdigits = "0123456789abcdef";
 
 PyObject *PyCodec_BackslashReplaceErrors(PyObject *exc)
 {
diff --git a/Python/compile.c b/Python/compile.c
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -197,16 +197,17 @@
 {
     /* Name mangling: __private becomes _classname__private.
        This is independent from how the name is used. */
-    const Py_UNICODE *p, *name = PyUnicode_AS_UNICODE(ident);
-    Py_UNICODE *buffer;
-    size_t nlen, plen;
+    PyObject *result;
+    size_t nlen, plen, ipriv;
+    Py_UCS4 maxchar;
     if (privateobj == NULL || !PyUnicode_Check(privateobj) ||
-        name == NULL || name[0] != '_' || name[1] != '_') {
+        PyUnicode_READ_CHAR(ident, 0) != '_' ||
+        PyUnicode_READ_CHAR(ident, 1) != '_') {
         Py_INCREF(ident);
         return ident;
     }
-    p = PyUnicode_AS_UNICODE(privateobj);
-    nlen = Py_UNICODE_strlen(name);
+    nlen = PyUnicode_GET_LENGTH(ident);
+    plen = PyUnicode_GET_LENGTH(privateobj);
     /* Don't mangle __id__ or names with dots.
 
        The only time a name with a dot can occur is when
@@ -216,32 +217,37 @@
        TODO(jhylton): Decide whether we want to support
        mangling of the module name, e.g. __M.X.
     */
-    if ((name[nlen-1] == '_' && name[nlen-2] == '_')
-        || Py_UNICODE_strchr(name, '.')) {
+    if ((PyUnicode_READ_CHAR(ident, nlen-1) == '_' &&
+         PyUnicode_READ_CHAR(ident, nlen-2) == '_') ||
+        PyUnicode_FindChar(ident, '.', 0, nlen, 1) != -1) {
         Py_INCREF(ident);
         return ident; /* Don't mangle __whatever__ */
     }
     /* Strip leading underscores from class name */
-    while (*p == '_')
-        p++;
-    if (*p == 0) {
+    ipriv = 0;
+    while (PyUnicode_READ_CHAR(privateobj, ipriv) == '_')
+        ipriv++;
+    if (ipriv == plen) {
         Py_INCREF(ident);
         return ident; /* Don't mangle if class is just underscores */
     }
-    plen = Py_UNICODE_strlen(p);
+    plen -= ipriv;
 
     assert(1 <= PY_SSIZE_T_MAX - nlen);
     assert(1 + nlen <= PY_SSIZE_T_MAX - plen);
 
-    ident = PyUnicode_FromStringAndSize(NULL, 1 + nlen + plen);
-    if (!ident)
+    maxchar = PyUnicode_MAX_CHAR_VALUE(ident);
+    if (PyUnicode_MAX_CHAR_VALUE(privateobj) > maxchar)
+        maxchar = PyUnicode_MAX_CHAR_VALUE(privateobj);
+
+    result = PyUnicode_New(1 + nlen + plen, maxchar);
+    if (!result)
         return 0;
-    /* ident = "_" + p[:plen] + name # i.e. 1+plen+nlen bytes */
-    buffer = PyUnicode_AS_UNICODE(ident);
-    buffer[0] = '_';
-    Py_UNICODE_strncpy(buffer+1, p, plen);
-    Py_UNICODE_strcpy(buffer+1+plen, name);
-    return ident;
+    /* ident = "_" + priv[ipriv:] + ident # i.e. 1+plen+nlen bytes */
+    PyUnicode_WRITE(PyUnicode_KIND(result), PyUnicode_DATA(result), 0, '_');
+    PyUnicode_CopyCharacters(result, 1, privateobj, ipriv, plen);
+    PyUnicode_CopyCharacters(result, plen+1, ident, 0, nlen);
+    return result;
 }
 
 static int
@@ -2085,22 +2091,27 @@
        If there is a dot in name, we need to split it and emit a
        LOAD_ATTR for each name.
     */
-    const Py_UNICODE *src = PyUnicode_AS_UNICODE(name);
-    const Py_UNICODE *dot = Py_UNICODE_strchr(src, '.');
-    if (dot) {
+    Py_ssize_t dot = PyUnicode_FindChar(name, '.', 0,
+                                        PyUnicode_GET_LENGTH(name), 1);
+    if (dot == -2)
+        return -1;
+    if (dot != -1) {
         /* Consume the base module name to get the first attribute */
-        src = dot + 1;
-        while (dot) {
-            /* NB src is only defined when dot != NULL */
+        Py_ssize_t pos = dot + 1;
+        while (dot != -1) {
             PyObject *attr;
-            dot = Py_UNICODE_strchr(src, '.');
-            attr = PyUnicode_FromUnicode(src,
-                                dot ? dot - src : Py_UNICODE_strlen(src));
+            dot = PyUnicode_FindChar(name, '.', pos,
+                                     PyUnicode_GET_LENGTH(name), 1);
+            if (dot == -2)
+                return -1;
+            attr = PyUnicode_Substring(name, pos,
+                                       (dot != -1) ? dot :
+                                       PyUnicode_GET_LENGTH(name));
             if (!attr)
                 return -1;
             ADDOP_O(c, LOAD_ATTR, attr, names);
             Py_DECREF(attr);
-            src = dot + 1;
+            pos = dot + 1;
         }
     }
     return compiler_nameop(c, asname, Store);
@@ -2139,13 +2150,12 @@
         }
         else {
             identifier tmp = alias->name;
-            const Py_UNICODE *base = PyUnicode_AS_UNICODE(alias->name);
-            Py_UNICODE *dot = Py_UNICODE_strchr(base, '.');
-            if (dot)
-                tmp = PyUnicode_FromUnicode(base,
-                                            dot - base);
+            Py_ssize_t dot = PyUnicode_FindChar(
+                alias->name, '.', 0, PyUnicode_GET_LENGTH(alias->name), 1);
+            if (dot != -1)
+                tmp = PyUnicode_Substring(alias->name, 0, dot);
             r = compiler_nameop(c, tmp, Store);
-            if (dot) {
+            if (dot != -1) {
                 Py_DECREF(tmp);
             }
             if (!r)
@@ -2208,7 +2218,7 @@
         alias_ty alias = (alias_ty)asdl_seq_GET(s->v.ImportFrom.names, i);
         identifier store_name;
 
-        if (i == 0 && *PyUnicode_AS_UNICODE(alias->name) == '*') {
+        if (i == 0 && PyUnicode_READ_CHAR(alias->name, 0) == '*') {
             assert(n == 1);
             ADDOP(c, IMPORT_STAR);
             return 1;
@@ -2522,7 +2532,7 @@
     }
 
     /* XXX Leave assert here, but handle __doc__ and the like better */
-    assert(scope || PyUnicode_AS_UNICODE(name)[0] == '_');
+    assert(scope || PyUnicode_READ_CHAR(name, 0) == '_');
 
     switch (optype) {
     case OP_DEREF:
@@ -3045,8 +3055,7 @@
         return PyObject_IsTrue(e->v.Str.s);
     case Name_kind:
         /* optimize away names that can't be reassigned */
-        id = PyBytes_AS_STRING(
-            _PyUnicode_AsDefaultEncodedString(e->v.Name.id));
+        id = PyUnicode_AsUTF8(e->v.Name.id);
         if (strcmp(id, "True") == 0) return 1;
         if (strcmp(id, "False") == 0) return 0;
         if (strcmp(id, "None") == 0) return 0;
diff --git a/Python/errors.c b/Python/errors.c
--- a/Python/errors.c
+++ b/Python/errors.c
@@ -395,7 +395,7 @@
                 /* remove trailing cr/lf and dots */
                 while (len > 0 && (s_buf[len-1] <= L' ' || s_buf[len-1] == L'.'))
                     s_buf[--len] = L'\0';
-                message = PyUnicode_FromUnicode(s_buf, len);
+                message = PyUnicode_FromWideChar(s_buf, len);
             }
         }
     }
@@ -487,7 +487,7 @@
         /* remove trailing cr/lf and dots */
         while (len > 0 && (s_buf[len-1] <= L' ' || s_buf[len-1] == L'.'))
             s_buf[--len] = L'\0';
-        message = PyUnicode_FromUnicode(s_buf, len);
+        message = PyUnicode_FromWideChar(s_buf, len);
     }
 
     if (message == NULL)
diff --git a/Python/formatter_unicode.c b/Python/formatter_unicode.c
--- a/Python/formatter_unicode.c
+++ b/Python/formatter_unicode.c
@@ -3,12 +3,1445 @@
    of int.__float__, etc., that take and return unicode objects */
 
 #include "Python.h"
-#include "../Objects/stringlib/unicodedefs.h"
+#include <locale.h>
 
+/* Raises an exception about an unknown presentation type for this
+ * type. */
 
-#define FORMAT_STRING  _PyUnicode_FormatAdvanced
-#define FORMAT_LONG    _PyLong_FormatAdvanced
-#define FORMAT_FLOAT   _PyFloat_FormatAdvanced
-#define FORMAT_COMPLEX _PyComplex_FormatAdvanced
+static void
+unknown_presentation_type(Py_UCS4 presentation_type,
+                          const char* type_name)
+{
+    /* %c might be out-of-range, hence the two cases. */
+    if (presentation_type > 32 && presentation_type < 128)
+        PyErr_Format(PyExc_ValueError,
+                     "Unknown format code '%c' "
+                     "for object of type '%.200s'",
+                     (char)presentation_type,
+                     type_name);
+    else
+        PyErr_Format(PyExc_ValueError,
+                     "Unknown format code '\\x%x' "
+                     "for object of type '%.200s'",
+                     (unsigned int)presentation_type,
+                     type_name);
+}
 
-#include "../Objects/stringlib/formatter.h"
+static void
+invalid_comma_type(Py_UCS4 presentation_type)
+{
+    if (presentation_type > 32 && presentation_type < 128)
+        PyErr_Format(PyExc_ValueError,
+                     "Cannot specify ',' with '%c'.",
+                     (char)presentation_type);
+    else
+        PyErr_Format(PyExc_ValueError,
+                     "Cannot specify ',' with '\\x%x'.",
+                     (unsigned int)presentation_type);
+}
+
+/*
+    get_integer consumes 0 or more decimal digit characters from an
+    input string, updates *result with the corresponding positive
+    integer, and returns the number of digits consumed.
+
+    returns -1 on error.
+*/
+static int
+get_integer(PyObject *str, Py_ssize_t *pos, Py_ssize_t end,
+                  Py_ssize_t *result)
+{
+    Py_ssize_t accumulator, digitval, oldaccumulator;
+    int numdigits;
+    accumulator = numdigits = 0;
+    for (;;(*pos)++, numdigits++) {
+        if (*pos >= end)
+            break;
+        digitval = Py_UNICODE_TODECIMAL(PyUnicode_READ_CHAR(str, *pos));
+        if (digitval < 0)
+            break;
+        /*
+           This trick was copied from old Unicode format code.  It's cute,
+           but would really suck on an old machine with a slow divide
+           implementation.  Fortunately, in the normal case we do not
+           expect too many digits.
+        */
+        oldaccumulator = accumulator;
+        accumulator *= 10;
+        if ((accumulator+10)/10 != oldaccumulator+1) {
+            PyErr_Format(PyExc_ValueError,
+                         "Too many decimal digits in format string");
+            return -1;
+        }
+        accumulator += digitval;
+    }
+    *result = accumulator;
+    return numdigits;
+}
+
+/************************************************************************/
+/*********** standard format specifier parsing **************************/
+/************************************************************************/
+
+/* returns true if this character is a specifier alignment token */
+Py_LOCAL_INLINE(int)
+is_alignment_token(Py_UCS4 c)
+{
+    switch (c) {
+    case '<': case '>': case '=': case '^':
+        return 1;
+    default:
+        return 0;
+    }
+}
+
+/* returns true if this character is a sign element */
+Py_LOCAL_INLINE(int)
+is_sign_element(Py_UCS4 c)
+{
+    switch (c) {
+    case ' ': case '+': case '-':
+        return 1;
+    default:
+        return 0;
+    }
+}
+
+
+typedef struct {
+    Py_UCS4 fill_char;
+    Py_UCS4 align;
+    int alternate;
+    Py_UCS4 sign;
+    Py_ssize_t width;
+    int thousands_separators;
+    Py_ssize_t precision;
+    Py_UCS4 type;
+} InternalFormatSpec;
+
+#if 0
+/* Occassionally useful for debugging. Should normally be commented out. */
+static void
+DEBUG_PRINT_FORMAT_SPEC(InternalFormatSpec *format)
+{
+    printf("internal format spec: fill_char %d\n", format->fill_char);
+    printf("internal format spec: align %d\n", format->align);
+    printf("internal format spec: alternate %d\n", format->alternate);
+    printf("internal format spec: sign %d\n", format->sign);
+    printf("internal format spec: width %zd\n", format->width);
+    printf("internal format spec: thousands_separators %d\n",
+           format->thousands_separators);
+    printf("internal format spec: precision %zd\n", format->precision);
+    printf("internal format spec: type %c\n", format->type);
+    printf("\n");
+}
+#endif
+
+
+/*
+  ptr points to the start of the format_spec, end points just past its end.
+  fills in format with the parsed information.
+  returns 1 on success, 0 on failure.
+  if failure, sets the exception
+*/
+static int
+parse_internal_render_format_spec(PyObject *format_spec,
+                                  Py_ssize_t start, Py_ssize_t end,
+                                  InternalFormatSpec *format,
+                                  char default_type,
+                                  char default_align)
+{
+    Py_ssize_t pos = start;
+    /* end-pos is used throughout this code to specify the length of
+       the input string */
+#define READ_spec(index) PyUnicode_READ_CHAR(format_spec, index)
+
+    Py_ssize_t consumed;
+    int align_specified = 0;
+
+    format->fill_char = '\0';
+    format->align = default_align;
+    format->alternate = 0;
+    format->sign = '\0';
+    format->width = -1;
+    format->thousands_separators = 0;
+    format->precision = -1;
+    format->type = default_type;
+
+    /* If the second char is an alignment token,
+       then parse the fill char */
+    if (end-pos >= 2 && is_alignment_token(READ_spec(pos+1))) {
+        format->align = READ_spec(pos+1);
+        format->fill_char = READ_spec(pos);
+        align_specified = 1;
+        pos += 2;
+    }
+    else if (end-pos >= 1 && is_alignment_token(READ_spec(pos))) {
+        format->align = READ_spec(pos);
+        align_specified = 1;
+        ++pos;
+    }
+
+    /* Parse the various sign options */
+    if (end-pos >= 1 && is_sign_element(READ_spec(pos))) {
+        format->sign = READ_spec(pos);
+        ++pos;
+    }
+
+    /* If the next character is #, we're in alternate mode.  This only
+       applies to integers. */
+    if (end-pos >= 1 && READ_spec(pos) == '#') {
+        format->alternate = 1;
+        ++pos;
+    }
+
+    /* The special case for 0-padding (backwards compat) */
+    if (format->fill_char == '\0' && end-pos >= 1 && READ_spec(pos) == '0') {
+        format->fill_char = '0';
+        if (!align_specified) {
+            format->align = '=';
+        }
+        ++pos;
+    }
+
+    consumed = get_integer(format_spec, &pos, end, &format->width);
+    if (consumed == -1)
+        /* Overflow error. Exception already set. */
+        return 0;
+
+    /* If consumed is 0, we didn't consume any characters for the
+       width. In that case, reset the width to -1, because
+       get_integer() will have set it to zero. -1 is how we record
+       that the width wasn't specified. */
+    if (consumed == 0)
+        format->width = -1;
+
+    /* Comma signifies add thousands separators */
+    if (end-pos && READ_spec(pos) == ',') {
+        format->thousands_separators = 1;
+        ++pos;
+    }
+
+    /* Parse field precision */
+    if (end-pos && READ_spec(pos) == '.') {
+        ++pos;
+
+        consumed = get_integer(format_spec, &pos, end, &format->precision);
+        if (consumed == -1)
+            /* Overflow error. Exception already set. */
+            return 0;
+
+        /* Not having a precision after a dot is an error. */
+        if (consumed == 0) {
+            PyErr_Format(PyExc_ValueError,
+                         "Format specifier missing precision");
+            return 0;
+        }
+
+    }
+
+    /* Finally, parse the type field. */
+
+    if (end-pos > 1) {
+        /* More than one char remain, invalid conversion spec. */
+        PyErr_Format(PyExc_ValueError, "Invalid conversion specification");
+        return 0;
+    }
+
+    if (end-pos == 1) {
+        format->type = READ_spec(pos);
+        ++pos;
+    }
+
+    /* Do as much validating as we can, just by looking at the format
+       specifier.  Do not take into account what type of formatting
+       we're doing (int, float, string). */
+
+    if (format->thousands_separators) {
+        switch (format->type) {
+        case 'd':
+        case 'e':
+        case 'f':
+        case 'g':
+        case 'E':
+        case 'G':
+        case '%':
+        case 'F':
+        case '\0':
+            /* These are allowed. See PEP 378.*/
+            break;
+        default:
+            invalid_comma_type(format->type);
+            return 0;
+        }
+    }
+
+    if (format->fill_char > 127 || format->align > 127 ||
+        format->sign > 127) {
+        PyErr_SetString(PyExc_ValueError, "fill character too large");
+        return 0;
+    }
+
+    return 1;
+}
+
+/* Calculate the padding needed. */
+static void
+calc_padding(Py_ssize_t nchars, Py_ssize_t width, Py_UCS4 align,
+             Py_ssize_t *n_lpadding, Py_ssize_t *n_rpadding,
+             Py_ssize_t *n_total)
+{
+    if (width >= 0) {
+        if (nchars > width)
+            *n_total = nchars;
+        else
+            *n_total = width;
+    }
+    else {
+        /* not specified, use all of the chars and no more */
+        *n_total = nchars;
+    }
+
+    /* Figure out how much leading space we need, based on the
+       aligning */
+    if (align == '>')
+        *n_lpadding = *n_total - nchars;
+    else if (align == '^')
+        *n_lpadding = (*n_total - nchars) / 2;
+    else if (align == '<' || align == '=')
+        *n_lpadding = 0;
+    else {
+        /* We should never have an unspecified alignment. */
+        *n_lpadding = 0;
+        assert(0);
+    }
+
+    *n_rpadding = *n_total - nchars - *n_lpadding;
+}
+
+static void
+unicode_fill(PyObject *str, Py_ssize_t start, Py_ssize_t end, Py_UCS4 ch)
+{
+    int kind = PyUnicode_KIND(str);
+    void *data = PyUnicode_DATA(str);
+    while (start < end)
+        PyUnicode_WRITE(kind, data, start++, ch);
+}
+
+/* Do the padding, and return a pointer to where the caller-supplied
+   content goes. */
+static Py_ssize_t
+fill_padding(PyObject *s, Py_ssize_t start, Py_ssize_t nchars,
+             Py_UCS4 fill_char, Py_ssize_t n_lpadding,
+             Py_ssize_t n_rpadding)
+{
+    /* Pad on left. */
+    if (n_lpadding)
+        unicode_fill(s, start, start + n_lpadding, fill_char);
+
+    /* Pad on right. */
+    if (n_rpadding)
+        unicode_fill(s, start + nchars + n_lpadding, 
+                     start + nchars + n_lpadding + n_rpadding, fill_char);
+
+    /* Pointer to the user content. */
+    return start + n_lpadding;
+}
+
+/************************************************************************/
+/*********** common routines for numeric formatting *********************/
+/************************************************************************/
+
+/* Locale type codes. */
+#define LT_CURRENT_LOCALE 0
+#define LT_DEFAULT_LOCALE 1
+#define LT_NO_LOCALE 2
+
+/* Locale info needed for formatting integers and the part of floats
+   before and including the decimal. Note that locales only support
+   8-bit chars, not unicode. */
+typedef struct {
+    char *decimal_point;
+    char *thousands_sep;
+    char *grouping;
+} LocaleInfo;
+
+/* describes the layout for an integer, see the comment in
+   calc_number_widths() for details */
+typedef struct {
+    Py_ssize_t n_lpadding;
+    Py_ssize_t n_prefix;
+    Py_ssize_t n_spadding;
+    Py_ssize_t n_rpadding;
+    char sign;
+    Py_ssize_t n_sign;      /* number of digits needed for sign (0/1) */
+    Py_ssize_t n_grouped_digits; /* Space taken up by the digits, including
+                                    any grouping chars. */
+    Py_ssize_t n_decimal;   /* 0 if only an integer */
+    Py_ssize_t n_remainder; /* Digits in decimal and/or exponent part,
+                               excluding the decimal itself, if
+                               present. */
+
+    /* These 2 are not the widths of fields, but are needed by
+       STRINGLIB_GROUPING. */
+    Py_ssize_t n_digits;    /* The number of digits before a decimal
+                               or exponent. */
+    Py_ssize_t n_min_width; /* The min_width we used when we computed
+                               the n_grouped_digits width. */
+} NumberFieldWidths;
+
+
+/* Given a number of the form:
+   digits[remainder]
+   where ptr points to the start and end points to the end, find where
+    the integer part ends. This could be a decimal, an exponent, both,
+    or neither.
+   If a decimal point is present, set *has_decimal and increment
+    remainder beyond it.
+   Results are undefined (but shouldn't crash) for improperly
+    formatted strings.
+*/
+static void
+parse_number(PyObject *s, Py_ssize_t pos, Py_ssize_t end,
+             Py_ssize_t *n_remainder, int *has_decimal)
+{
+    Py_ssize_t remainder;
+
+    while (pos<end && isdigit(PyUnicode_READ_CHAR(s, pos)))
+        ++pos;
+    remainder = pos;
+
+    /* Does remainder start with a decimal point? */
+    *has_decimal = pos<end && PyUnicode_READ_CHAR(s, remainder) == '.';
+
+    /* Skip the decimal point. */
+    if (*has_decimal)
+        remainder++;
+
+    *n_remainder = end - remainder;
+}
+
+/* not all fields of format are used.  for example, precision is
+   unused.  should this take discrete params in order to be more clear
+   about what it does?  or is passing a single format parameter easier
+   and more efficient enough to justify a little obfuscation? */
+static Py_ssize_t
+calc_number_widths(NumberFieldWidths *spec, Py_ssize_t n_prefix,
+                   Py_UCS4 sign_char, PyObject *number, Py_ssize_t n_start,
+                   Py_ssize_t n_end, Py_ssize_t n_remainder,
+                   int has_decimal, const LocaleInfo *locale,
+                   const InternalFormatSpec *format)
+{
+    Py_ssize_t n_non_digit_non_padding;
+    Py_ssize_t n_padding;
+
+    spec->n_digits = n_end - n_start - n_remainder - (has_decimal?1:0);
+    spec->n_lpadding = 0;
+    spec->n_prefix = n_prefix;
+    spec->n_decimal = has_decimal ? strlen(locale->decimal_point) : 0;
+    spec->n_remainder = n_remainder;
+    spec->n_spadding = 0;
+    spec->n_rpadding = 0;
+    spec->sign = '\0';
+    spec->n_sign = 0;
+
+    /* the output will look like:
+       |                                                                                         |
+       | <lpadding> <sign> <prefix> <spadding> <grouped_digits> <decimal> <remainder> <rpadding> |
+       |                                                                                         |
+
+       sign is computed from format->sign and the actual
+       sign of the number
+
+       prefix is given (it's for the '0x' prefix)
+
+       digits is already known
+
+       the total width is either given, or computed from the
+       actual digits
+
+       only one of lpadding, spadding, and rpadding can be non-zero,
+       and it's calculated from the width and other fields
+    */
+
+    /* compute the various parts we're going to write */
+    switch (format->sign) {
+    case '+':
+        /* always put a + or - */
+        spec->n_sign = 1;
+        spec->sign = (sign_char == '-' ? '-' : '+');
+        break;
+    case ' ':
+        spec->n_sign = 1;
+        spec->sign = (sign_char == '-' ? '-' : ' ');
+        break;
+    default:
+        /* Not specified, or the default (-) */
+        if (sign_char == '-') {
+            spec->n_sign = 1;
+            spec->sign = '-';
+        }
+    }
+
+    /* The number of chars used for non-digits and non-padding. */
+    n_non_digit_non_padding = spec->n_sign + spec->n_prefix + spec->n_decimal +
+        spec->n_remainder;
+
+    /* min_width can go negative, that's okay. format->width == -1 means
+       we don't care. */
+    if (format->fill_char == '0' && format->align == '=')
+        spec->n_min_width = format->width - n_non_digit_non_padding;
+    else
+        spec->n_min_width = 0;
+
+    if (spec->n_digits == 0)
+        /* This case only occurs when using 'c' formatting, we need
+           to special case it because the grouping code always wants
+           to have at least one character. */
+        spec->n_grouped_digits = 0;
+    else
+        spec->n_grouped_digits = _PyUnicode_InsertThousandsGrouping(
+            PyUnicode_1BYTE_KIND, NULL, 0, NULL, 
+            spec->n_digits, spec->n_min_width,
+            locale->grouping, locale->thousands_sep);
+
+    /* Given the desired width and the total of digit and non-digit
+       space we consume, see if we need any padding. format->width can
+       be negative (meaning no padding), but this code still works in
+       that case. */
+    n_padding = format->width -
+                        (n_non_digit_non_padding + spec->n_grouped_digits);
+    if (n_padding > 0) {
+        /* Some padding is needed. Determine if it's left, space, or right. */
+        switch (format->align) {
+        case '<':
+            spec->n_rpadding = n_padding;
+            break;
+        case '^':
+            spec->n_lpadding = n_padding / 2;
+            spec->n_rpadding = n_padding - spec->n_lpadding;
+            break;
+        case '=':
+            spec->n_spadding = n_padding;
+            break;
+        case '>':
+            spec->n_lpadding = n_padding;
+            break;
+        default:
+            /* Shouldn't get here, but treat it as '>' */
+            spec->n_lpadding = n_padding;
+            assert(0);
+            break;
+        }
+    }
+    return spec->n_lpadding + spec->n_sign + spec->n_prefix +
+        spec->n_spadding + spec->n_grouped_digits + spec->n_decimal +
+        spec->n_remainder + spec->n_rpadding;
+}
+
+/* Fill in the digit parts of a numbers's string representation,
+   as determined in calc_number_widths().
+   No error checking, since we know the buffer is the correct size. */
+static void
+fill_number(PyObject *out, Py_ssize_t pos, const NumberFieldWidths *spec,
+            PyObject *digits, Py_ssize_t d_start, Py_ssize_t d_end,
+            PyObject *prefix, Py_ssize_t p_start, Py_UCS4 fill_char,
+            LocaleInfo *locale, int toupper)
+{
+    /* Used to keep track of digits, decimal, and remainder. */
+    Py_ssize_t d_pos = d_start;
+    unsigned int kind = PyUnicode_KIND(out);
+    void *data = PyUnicode_DATA(out);
+
+#ifndef NDEBUG
+    Py_ssize_t r;
+#endif
+
+    if (spec->n_lpadding) {
+        unicode_fill(out, pos, pos + spec->n_lpadding, fill_char);
+        pos += spec->n_lpadding;
+    }
+    if (spec->n_sign == 1) {
+        PyUnicode_WRITE(kind, data, pos++, spec->sign);
+    }
+    if (spec->n_prefix) {
+        PyUnicode_CopyCharacters(out, pos, prefix, p_start, spec->n_prefix);
+        if (toupper) {
+            Py_ssize_t t;
+            /* XXX if the upper-case prefix is wider than the target
+               buffer, the caller should have allocated a wider string,
+               but currently doesn't. */
+            for (t = 0; t < spec->n_prefix; ++t)
+                PyUnicode_WRITE(kind, data, pos + t,
+                                Py_UNICODE_TOUPPER(
+                                    PyUnicode_READ(kind, data, pos + t)));
+        }
+        pos += spec->n_prefix;
+    }
+    if (spec->n_spadding) {
+        unicode_fill(out, pos, pos + spec->n_spadding, fill_char);
+        pos += spec->n_spadding;
+    }
+
+    /* Only for type 'c' special case, it has no digits. */
+    if (spec->n_digits != 0) {
+        /* Fill the digits with InsertThousandsGrouping. */
+        char *pdigits = PyUnicode_DATA(digits);
+        if (PyUnicode_KIND(digits) < kind) {
+            pdigits = _PyUnicode_AsKind(digits, kind);
+            if (pdigits == NULL) {
+                /* XXX report exception */
+                Py_FatalError("out of memory");
+                return;
+            }
+        }
+#ifndef NDEBUG
+        r =
+#endif
+            _PyUnicode_InsertThousandsGrouping(
+                kind,
+                (char*)data + PyUnicode_KIND_SIZE(kind, pos),
+                spec->n_grouped_digits,
+                pdigits + PyUnicode_KIND_SIZE(kind, d_pos),
+                spec->n_digits, spec->n_min_width,
+                locale->grouping, locale->thousands_sep);
+#ifndef NDEBUG
+        assert(r == spec->n_grouped_digits);
+#endif
+        if (PyUnicode_KIND(digits) < kind)
+            PyMem_Free(pdigits);
+        d_pos += spec->n_digits;
+    }
+    if (toupper) {
+        Py_ssize_t t;
+        for (t = 0; t < spec->n_grouped_digits; ++t)
+            PyUnicode_WRITE(kind, data, pos + t,
+                            Py_UNICODE_TOUPPER(
+                                PyUnicode_READ(kind, data, pos + t)));
+    }
+    pos += spec->n_grouped_digits;
+
+    if (spec->n_decimal) {
+        Py_ssize_t t;
+        for (t = 0; t < spec->n_decimal; ++t)
+            PyUnicode_WRITE(kind, data, pos + t,
+                            locale->decimal_point[t]);
+        pos += spec->n_decimal;
+        d_pos += 1;
+    }
+
+    if (spec->n_remainder) {
+        PyUnicode_CopyCharacters(out, pos, digits, d_pos, spec->n_remainder);
+        pos += spec->n_remainder;
+        d_pos += spec->n_remainder;
+    }
+
+    if (spec->n_rpadding) {
+        unicode_fill(out, pos, pos + spec->n_rpadding, fill_char);
+        pos += spec->n_rpadding;
+    }
+}
+
+static char no_grouping[1] = {CHAR_MAX};
+
+/* Find the decimal point character(s?), thousands_separator(s?), and
+   grouping description, either for the current locale if type is
+   LT_CURRENT_LOCALE, a hard-coded locale if LT_DEFAULT_LOCALE, or
+   none if LT_NO_LOCALE. */
+static void
+get_locale_info(int type, LocaleInfo *locale_info)
+{
+    switch (type) {
+    case LT_CURRENT_LOCALE: {
+        struct lconv *locale_data = localeconv();
+        locale_info->decimal_point = locale_data->decimal_point;
+        locale_info->thousands_sep = locale_data->thousands_sep;
+        locale_info->grouping = locale_data->grouping;
+        break;
+    }
+    case LT_DEFAULT_LOCALE:
+        locale_info->decimal_point = ".";
+        locale_info->thousands_sep = ",";
+        locale_info->grouping = "\3"; /* Group every 3 characters.  The
+                                         (implicit) trailing 0 means repeat
+                                         infinitely. */
+        break;
+    case LT_NO_LOCALE:
+        locale_info->decimal_point = ".";
+        locale_info->thousands_sep = "";
+        locale_info->grouping = no_grouping;
+        break;
+    default:
+        assert(0);
+    }
+}
+
+/************************************************************************/
+/*********** string formatting ******************************************/
+/************************************************************************/
+
+static PyObject *
+format_string_internal(PyObject *value, const InternalFormatSpec *format)
+{
+    Py_ssize_t lpad;
+    Py_ssize_t rpad;
+    Py_ssize_t total;
+    Py_ssize_t pos;
+    Py_ssize_t len = PyUnicode_GET_SIZE(value);
+    PyObject *result = NULL;
+    int maxchar = 127;
+
+    /* sign is not allowed on strings */
+    if (format->sign != '\0') {
+        PyErr_SetString(PyExc_ValueError,
+                        "Sign not allowed in string format specifier");
+        goto done;
+    }
+
+    /* alternate is not allowed on strings */
+    if (format->alternate) {
+        PyErr_SetString(PyExc_ValueError,
+                        "Alternate form (#) not allowed in string format "
+                        "specifier");
+        goto done;
+    }
+
+    /* '=' alignment not allowed on strings */
+    if (format->align == '=') {
+        PyErr_SetString(PyExc_ValueError,
+                        "'=' alignment not allowed "
+                        "in string format specifier");
+        goto done;
+    }
+
+    /* if precision is specified, output no more that format.precision
+       characters */
+    if (format->precision >= 0 && len >= format->precision) {
+        len = format->precision;
+    }
+
+    calc_padding(len, format->width, format->align, &lpad, &rpad, &total);
+
+    /* allocate the resulting string */
+    result = PyUnicode_New(total, maxchar);
+    if (result == NULL)
+        goto done;
+
+    /* Write into that space. First the padding. */
+    pos = fill_padding(result, 0, len,
+                       format->fill_char=='\0'?' ':format->fill_char,
+                       lpad, rpad);
+
+    /* Then the source string. */
+    PyUnicode_CopyCharacters(result, pos, value, 0, len);
+
+done:
+    return result;
+}
+
+
+/************************************************************************/
+/*********** long formatting ********************************************/
+/************************************************************************/
+
+typedef PyObject*
+(*IntOrLongToString)(PyObject *value, int base);
+
+static PyObject *
+format_int_or_long_internal(PyObject *value, const InternalFormatSpec *format,
+                            IntOrLongToString tostring)
+{
+    PyObject *result = NULL;
+    int maxchar = 127;
+    PyObject *tmp = NULL;
+    Py_ssize_t inumeric_chars;
+    Py_UCS4 sign_char = '\0';
+    Py_ssize_t n_digits;       /* count of digits need from the computed
+                                  string */
+    Py_ssize_t n_remainder = 0; /* Used only for 'c' formatting, which
+                                   produces non-digits */
+    Py_ssize_t n_prefix = 0;   /* Count of prefix chars, (e.g., '0x') */
+    Py_ssize_t n_total;
+    Py_ssize_t prefix;
+    NumberFieldWidths spec;
+    long x;
+
+    /* Locale settings, either from the actual locale or
+       from a hard-code pseudo-locale */
+    LocaleInfo locale;
+
+    /* no precision allowed on integers */
+    if (format->precision != -1) {
+        PyErr_SetString(PyExc_ValueError,
+                        "Precision not allowed in integer format specifier");
+        goto done;
+    }
+
+    /* special case for character formatting */
+    if (format->type == 'c') {
+        /* error to specify a sign */
+        if (format->sign != '\0') {
+            PyErr_SetString(PyExc_ValueError,
+                            "Sign not allowed with integer"
+                            " format specifier 'c'");
+            goto done;
+        }
+
+        /* taken from unicodeobject.c formatchar() */
+        /* Integer input truncated to a character */
+/* XXX: won't work for int */
+        x = PyLong_AsLong(value);
+        if (x == -1 && PyErr_Occurred())
+            goto done;
+        if (x < 0 || x > 0x10ffff) {
+            PyErr_SetString(PyExc_OverflowError,
+                            "%c arg not in range(0x110000) "
+                            "(wide Python build)");
+            goto done;
+        }
+        tmp = PyUnicode_FromOrdinal(x);
+        inumeric_chars = 0;
+        n_digits = 1;
+        if (x > maxchar)
+            maxchar = x;
+
+        /* As a sort-of hack, we tell calc_number_widths that we only
+           have "remainder" characters. calc_number_widths thinks
+           these are characters that don't get formatted, only copied
+           into the output string. We do this for 'c' formatting,
+           because the characters are likely to be non-digits. */
+        n_remainder = 1;
+    }
+    else {
+        int base;
+        int leading_chars_to_skip = 0;  /* Number of characters added by
+                                           PyNumber_ToBase that we want to
+                                           skip over. */
+
+        /* Compute the base and how many characters will be added by
+           PyNumber_ToBase */
+        switch (format->type) {
+        case 'b':
+            base = 2;
+            leading_chars_to_skip = 2; /* 0b */
+            break;
+        case 'o':
+            base = 8;
+            leading_chars_to_skip = 2; /* 0o */
+            break;
+        case 'x':
+        case 'X':
+            base = 16;
+            leading_chars_to_skip = 2; /* 0x */
+            break;
+        default:  /* shouldn't be needed, but stops a compiler warning */
+        case 'd':
+        case 'n':
+            base = 10;
+            break;
+        }
+
+        /* The number of prefix chars is the same as the leading
+           chars to skip */
+        if (format->alternate)
+            n_prefix = leading_chars_to_skip;
+
+        /* Do the hard part, converting to a string in a given base */
+        tmp = tostring(value, base);
+        if (tmp == NULL || PyUnicode_READY(tmp) == -1)
+            goto done;
+
+        inumeric_chars = 0;
+        n_digits = PyUnicode_GET_LENGTH(tmp);
+
+        prefix = inumeric_chars;
+
+        /* Is a sign character present in the output?  If so, remember it
+           and skip it */
+        if (PyUnicode_READ_CHAR(tmp, inumeric_chars) == '-') {
+            sign_char = '-';
+            ++prefix;
+            ++leading_chars_to_skip;
+        }
+
+        /* Skip over the leading chars (0x, 0b, etc.) */
+        n_digits -= leading_chars_to_skip;
+        inumeric_chars += leading_chars_to_skip;
+    }
+
+    /* Determine the grouping, separator, and decimal point, if any. */
+    get_locale_info(format->type == 'n' ? LT_CURRENT_LOCALE :
+                    (format->thousands_separators ?
+                     LT_DEFAULT_LOCALE :
+                     LT_NO_LOCALE),
+                    &locale);
+
+    /* Calculate how much memory we'll need. */
+    n_total = calc_number_widths(&spec, n_prefix, sign_char, tmp, inumeric_chars,
+                                 inumeric_chars + n_digits, n_remainder, 0, &locale, format);
+
+    /* Allocate the memory. */
+    result = PyUnicode_New(n_total, maxchar);
+    if (!result)
+        goto done;
+
+    /* Populate the memory. */
+    fill_number(result, 0, &spec, tmp, inumeric_chars, inumeric_chars + n_digits,
+                tmp, prefix,
+                format->fill_char == '\0' ? ' ' : format->fill_char,
+                &locale, format->type == 'X');
+
+done:
+    Py_XDECREF(tmp);
+    return result;
+}
+
+/************************************************************************/
+/*********** float formatting *******************************************/
+/************************************************************************/
+
+static PyObject*
+strtounicode(char *charbuffer, Py_ssize_t len)
+{
+    return PyUnicode_FromKindAndData(PyUnicode_1BYTE_KIND, charbuffer, len);
+}
+
+/* much of this is taken from unicodeobject.c */
+static PyObject *
+format_float_internal(PyObject *value,
+                      const InternalFormatSpec *format)
+{
+    char *buf = NULL;       /* buffer returned from PyOS_double_to_string */
+    Py_ssize_t n_digits;
+    Py_ssize_t n_remainder;
+    Py_ssize_t n_total;
+    int has_decimal;
+    double val;
+    Py_ssize_t precision = format->precision;
+    Py_ssize_t default_precision = 6;
+    Py_UCS4 type = format->type;
+    int add_pct = 0;
+    Py_ssize_t index;
+    NumberFieldWidths spec;
+    int flags = 0;
+    PyObject *result = NULL;
+    int maxchar = 127;
+    Py_UCS4 sign_char = '\0';
+    int float_type; /* Used to see if we have a nan, inf, or regular float. */
+    PyObject *unicode_tmp = NULL;
+
+    /* Locale settings, either from the actual locale or
+       from a hard-code pseudo-locale */
+    LocaleInfo locale;
+
+    if (format->alternate)
+        flags |= Py_DTSF_ALT;
+
+    if (type == '\0') {
+        /* Omitted type specifier.  Behaves in the same way as repr(x)
+           and str(x) if no precision is given, else like 'g', but with
+           at least one digit after the decimal point. */
+        flags |= Py_DTSF_ADD_DOT_0;
+        type = 'r';
+        default_precision = 0;
+    }
+
+    if (type == 'n')
+        /* 'n' is the same as 'g', except for the locale used to
+           format the result. We take care of that later. */
+        type = 'g';
+
+    val = PyFloat_AsDouble(value);
+    if (val == -1.0 && PyErr_Occurred())
+        goto done;
+
+    if (type == '%') {
+        type = 'f';
+        val *= 100;
+        add_pct = 1;
+    }
+
+    if (precision < 0)
+        precision = default_precision;
+    else if (type == 'r')
+        type = 'g';
+
+    /* Cast "type", because if we're in unicode we need to pass a
+       8-bit char. This is safe, because we've restricted what "type"
+       can be. */
+    buf = PyOS_double_to_string(val, (char)type, precision, flags,
+                                &float_type);
+    if (buf == NULL)
+        goto done;
+    n_digits = strlen(buf);
+
+    if (add_pct) {
+        /* We know that buf has a trailing zero (since we just called
+           strlen() on it), and we don't use that fact any more. So we
+           can just write over the trailing zero. */
+        buf[n_digits] = '%';
+        n_digits += 1;
+    }
+
+    /* Since there is no unicode version of PyOS_double_to_string,
+       just use the 8 bit version and then convert to unicode. */
+    unicode_tmp = strtounicode(buf, n_digits);
+    if (unicode_tmp == NULL)
+        goto done;
+    index = 0;
+
+    /* Is a sign character present in the output?  If so, remember it
+       and skip it */
+    if (PyUnicode_READ_CHAR(unicode_tmp, index) == '-') {
+        sign_char = '-';
+        ++index;
+        --n_digits;
+    }
+
+    /* Determine if we have any "remainder" (after the digits, might include
+       decimal or exponent or both (or neither)) */
+    parse_number(unicode_tmp, index, index + n_digits, &n_remainder, &has_decimal);
+
+    /* Determine the grouping, separator, and decimal point, if any. */
+    get_locale_info(format->type == 'n' ? LT_CURRENT_LOCALE :
+                    (format->thousands_separators ?
+                     LT_DEFAULT_LOCALE :
+                     LT_NO_LOCALE),
+                    &locale);
+
+    /* Calculate how much memory we'll need. */
+    n_total = calc_number_widths(&spec, 0, sign_char, unicode_tmp, index, 
+                                 index + n_digits, n_remainder, has_decimal,
+                                 &locale, format);
+
+    /* Allocate the memory. */
+    result = PyUnicode_New(n_total, maxchar);
+    if (result == NULL)
+        goto done;
+
+    /* Populate the memory. */
+    fill_number(result, 0, &spec, unicode_tmp, index, index + n_digits,
+                NULL, 0,
+                format->fill_char == '\0' ? ' ' : format->fill_char, &locale,
+                0);
+
+done:
+    PyMem_Free(buf);
+    Py_DECREF(unicode_tmp);
+    return result;
+}
+
+/************************************************************************/
+/*********** complex formatting *****************************************/
+/************************************************************************/
+
+static PyObject *
+format_complex_internal(PyObject *value,
+                        const InternalFormatSpec *format)
+{
+    double re;
+    double im;
+    char *re_buf = NULL;       /* buffer returned from PyOS_double_to_string */
+    char *im_buf = NULL;       /* buffer returned from PyOS_double_to_string */
+
+    InternalFormatSpec tmp_format = *format;
+    Py_ssize_t n_re_digits;
+    Py_ssize_t n_im_digits;
+    Py_ssize_t n_re_remainder;
+    Py_ssize_t n_im_remainder;
+    Py_ssize_t n_re_total;
+    Py_ssize_t n_im_total;
+    int re_has_decimal;
+    int im_has_decimal;
+    Py_ssize_t precision = format->precision;
+    Py_ssize_t default_precision = 6;
+    Py_UCS4 type = format->type;
+    Py_ssize_t i_re;
+    Py_ssize_t i_im;
+    NumberFieldWidths re_spec;
+    NumberFieldWidths im_spec;
+    int flags = 0;
+    PyObject *result = NULL;
+    int maxchar = 127;
+    int rkind;
+    void *rdata;
+    Py_ssize_t index;
+    Py_UCS4 re_sign_char = '\0';
+    Py_UCS4 im_sign_char = '\0';
+    int re_float_type; /* Used to see if we have a nan, inf, or regular float. */
+    int im_float_type;
+    int add_parens = 0;
+    int skip_re = 0;
+    Py_ssize_t lpad;
+    Py_ssize_t rpad;
+    Py_ssize_t total;
+    PyObject *re_unicode_tmp = NULL;
+    PyObject *im_unicode_tmp = NULL;
+
+    /* Locale settings, either from the actual locale or
+       from a hard-code pseudo-locale */
+    LocaleInfo locale;
+
+    /* Zero padding is not allowed. */
+    if (format->fill_char == '0') {
+        PyErr_SetString(PyExc_ValueError,
+                        "Zero padding is not allowed in complex format "
+                        "specifier");
+        goto done;
+    }
+
+    /* Neither is '=' alignment . */
+    if (format->align == '=') {
+        PyErr_SetString(PyExc_ValueError,
+                        "'=' alignment flag is not allowed in complex format "
+                        "specifier");
+        goto done;
+    }
+
+    re = PyComplex_RealAsDouble(value);
+    if (re == -1.0 && PyErr_Occurred())
+        goto done;
+    im = PyComplex_ImagAsDouble(value);
+    if (im == -1.0 && PyErr_Occurred())
+        goto done;
+
+    if (format->alternate)
+        flags |= Py_DTSF_ALT;
+
+    if (type == '\0') {
+        /* Omitted type specifier. Should be like str(self). */
+        type = 'r';
+        default_precision = 0;
+        if (re == 0.0 && copysign(1.0, re) == 1.0)
+            skip_re = 1;
+        else
+            add_parens = 1;
+    }
+
+    if (type == 'n')
+        /* 'n' is the same as 'g', except for the locale used to
+           format the result. We take care of that later. */
+        type = 'g';
+
+    if (precision < 0)
+        precision = default_precision;
+    else if (type == 'r')
+        type = 'g';
+
+    /* Cast "type", because if we're in unicode we need to pass a
+       8-bit char. This is safe, because we've restricted what "type"
+       can be. */
+    re_buf = PyOS_double_to_string(re, (char)type, precision, flags,
+                                   &re_float_type);
+    if (re_buf == NULL)
+        goto done;
+    im_buf = PyOS_double_to_string(im, (char)type, precision, flags,
+                                   &im_float_type);
+    if (im_buf == NULL)
+        goto done;
+
+    n_re_digits = strlen(re_buf);
+    n_im_digits = strlen(im_buf);
+
+    /* Since there is no unicode version of PyOS_double_to_string,
+       just use the 8 bit version and then convert to unicode. */
+    re_unicode_tmp = strtounicode(re_buf, n_re_digits);
+    if (re_unicode_tmp == NULL)
+        goto done;
+    i_re = 0;
+
+    im_unicode_tmp = strtounicode(im_buf, n_im_digits);
+    if (im_unicode_tmp == NULL)
+        goto done;
+    i_im = 0;
+
+    /* Is a sign character present in the output?  If so, remember it
+       and skip it */
+    if (PyUnicode_READ_CHAR(re_unicode_tmp, i_re) == '-') {
+        re_sign_char = '-';
+        ++i_re;
+        --n_re_digits;
+    }
+    if (PyUnicode_READ_CHAR(im_unicode_tmp, i_im) == '-') {
+        im_sign_char = '-';
+        ++i_im;
+        --n_im_digits;
+    }
+
+    /* Determine if we have any "remainder" (after the digits, might include
+       decimal or exponent or both (or neither)) */
+    parse_number(re_unicode_tmp, i_re, i_re + n_re_digits, 
+                 &n_re_remainder, &re_has_decimal);
+    parse_number(im_unicode_tmp, i_im, i_im + n_im_digits, 
+                 &n_im_remainder, &im_has_decimal);
+
+    /* Determine the grouping, separator, and decimal point, if any. */
+    get_locale_info(format->type == 'n' ? LT_CURRENT_LOCALE :
+                    (format->thousands_separators ?
+                     LT_DEFAULT_LOCALE :
+                     LT_NO_LOCALE),
+                    &locale);
+
+    /* Turn off any padding. We'll do it later after we've composed
+       the numbers without padding. */
+    tmp_format.fill_char = '\0';
+    tmp_format.align = '<';
+    tmp_format.width = -1;
+
+    /* Calculate how much memory we'll need. */
+    n_re_total = calc_number_widths(&re_spec, 0, re_sign_char, re_unicode_tmp,
+                                    i_re, i_re + n_re_digits, n_re_remainder,
+                                    re_has_decimal, &locale, &tmp_format);
+
+    /* Same formatting, but always include a sign, unless the real part is
+     * going to be omitted, in which case we use whatever sign convention was
+     * requested by the original format. */
+    if (!skip_re)
+        tmp_format.sign = '+';
+    n_im_total = calc_number_widths(&im_spec, 0, im_sign_char, im_unicode_tmp,
+                                    i_im, i_im + n_im_digits, n_im_remainder,
+                                    im_has_decimal, &locale, &tmp_format);
+
+    if (skip_re)
+        n_re_total = 0;
+
+    /* Add 1 for the 'j', and optionally 2 for parens. */
+    calc_padding(n_re_total + n_im_total + 1 + add_parens * 2,
+                 format->width, format->align, &lpad, &rpad, &total);
+
+    result = PyUnicode_New(total, maxchar);
+    if (result == NULL)
+        goto done;
+    rkind = PyUnicode_KIND(result);
+    rdata = PyUnicode_DATA(result);
+
+    /* Populate the memory. First, the padding. */
+    index = fill_padding(result, 0,
+                         n_re_total + n_im_total + 1 + add_parens * 2,
+                         format->fill_char=='\0' ? ' ' : format->fill_char,
+                         lpad, rpad);
+
+    if (add_parens)
+        PyUnicode_WRITE(rkind, rdata, index++, '(');
+
+    if (!skip_re) {
+        fill_number(result, index, &re_spec, re_unicode_tmp, 
+                    i_re, i_re + n_re_digits, NULL, 0, 0, &locale, 0);
+        index += n_re_total;
+    }
+    fill_number(result, index, &im_spec, im_unicode_tmp,
+                i_im, i_im + n_im_digits, NULL, 0, 0, &locale, 0);
+    index += n_im_total;
+    PyUnicode_WRITE(rkind, rdata, index++, 'j');
+
+    if (add_parens)
+        PyUnicode_WRITE(rkind, rdata, index++, ')');
+
+done:
+    PyMem_Free(re_buf);
+    PyMem_Free(im_buf);
+    Py_XDECREF(re_unicode_tmp);
+    Py_XDECREF(im_unicode_tmp);
+    return result;
+}
+
+/************************************************************************/
+/*********** built in formatters ****************************************/
+/************************************************************************/
+PyObject *
+_PyUnicode_FormatAdvanced(PyObject *obj,
+                          PyObject *format_spec,
+                          Py_ssize_t start, Py_ssize_t end)
+{
+    InternalFormatSpec format;
+    PyObject *result = NULL;
+
+    /* check for the special case of zero length format spec, make
+       it equivalent to str(obj) */
+    if (start == end) {
+        result = PyObject_Str(obj);
+        goto done;
+    }
+
+    /* parse the format_spec */
+    if (!parse_internal_render_format_spec(format_spec, start, end,
+                                           &format, 's', '<'))
+        goto done;
+
+    /* type conversion? */
+    switch (format.type) {
+    case 's':
+        /* no type conversion needed, already a string.  do the formatting */
+        result = format_string_internal(obj, &format);
+        break;
+    default:
+        /* unknown */
+        unknown_presentation_type(format.type, obj->ob_type->tp_name);
+        goto done;
+    }
+
+done:
+    return result;
+}
+
+static PyObject*
+format_int_or_long(PyObject* obj, PyObject* format_spec,
+                   Py_ssize_t start, Py_ssize_t end,
+                   IntOrLongToString tostring)
+{
+    PyObject *result = NULL;
+    PyObject *tmp = NULL;
+    InternalFormatSpec format;
+
+    /* check for the special case of zero length format spec, make
+       it equivalent to str(obj) */
+    if (start == end) {
+        result = PyObject_Str(obj);
+        goto done;
+    }
+
+    /* parse the format_spec */
+    if (!parse_internal_render_format_spec(format_spec, start, end,
+                                           &format, 'd', '>'))
+        goto done;
+
+    /* type conversion? */
+    switch (format.type) {
+    case 'b':
+    case 'c':
+    case 'd':
+    case 'o':
+    case 'x':
+    case 'X':
+    case 'n':
+        /* no type conversion needed, already an int (or long).  do
+           the formatting */
+            result = format_int_or_long_internal(obj, &format, tostring);
+        break;
+
+    case 'e':
+    case 'E':
+    case 'f':
+    case 'F':
+    case 'g':
+    case 'G':
+    case '%':
+        /* convert to float */
+        tmp = PyNumber_Float(obj);
+        if (tmp == NULL)
+            goto done;
+        result = format_float_internal(tmp, &format);
+        break;
+
+    default:
+        /* unknown */
+        unknown_presentation_type(format.type, obj->ob_type->tp_name);
+        goto done;
+    }
+
+done:
+    Py_XDECREF(tmp);
+    return result;
+}
+
+/* Need to define long_format as a function that will convert a long
+   to a string.  In 3.0, _PyLong_Format has the correct signature. */
+#define long_format _PyLong_Format
+
+PyObject *
+_PyLong_FormatAdvanced(PyObject *obj,
+                       PyObject *format_spec,
+                       Py_ssize_t start, Py_ssize_t end)
+{
+    return format_int_or_long(obj, format_spec, start, end,
+                              long_format);
+}
+
+PyObject *
+_PyFloat_FormatAdvanced(PyObject *obj,
+                        PyObject *format_spec,
+                        Py_ssize_t start, Py_ssize_t end)
+{
+    PyObject *result = NULL;
+    InternalFormatSpec format;
+
+    /* check for the special case of zero length format spec, make
+       it equivalent to str(obj) */
+    if (start == end) {
+        result = PyObject_Str(obj);
+        goto done;
+    }
+
+    /* parse the format_spec */
+    if (!parse_internal_render_format_spec(format_spec, start, end,
+                                           &format, '\0', '>'))
+        goto done;
+
+    /* type conversion? */
+    switch (format.type) {
+    case '\0': /* No format code: like 'g', but with at least one decimal. */
+    case 'e':
+    case 'E':
+    case 'f':
+    case 'F':
+    case 'g':
+    case 'G':
+    case 'n':
+    case '%':
+        /* no conversion, already a float.  do the formatting */
+        result = format_float_internal(obj, &format);
+        break;
+
+    default:
+        /* unknown */
+        unknown_presentation_type(format.type, obj->ob_type->tp_name);
+        goto done;
+    }
+
+done:
+    return result;
+}
+
+PyObject *
+_PyComplex_FormatAdvanced(PyObject *obj,
+                          PyObject *format_spec,
+                          Py_ssize_t start, Py_ssize_t end)
+{
+    PyObject *result = NULL;
+    InternalFormatSpec format;
+
+    /* check for the special case of zero length format spec, make
+       it equivalent to str(obj) */
+    if (start == end) {
+        result = PyObject_Str(obj);
+        goto done;
+    }
+
+    /* parse the format_spec */
+    if (!parse_internal_render_format_spec(format_spec, start, end,
+                                           &format, '\0', '>'))
+        goto done;
+
+    /* type conversion? */
+    switch (format.type) {
+    case '\0': /* No format code: like 'g', but with at least one decimal. */
+    case 'e':
+    case 'E':
+    case 'f':
+    case 'F':
+    case 'g':
+    case 'G':
+    case 'n':
+        /* no conversion, already a complex.  do the formatting */
+        result = format_complex_internal(obj, &format);
+        break;
+
+    default:
+        /* unknown */
+        unknown_presentation_type(format.type, obj->ob_type->tp_name);
+        goto done;
+    }
+
+done:
+    return result;
+}
diff --git a/Python/getargs.c b/Python/getargs.c
--- a/Python/getargs.c
+++ b/Python/getargs.c
@@ -546,9 +546,6 @@
 
 
 
-#define UNICODE_DEFAULT_ENCODING(arg) \
-    _PyUnicode_AsDefaultEncodedString(arg)
-
 /* Format an error message generated by convertsimple(). */
 
 static char *
@@ -611,7 +608,7 @@
 
     const char *format = *p_format;
     char c = *format++;
-    PyObject *uarg;
+    char *sarg;
 
     switch (c) {
 
@@ -838,8 +835,11 @@
     case 'C': {/* unicode char */
         int *p = va_arg(*p_va, int *);
         if (PyUnicode_Check(arg) &&
-            PyUnicode_GET_SIZE(arg) == 1)
-            *p = PyUnicode_AS_UNICODE(arg)[0];
+            PyUnicode_GET_LENGTH(arg) == 1) {
+            int kind = PyUnicode_KIND(arg);
+            void *data = PyUnicode_DATA(arg);
+            *p = PyUnicode_READ(kind, data, 0);
+        }
         else
             return converterr("a unicode character", arg, msgbuf, bufsize);
         break;
@@ -889,13 +889,12 @@
             if (c == 'z' && arg == Py_None)
                 PyBuffer_FillInfo(p, NULL, NULL, 0, 1, 0);
             else if (PyUnicode_Check(arg)) {
-                uarg = UNICODE_DEFAULT_ENCODING(arg);
-                if (uarg == NULL)
+                Py_ssize_t len;
+                sarg = PyUnicode_AsUTF8AndSize(arg, &len);
+                if (sarg == NULL)
                     return converterr(CONV_UNICODE,
                                       arg, msgbuf, bufsize);
-                PyBuffer_FillInfo(p, arg,
-                                  PyBytes_AS_STRING(uarg), PyBytes_GET_SIZE(uarg),
-                                  1, 0);
+                PyBuffer_FillInfo(p, arg, sarg, len, 1, 0);
             }
             else { /* any buffer-like object */
                 char *buf;
@@ -918,12 +917,13 @@
                 STORE_SIZE(0);
             }
             else if (PyUnicode_Check(arg)) {
-                uarg = UNICODE_DEFAULT_ENCODING(arg);
-                if (uarg == NULL)
+                Py_ssize_t len;
+                sarg = PyUnicode_AsUTF8AndSize(arg, &len);
+                if (sarg == NULL)
                     return converterr(CONV_UNICODE,
                                       arg, msgbuf, bufsize);
-                *p = PyBytes_AS_STRING(uarg);
-                STORE_SIZE(PyBytes_GET_SIZE(uarg));
+                *p = sarg;
+                STORE_SIZE(len);
             }
             else { /* any buffer-like object */
                 /* XXX Really? */
@@ -937,22 +937,22 @@
         } else {
             /* "s" or "z" */
             char **p = va_arg(*p_va, char **);
-            uarg = NULL;
+            Py_ssize_t len;
+            sarg = NULL;
 
             if (c == 'z' && arg == Py_None)
                 *p = NULL;
             else if (PyUnicode_Check(arg)) {
-                uarg = UNICODE_DEFAULT_ENCODING(arg);
-                if (uarg == NULL)
+                sarg = PyUnicode_AsUTF8AndSize(arg, &len);
+                if (sarg == NULL)
                     return converterr(CONV_UNICODE,
                                       arg, msgbuf, bufsize);
-                *p = PyBytes_AS_STRING(uarg);
+                *p = sarg;
             }
             else
                 return converterr(c == 'z' ? "str or None" : "str",
                                   arg, msgbuf, bufsize);
-            if (*p != NULL && uarg != NULL &&
-                (Py_ssize_t) strlen(*p) != PyBytes_GET_SIZE(uarg))
+            if (*p != NULL && sarg != NULL && (Py_ssize_t) strlen(*p) != len)
                 return converterr(
                     c == 'z' ? "str without null bytes or None"
                              : "str without null bytes",
@@ -976,6 +976,8 @@
             }
             else if (PyUnicode_Check(arg)) {
                 *p = PyUnicode_AS_UNICODE(arg);
+                if (*p == NULL)
+                    RETURN_ERR_OCCURRED;
                 STORE_SIZE(PyUnicode_GET_SIZE(arg));
             }
             else
@@ -987,6 +989,8 @@
                 *p = NULL;
             else if (PyUnicode_Check(arg)) {
                 *p = PyUnicode_AS_UNICODE(arg);
+                if (*p == NULL)
+                    RETURN_ERR_OCCURRED;
                 if (Py_UNICODE_strlen(*p) != PyUnicode_GET_SIZE(arg))
                     return converterr(
                         "str without null character or None",
diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -118,12 +118,12 @@
 #define MAGIC (3190 | ((long)'\r'<<16) | ((long)'\n'<<24))
 #define TAG "cpython-" MAJOR MINOR;
 #define CACHEDIR "__pycache__"
-static const Py_UNICODE CACHEDIR_UNICODE[] = {
+static const Py_UCS4 CACHEDIR_UNICODE[] = {
     '_', '_', 'p', 'y', 'c', 'a', 'c', 'h', 'e', '_', '_', '\0'};
 /* Current magic word and string tag as globals. */
 static long pyc_magic = MAGIC;
 static const char *pyc_tag = TAG;
-static const Py_UNICODE PYC_TAG_UNICODE[] = {
+static const Py_UCS4 PYC_TAG_UNICODE[] = {
     'c', 'p', 'y', 't', 'h', 'o', 'n', '-', PY_MAJOR_VERSION + 48, PY_MINOR_VERSION + 48, '\0'};
 #undef QUOTE
 #undef STRIFY
@@ -762,7 +762,7 @@
 
 static PyObject * get_sourcefile(PyObject *filename);
 static PyObject *make_source_pathname(PyObject *pathname);
-static PyObject* make_compiled_pathname(Py_UNICODE *pathname, int debug);
+static PyObject* make_compiled_pathname(PyObject *pathname, int debug);
 
 /* Execute a code object in a module and return the module object
  * WITH INCREMENTED REFERENCE COUNT.  If an error occurs, name is
@@ -886,10 +886,10 @@
 /* Like strrchr(string, '/') but searches for the rightmost of either SEP
    or ALTSEP, if the latter is defined.
 */
-static Py_UNICODE*
-rightmost_sep(Py_UNICODE *s)
+static Py_UCS4*
+rightmost_sep(Py_UCS4 *s)
 {
-    Py_UNICODE *found, c;
+    Py_UCS4 *found, c;
     for (found = NULL; (c = *s); s++) {
         if (c == SEP
 #ifdef ALTSEP
@@ -912,15 +912,21 @@
    foo.py -> __pycache__/foo.<tag>.pyc */
 
 static PyObject*
-make_compiled_pathname(Py_UNICODE *pathname, int debug)
+make_compiled_pathname(PyObject *pathstr, int debug)
 {
-    Py_UNICODE buf[MAXPATHLEN];
+    Py_UCS4 *pathname;
+    Py_UCS4 buf[MAXPATHLEN];
     size_t buflen = (size_t)MAXPATHLEN;
-    size_t len = Py_UNICODE_strlen(pathname);
+    size_t len;
     size_t i, save;
-    Py_UNICODE *pos;
+    Py_UCS4 *pos;
     int sep = SEP;
 
+    pathname = PyUnicode_AsUCS4Copy(pathstr);
+    if (!pathname)
+        return NULL;
+    len = Py_UCS4_strlen(pathname);
+
     /* Sanity check that the buffer has roughly enough space to hold what
        will eventually be the full path to the compiled file.  The 5 extra
        bytes include the slash afer __pycache__, the two extra dots, the
@@ -930,8 +936,10 @@
        sanity check before writing the extension to ensure we do not
        overflow the buffer.
     */
-    if (len + Py_UNICODE_strlen(CACHEDIR_UNICODE) + Py_UNICODE_strlen(PYC_TAG_UNICODE) + 5 > buflen)
+    if (len + Py_UCS4_strlen(CACHEDIR_UNICODE) + Py_UCS4_strlen(PYC_TAG_UNICODE) + 5 > buflen) {
+        PyMem_Free(pathname);
         return NULL;
+    }
 
     /* Find the last path separator and copy everything from the start of
        the source string up to and including the separator.
@@ -943,24 +951,28 @@
     else {
         sep = *pos;
         i = pos - pathname + 1;
-        Py_UNICODE_strncpy(buf, pathname, i);
+        Py_UCS4_strncpy(buf, pathname, i);
     }
 
     save = i;
     buf[i++] = '\0';
     /* Add __pycache__/ */
-    Py_UNICODE_strcat(buf, CACHEDIR_UNICODE);
-    i += Py_UNICODE_strlen(CACHEDIR_UNICODE) - 1;
+    Py_UCS4_strcat(buf, CACHEDIR_UNICODE);
+    i += Py_UCS4_strlen(CACHEDIR_UNICODE) - 1;
     buf[i++] = sep;
     buf[i] = '\0';
     /* Add the base filename, but remove the .py or .pyw extension, since
        the tag name must go before the extension.
     */
-    Py_UNICODE_strcat(buf, pathname + save);
-    pos = Py_UNICODE_strrchr(buf + i, '.');
+    Py_UCS4_strcat(buf, pathname + save);
+    pos = Py_UCS4_strrchr(buf + i, '.');
     if (pos != NULL)
         *++pos = '\0';
-    Py_UNICODE_strcat(buf, PYC_TAG_UNICODE);
+
+    /* pathname is not used from here on. */
+    PyMem_Free(pathname);
+
+    Py_UCS4_strcat(buf, PYC_TAG_UNICODE);
     /* The length test above assumes that we're only adding one character
        to the end of what would normally be the extension.  What if there
        is no extension, or the string ends in '.' or '.p', and otherwise
@@ -1010,7 +1022,7 @@
 #if 0
     printf("strlen(buf): %d; buflen: %d\n", (int)strlen(buf), (int)buflen);
 #endif
-    len = Py_UNICODE_strlen(buf);
+    len = Py_UCS4_strlen(buf);
     if (len + 5 > buflen)
         return NULL;
     buf[len] = '.'; len++;
@@ -1018,7 +1030,7 @@
     buf[len] = 'y'; len++;
     buf[len] = debug ? 'c' : 'o'; len++;
     assert(len <= buflen);
-    return PyUnicode_FromUnicode(buf, len);
+    return PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, buf, len);
 }
 
 
@@ -1033,14 +1045,16 @@
 static PyObject*
 make_source_pathname(PyObject *pathobj)
 {
-    Py_UNICODE buf[MAXPATHLEN];
-    Py_UNICODE *pathname;
-    Py_UNICODE *left, *right, *dot0, *dot1, sep;
+    Py_UCS4 buf[MAXPATHLEN];
+    Py_UCS4 *pathname;
+    Py_UCS4 *left, *right, *dot0, *dot1, sep;
     size_t i, j;
 
-    if (PyUnicode_GET_SIZE(pathobj) > MAXPATHLEN)
+    if (PyUnicode_GET_LENGTH(pathobj) > MAXPATHLEN)
         return NULL;
-    pathname = PyUnicode_AS_UNICODE(pathobj);
+    pathname = PyUnicode_AsUCS4Copy(pathobj);
+    if (!pathname)
+        return NULL;
 
     /* Look back two slashes from the end.  In between these two slashes
        must be the string __pycache__ or this is not a PEP 3147 style
@@ -1057,31 +1071,35 @@
         left = pathname;
     else
         left++;
-    if (right-left != Py_UNICODE_strlen(CACHEDIR_UNICODE) ||
-        Py_UNICODE_strncmp(left, CACHEDIR_UNICODE, right-left) != 0)
-        return NULL;
+    if (right-left != Py_UCS4_strlen(CACHEDIR_UNICODE) ||
+        Py_UCS4_strncmp(left, CACHEDIR_UNICODE, right-left) != 0)
+        goto error;
 
     /* Now verify that the path component to the right of the last slash
        has two dots in it.
     */
-    if ((dot0 = Py_UNICODE_strchr(right + 1, '.')) == NULL)
-        return NULL;
-    if ((dot1 = Py_UNICODE_strchr(dot0 + 1, '.')) == NULL)
-        return NULL;
+    if ((dot0 = Py_UCS4_strchr(right + 1, '.')) == NULL)
+       goto error;
+    if ((dot1 = Py_UCS4_strchr(dot0 + 1, '.')) == NULL)
+        goto error;
     /* Too many dots? */
-    if (Py_UNICODE_strchr(dot1 + 1, '.') != NULL)
-        return NULL;
+    if (Py_UCS4_strchr(dot1 + 1, '.') != NULL)
+        goto error;
 
     /* This is a PEP 3147 path.  Start by copying everything from the
        start of pathname up to and including the leftmost slash.  Then
        copy the file's basename, removing the magic tag and adding a .py
        suffix.
     */
-    Py_UNICODE_strncpy(buf, pathname, (i=left-pathname));
-    Py_UNICODE_strncpy(buf+i, right+1, (j=dot0-right));
+    Py_UCS4_strncpy(buf, pathname, (i=left-pathname));
+    Py_UCS4_strncpy(buf+i, right+1, (j=dot0-right));
     buf[i+j] = 'p';
     buf[i+j+1] = 'y';
-    return PyUnicode_FromUnicode(buf, i+j+2);
+    PyMem_Free(pathname);
+    return PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, buf, i+j+2);
+  error:
+    PyMem_Free(pathname);
+    return NULL;
 }
 
 /* Given a pathname for a Python source file, its time of last
@@ -1250,6 +1268,7 @@
 write_compiled_module(PyCodeObject *co, PyObject *cpathname,
                       struct stat *srcstat)
 {
+    Py_UCS4 *cpathname_ucs4;
     FILE *fp;
     time_t mtime = srcstat->st_mtime;
 #ifdef MS_WINDOWS   /* since Windows uses different permissions  */
@@ -1267,18 +1286,23 @@
     PyObject *cpathbytes;
 #endif
     PyObject *dirname;
-    Py_UNICODE *dirsep;
+    Py_UCS4 *dirsep;
     int res, ok;
 
     /* Ensure that the __pycache__ directory exists. */
-    dirsep = rightmost_sep(PyUnicode_AS_UNICODE(cpathname));
+    cpathname_ucs4 = PyUnicode_AsUCS4Copy(cpathname);
+    if (!cpathname_ucs4)
+        return;
+    dirsep = rightmost_sep(cpathname_ucs4);
     if (dirsep == NULL) {
         if (Py_VerboseFlag)
             PySys_FormatStderr("# no %s path found %R\n", CACHEDIR, cpathname);
         return;
     }
-    dirname = PyUnicode_FromUnicode(PyUnicode_AS_UNICODE(cpathname),
-                                    dirsep - PyUnicode_AS_UNICODE(cpathname));
+    dirname = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                        cpathname_ucs4,
+                                        dirsep - cpathname_ucs4);
+    PyMem_Free(cpathname_ucs4);
     if (dirname == NULL) {
         PyErr_Clear();
         return;
@@ -1461,9 +1485,7 @@
         goto error;
     }
 #endif
-    cpathname = make_compiled_pathname(
-        PyUnicode_AS_UNICODE(pathname),
-        !Py_OptimizeFlag);
+    cpathname = make_compiled_pathname(pathname, !Py_OptimizeFlag);
 
     if (cpathname != NULL)
         fpc = check_compiled_module(pathname, st.st_mtime, cpathname);
@@ -1512,16 +1534,18 @@
 get_sourcefile(PyObject *filename)
 {
     Py_ssize_t len;
-    Py_UNICODE *fileuni;
+    Py_UCS4 *fileuni;
     PyObject *py;
     struct stat statbuf;
 
-    len = PyUnicode_GET_SIZE(filename);
+    len = PyUnicode_GET_LENGTH(filename);
     if (len == 0)
         Py_RETURN_NONE;
 
     /* don't match *.pyc or *.pyo? */
-    fileuni = PyUnicode_AS_UNICODE(filename);
+    fileuni = PyUnicode_AsUCS4Copy(filename);
+    if (!fileuni)
+        return NULL;
     if (len < 5
         || fileuni[len-4] != '.'
         || (fileuni[len-3] != 'p' && fileuni[len-3] != 'P')
@@ -1535,7 +1559,7 @@
     py = make_source_pathname(filename);
     if (py == NULL) {
         PyErr_Clear();
-        py = PyUnicode_FromUnicode(fileuni, len - 1);
+        py = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, fileuni, len - 1);
     }
     if (py == NULL)
         goto error;
@@ -1548,6 +1572,7 @@
 error:
     PyErr_Clear();
 unchanged:
+    PyMem_Free(fileuni);
     Py_INCREF(filename);
     return filename;
 }
@@ -1739,8 +1764,7 @@
                  PyObject *path_hooks, PyObject *path_importer_cache,
                  PyObject **p_path, PyObject **p_loader, struct filedescr **p_fd)
 {
-    Py_UNICODE buf[MAXPATHLEN+1];
-    Py_ssize_t buflen = MAXPATHLEN+1;
+    Py_UCS4 buf[MAXPATHLEN+1];
     PyObject *path_unicode, *filename;
     Py_ssize_t len;
     struct stat statbuf;
@@ -1759,15 +1783,15 @@
     else
         return 0;
 
-    len = PyUnicode_GET_SIZE(path_unicode);
-    if (len + 2 + PyUnicode_GET_SIZE(name) + MAXSUFFIXSIZE >= buflen) {
+    len = PyUnicode_GET_LENGTH(path_unicode);
+    if (!PyUnicode_AsUCS4(path_unicode, buf, PY_ARRAY_LENGTH(buf), 1)) {
         Py_DECREF(path_unicode);
-        return 0; /* Too long */
+        PyErr_Clear();
+        return 0;
     }
-    Py_UNICODE_strcpy(buf, PyUnicode_AS_UNICODE(path_unicode));
     Py_DECREF(path_unicode);
 
-    if (Py_UNICODE_strlen(buf) != len)
+    if (Py_UCS4_strlen(buf) != len)
         return 0; /* path contains '\0' */
 
     /* sys.path_hooks import hook */
@@ -1804,10 +1828,14 @@
 #endif
         )
         buf[len++] = SEP;
-    Py_UNICODE_strcpy(buf+len, PyUnicode_AS_UNICODE(name));
-    len += PyUnicode_GET_SIZE(name);
-
-    filename = PyUnicode_FromUnicode(buf, len);
+    if (!PyUnicode_AsUCS4(name, buf+len, PY_ARRAY_LENGTH(buf)-len, 1)) {
+        PyErr_Clear();
+        return 0;
+    }
+    len += PyUnicode_GET_LENGTH(name);
+
+    filename = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                         buf, len);
     if (filename == NULL)
         return -1;
 
@@ -1989,6 +2017,12 @@
     if (p_loader != NULL)
         *p_loader = NULL;
 
+    if (PyUnicode_GET_LENGTH(name) > MAXPATHLEN) {
+        PyErr_SetString(PyExc_OverflowError,
+                        "module name is too long");
+        return NULL;
+    }
+
     /* sys.meta_path import hook */
     if (p_loader != NULL) {
         PyObject *meta_path;
@@ -2704,7 +2738,7 @@
                             int level);
 static PyObject *load_next(PyObject *mod, PyObject *altmod,
                            PyObject *inputname, PyObject **p_outputname,
-                           Py_UNICODE *buf, Py_ssize_t *p_buflen,
+                           Py_UCS4 *buf, Py_ssize_t *p_buflen,
                            Py_ssize_t bufsize);
 static int mark_miss(PyObject *name);
 static int ensure_fromlist(PyObject *mod, PyObject *fromlist,
@@ -2718,37 +2752,47 @@
 import_module_level(PyObject *name, PyObject *globals, PyObject *locals,
                     PyObject *fromlist, int level)
 {
-    Py_UNICODE buf[MAXPATHLEN+1];
+    Py_UCS4 buf[MAXPATHLEN+1];
     Py_ssize_t buflen;
     Py_ssize_t bufsize = MAXPATHLEN+1;
     PyObject *parent, *head, *next, *tail, *inputname, *outputname;
     PyObject *parent_name, *ensure_name;
-    const Py_UNICODE *nameunicode;
-
-    nameunicode = PyUnicode_AS_UNICODE(name);
-
-    if (Py_UNICODE_strchr(nameunicode, SEP) != NULL
+    Py_ssize_t sep, altsep;
+
+    if (PyUnicode_READY(name))
+        return NULL;
+
+    sep = PyUnicode_FindChar(name, SEP, 0, PyUnicode_GET_LENGTH(name), 1);
+    if (sep == -2)
+        return NULL;
 #ifdef ALTSEP
-        || Py_UNICODE_strchr(nameunicode, ALTSEP) != NULL
+    altsep = PyUnicode_FindChar(name, ALTSEP, 0, PyUnicode_GET_LENGTH(name), 1);
+    if (altsep == -2)
+        return NULL;
+#else
+    altsep = -1;
 #endif
-        ) {
+    if (sep != -1 || altsep != -1)
+    {
         PyErr_SetString(PyExc_ImportError,
                         "Import by filename is not supported.");
         return NULL;
     }
 
     parent = get_parent(globals, &parent_name, level);
-    if (parent == NULL)
+    if (parent == NULL) {
         return NULL;
-
-    buflen = PyUnicode_GET_SIZE(parent_name);
-    if (buflen+1 > bufsize) {
+    }
+
+    if (PyUnicode_READY(parent_name))
+        return NULL;
+    buflen = PyUnicode_GET_LENGTH(parent_name);
+    if (!PyUnicode_AsUCS4(parent_name, buf, PY_ARRAY_LENGTH(buf), 1)) {
         Py_DECREF(parent_name);
         PyErr_SetString(PyExc_ValueError,
                         "Module name too long");
         return NULL;
     }
-    Py_UNICODE_strcpy(buf, PyUnicode_AS_UNICODE(parent_name));
     Py_DECREF(parent_name);
 
     head = load_next(parent, level < 0 ? Py_None : parent, name, &outputname,
@@ -2799,7 +2843,8 @@
 
     Py_DECREF(head);
 
-    ensure_name = PyUnicode_FromUnicode(buf, Py_UNICODE_strlen(buf));
+    ensure_name = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                            buf, Py_UCS4_strlen(buf));
     if (ensure_name == NULL) {
         Py_DECREF(tail);
         return NULL;
@@ -2859,8 +2904,6 @@
 static PyObject *
 get_parent(PyObject *globals, PyObject **p_name, int level)
 {
-    Py_UNICODE name[MAXPATHLEN+1];
-    const Py_ssize_t bufsize = MAXPATHLEN+1;
     PyObject *nameobj;
 
     static PyObject *namestr = NULL;
@@ -2897,7 +2940,7 @@
                             "__package__ set to non-string");
             return NULL;
         }
-        if (PyUnicode_GET_SIZE(pkgname) == 0) {
+        if (PyUnicode_GET_LENGTH(pkgname) == 0) {
             if (level > 0) {
                 PyErr_SetString(PyExc_ValueError,
                     "Attempted relative import in non-package");
@@ -2905,12 +2948,8 @@
             }
             goto return_none;
         }
-        if (PyUnicode_GET_SIZE(pkgname)+1 > bufsize) {
-            PyErr_SetString(PyExc_ValueError,
-                            "Package name too long");
-            return NULL;
-        }
-        Py_UNICODE_strcpy(name, PyUnicode_AS_UNICODE(pkgname));
+        Py_INCREF(pkgname);
+        nameobj = pkgname;
     } else {
         /* __package__ not set, so figure it out and set it */
         modname = PyDict_GetItem(globals, namestr);
@@ -2922,74 +2961,71 @@
             /* __path__ is set, so modname is already the package name */
             int error;
 
-            if (PyUnicode_GET_SIZE(modname)+1 > bufsize) {
-                PyErr_SetString(PyExc_ValueError,
-                                "Module name too long");
-                return NULL;
-            }
-            Py_UNICODE_strcpy(name, PyUnicode_AS_UNICODE(modname));
             error = PyDict_SetItem(globals, pkgstr, modname);
             if (error) {
                 PyErr_SetString(PyExc_ValueError,
                                 "Could not set __package__");
                 return NULL;
             }
+            Py_INCREF(modname);
+            nameobj = modname;
         } else {
             /* Normal module, so work out the package name if any */
-            Py_UNICODE *start = PyUnicode_AS_UNICODE(modname);
-            Py_UNICODE *lastdot = Py_UNICODE_strrchr(start, '.');
             Py_ssize_t len;
-            int error;
-            if (lastdot == NULL && level > 0) {
-                PyErr_SetString(PyExc_ValueError,
-                    "Attempted relative import in non-package");
+            len = PyUnicode_FindChar(modname, '.',
+                                     0, PyUnicode_GET_LENGTH(modname), -1);
+            if (len == -2)
                 return NULL;
-            }
-            if (lastdot == NULL) {
-                error = PyDict_SetItem(globals, pkgstr, Py_None);
-                if (error) {
+            if (len < 0) {
+                if (level > 0) {
+                    PyErr_SetString(PyExc_ValueError,
+                        "Attempted relative import in non-package");
+                    return NULL;
+                }
+                if (PyDict_SetItem(globals, pkgstr, Py_None)) {
                     PyErr_SetString(PyExc_ValueError,
                         "Could not set __package__");
                     return NULL;
                 }
                 goto return_none;
             }
-            len = lastdot - start;
-            if (len+1 > bufsize) {
-                PyErr_SetString(PyExc_ValueError,
-                                "Module name too long");
+            pkgname = PyUnicode_Substring(modname, 0, len);
+            if (pkgname == NULL)
                 return NULL;
-            }
-            Py_UNICODE_strncpy(name, start, len);
-            name[len] = '\0';
-            pkgname = PyUnicode_FromUnicode(name, len);
-            if (pkgname == NULL) {
-                return NULL;
-            }
-            error = PyDict_SetItem(globals, pkgstr, pkgname);
-            Py_DECREF(pkgname);
-            if (error) {
+            if (PyDict_SetItem(globals, pkgstr, pkgname)) {
+                Py_DECREF(pkgname);
                 PyErr_SetString(PyExc_ValueError,
                                 "Could not set __package__");
                 return NULL;
             }
+            nameobj = pkgname;
         }
     }
-    while (--level > 0) {
-        Py_UNICODE *dot = Py_UNICODE_strrchr(name, '.');
-        if (dot == NULL) {
-            PyErr_SetString(PyExc_ValueError,
-                "Attempted relative import beyond "
-                "toplevel package");
+    if (level > 1) {
+        Py_ssize_t dot, end = PyUnicode_GET_LENGTH(nameobj);
+        PyObject *newname;
+        while (--level > 0) {
+            dot = PyUnicode_FindChar(nameobj, '.', 0, end, -1);
+            if (dot == -2) {
+                Py_DECREF(nameobj);
+                return NULL;
+            }
+            if (dot < 0) {
+                Py_DECREF(nameobj);
+                PyErr_SetString(PyExc_ValueError,
+                    "Attempted relative import beyond "
+                    "toplevel package");
+                return NULL;
+            }
+            end = dot;
+        }
+        newname = PyUnicode_Substring(nameobj, 0, end);
+        Py_DECREF(nameobj);
+        if (newname == NULL)
             return NULL;
-        }
-        *dot = '\0';
+        nameobj = newname;
     }
 
-    nameobj = PyUnicode_FromUnicode(name, Py_UNICODE_strlen(name));
-    if (nameobj == NULL)
-        return NULL;
-
     modules = PyImport_GetModuleDict();
     parent = PyDict_GetItem(modules, nameobj);
     if (parent == NULL) {
@@ -3021,7 +3057,7 @@
        If this is violated...  Who cares? */
 
 return_none:
-    nameobj = PyUnicode_FromUnicode(NULL, 0);
+    nameobj = PyUnicode_New(0, 0);
     if (nameobj == NULL)
         return NULL;
     *p_name = nameobj;
@@ -3032,28 +3068,28 @@
 static PyObject *
 load_next(PyObject *mod, PyObject *altmod,
           PyObject *inputname, PyObject **p_outputname,
-          Py_UNICODE *buf, Py_ssize_t *p_buflen, Py_ssize_t bufsize)
+          Py_UCS4 *buf, Py_ssize_t *p_buflen, Py_ssize_t bufsize)
 {
-    const Py_UNICODE *dot;
+    Py_UCS4 *dot;
     Py_ssize_t len;
-    Py_UNICODE *p;
+    Py_UCS4 *p;
     PyObject *fullname, *name, *result, *mark_name;
-    const Py_UNICODE *nameuni;
+    const Py_UCS4 *nameuni;
 
     *p_outputname = NULL;
 
-    if (PyUnicode_GET_SIZE(inputname) == 0) {
+    if (PyUnicode_GET_LENGTH(inputname) == 0) {
         /* completely empty module name should only happen in
            'from . import' (or '__import__("")')*/
         Py_INCREF(mod);
         return mod;
     }
 
-    nameuni = PyUnicode_AS_UNICODE(inputname);
+    nameuni = PyUnicode_AsUCS4Copy(inputname);
     if (nameuni == NULL)
         return NULL;
 
-    dot = Py_UNICODE_strchr(nameuni, '.');
+    dot = Py_UCS4_strchr(nameuni, '.');
     if (dot != NULL) {
         len = dot - nameuni;
         if (len == 0) {
@@ -3063,7 +3099,7 @@
         }
     }
     else
-        len = PyUnicode_GET_SIZE(inputname);
+        len = PyUnicode_GET_LENGTH(inputname);
 
     if (*p_buflen+len+1 >= bufsize) {
         PyErr_SetString(PyExc_ValueError,
@@ -3076,14 +3112,16 @@
         *p++ = '.';
         *p_buflen += 1;
     }
-    Py_UNICODE_strncpy(p, nameuni, len);
+    Py_UCS4_strncpy(p, nameuni, len);
     p[len] = '\0';
     *p_buflen += len;
 
-    fullname = PyUnicode_FromUnicode(buf, *p_buflen);
+    fullname = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                         buf, *p_buflen);
     if (fullname == NULL)
         return NULL;
-    name = PyUnicode_FromUnicode(p, len);
+    name = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                     p, len);
     if (name == NULL) {
         Py_DECREF(fullname);
         return NULL;
@@ -3096,7 +3134,8 @@
         result = import_submodule(altmod, name, name);
         Py_DECREF(name);
         if (result != NULL && result != Py_None) {
-            mark_name = PyUnicode_FromUnicode(buf, *p_buflen);
+            mark_name = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                                  buf, *p_buflen);
             if (mark_name == NULL) {
                 Py_DECREF(result);
                 return NULL;
@@ -3107,7 +3146,7 @@
                 return NULL;
             }
             Py_DECREF(mark_name);
-            Py_UNICODE_strncpy(buf, nameuni, len);
+            Py_UCS4_strncpy(buf, nameuni, len);
             buf[len] = '\0';
             *p_buflen = len;
         }
@@ -3125,7 +3164,8 @@
     }
 
     if (dot != NULL) {
-        *p_outputname = PyUnicode_FromUnicode(dot+1, Py_UNICODE_strlen(dot+1));
+        *p_outputname = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                                  dot+1, Py_UCS4_strlen(dot+1));
         if (*p_outputname == NULL) {
             Py_DECREF(result);
             return NULL;
@@ -3166,7 +3206,7 @@
             Py_DECREF(item);
             return 0;
         }
-        if (PyUnicode_AS_UNICODE(item)[0] == '*') {
+        if (PyUnicode_READ_CHAR(item, 0) == '*') {
             PyObject *all;
             Py_DECREF(item);
             /* See if the package defines __all__ */
@@ -3304,7 +3344,7 @@
     PyObject *modules = PyImport_GetModuleDict();
     PyObject *path_list = NULL, *loader = NULL, *existing_m = NULL;
     PyObject *nameobj, *bufobj, *subnameobj;
-    Py_UNICODE *name, *subname;
+    Py_UCS4 *name = NULL, *subname;
     struct filedescr *fdp;
     FILE *fp = NULL;
     PyObject *newm = NULL;
@@ -3321,7 +3361,7 @@
         return NULL;
     }
     nameobj = PyModule_GetNameObject(m);
-    if (nameobj == NULL)
+    if (nameobj == NULL || PyUnicode_READY(nameobj) == -1)
         return NULL;
     if (m != PyDict_GetItem(modules, nameobj)) {
         PyErr_Format(PyExc_ImportError,
@@ -3343,8 +3383,12 @@
         return NULL;
     }
 
-    name = PyUnicode_AS_UNICODE(nameobj);
-    subname = Py_UNICODE_strrchr(name, '.');
+    name = PyUnicode_AsUCS4Copy(nameobj);
+    if (!name) {
+        Py_DECREF(nameobj);
+        return NULL;
+    }
+    subname = Py_UCS4_strrchr(name, '.');
     if (subname == NULL) {
         Py_INCREF(nameobj);
         subnameobj = nameobj;
@@ -3353,7 +3397,8 @@
         PyObject *parentname, *parent;
         Py_ssize_t len;
         len = subname - name;
-        parentname = PyUnicode_FromUnicode(name, len);
+        parentname = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                               name, len);
         if (parentname == NULL) {
             goto error;
         }
@@ -3370,8 +3415,9 @@
         if (path_list == NULL)
             PyErr_Clear();
         subname++;
-        len = PyUnicode_GET_SIZE(nameobj) - (len + 1);
-        subnameobj = PyUnicode_FromUnicode(subname, len);
+        len = PyUnicode_GET_LENGTH(nameobj) - (len + 1);
+        subnameobj = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                               subname, len);
     }
     if (subnameobj == NULL)
         goto error;
@@ -3403,6 +3449,7 @@
 error:
     imp_modules_reloading_clear();
     Py_DECREF(nameobj);
+    PyMem_Free(name);
     return newm;
 }
 
@@ -3910,9 +3957,7 @@
         return NULL;
     }
 
-    cpathname = make_compiled_pathname(
-        PyUnicode_AS_UNICODE(pathname),
-        debug);
+    cpathname = make_compiled_pathname(pathname, debug);
     Py_DECREF(pathname);
 
     if (cpathname == NULL) {
@@ -4105,7 +4150,7 @@
                           &pathobj))
         return -1;
 
-    if (PyUnicode_GET_SIZE(pathobj) == 0) {
+    if (PyUnicode_GET_LENGTH(pathobj) == 0) {
         PyErr_SetString(PyExc_ImportError, "empty pathname");
         return -1;
     }
diff --git a/Python/marshal.c b/Python/marshal.c
--- a/Python/marshal.c
+++ b/Python/marshal.c
@@ -311,9 +311,7 @@
     }
     else if (PyUnicode_CheckExact(v)) {
         PyObject *utf8;
-        utf8 = PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(v),
-                                    PyUnicode_GET_SIZE(v),
-                                    "surrogatepass");
+        utf8 = PyUnicode_AsEncodedString(v, "utf8", "surrogatepass");
         if (utf8 == NULL) {
             p->depth--;
             p->error = WFERR_UNMARSHALLABLE;
diff --git a/Python/peephole.c b/Python/peephole.c
--- a/Python/peephole.c
+++ b/Python/peephole.c
@@ -183,24 +183,6 @@
             break;
         case BINARY_SUBSCR:
             newconst = PyObject_GetItem(v, w);
-            /* #5057: if v is unicode, there might be differences between
-               wide and narrow builds in cases like '\U00012345'[0].
-               Wide builds will return a non-BMP char, whereas narrow builds
-               will return a surrogate.  In both the cases skip the
-               optimization in order to produce compatible pycs.
-             */
-            if (newconst != NULL &&
-                PyUnicode_Check(v) && PyUnicode_Check(newconst)) {
-                Py_UNICODE ch = PyUnicode_AS_UNICODE(newconst)[0];
-#ifdef Py_UNICODE_WIDE
-                if (ch > 0xFFFF) {
-#else
-                if (ch >= 0xD800 && ch <= 0xDFFF) {
-#endif
-                    Py_DECREF(newconst);
-                    return 0;
-                }
-            }
             break;
         case BINARY_LSHIFT:
             newconst = PyNumber_Lshift(v, w);
diff --git a/Python/symtable.c b/Python/symtable.c
--- a/Python/symtable.c
+++ b/Python/symtable.c
@@ -1525,10 +1525,10 @@
     */
     PyObject *store_name;
     PyObject *name = (a->asname == NULL) ? a->name : a->asname;
-    const Py_UNICODE *base = PyUnicode_AS_UNICODE(name);
-    Py_UNICODE *dot = Py_UNICODE_strchr(base, '.');
-    if (dot) {
-        store_name = PyUnicode_FromUnicode(base, dot - base);
+    Py_ssize_t dot = PyUnicode_FindChar(name, '.', 0,
+                                        PyUnicode_GET_LENGTH(name), 1);
+    if (dot != -1) {
+        store_name = PyUnicode_Substring(name, 0, dot);
         if (!store_name)
             return 0;
     }
diff --git a/Python/traceback.c b/Python/traceback.c
--- a/Python/traceback.c
+++ b/Python/traceback.c
@@ -229,8 +229,8 @@
     PyObject *lineobj = NULL;
     PyObject *res;
     char buf[MAXPATHLEN+1];
-    Py_UNICODE *u, *p;
-    Py_ssize_t len;
+    int kind;
+    void *data;
 
     /* open the file */
     if (filename == NULL)
@@ -285,13 +285,16 @@
     }
 
     /* remove the indentation of the line */
-    u = PyUnicode_AS_UNICODE(lineobj);
-    len = PyUnicode_GET_SIZE(lineobj);
-    for (p=u; *p == ' ' || *p == '\t' || *p == '\014'; p++)
-        len--;
-    if (u != p) {
+    kind = PyUnicode_KIND(lineobj);
+    data = PyUnicode_DATA(lineobj);
+    for (i=0; i < PyUnicode_GET_LENGTH(lineobj); i++) {
+        Py_UCS4 ch = PyUnicode_READ(kind, data, i);
+        if (ch != ' ' && ch != '\t' && ch != '\014')
+            break;
+    }
+    if (i) {
         PyObject *truncated;
-        truncated = PyUnicode_FromUnicode(p, len);
+        truncated = PyUnicode_Substring(lineobj, i, PyUnicode_GET_LENGTH(lineobj));
         if (truncated) {
             Py_DECREF(lineobj);
             lineobj = truncated;
@@ -476,13 +479,26 @@
 static void
 dump_ascii(int fd, PyObject *text)
 {
+    PyASCIIObject *ascii = (PyASCIIObject *)text;
     Py_ssize_t i, size;
     int truncated;
-    Py_UNICODE *u;
-    char c;
+    int kind;
+    void *data;
+    Py_UCS4 ch;
 
-    size = PyUnicode_GET_SIZE(text);
-    u = PyUnicode_AS_UNICODE(text);
+    size = ascii->length;
+    kind = ascii->state.kind;
+    if (ascii->state.compact) {
+        if (ascii->state.ascii)
+            data = ((PyASCIIObject*)text) + 1;
+        else
+            data = ((PyCompactUnicodeObject*)text) + 1;
+    }
+    else {
+        data = ((PyUnicodeObject *)text)->data.any;
+        if (data == NULL)
+            return;
+    }
 
     if (MAX_STRING_LENGTH < size) {
         size = MAX_STRING_LENGTH;
@@ -491,27 +507,28 @@
     else
         truncated = 0;
 
-    for (i=0; i < size; i++, u++) {
-        if (*u < 128) {
-            c = (char)*u;
+    for (i=0; i < size; i++) {
+        ch = PyUnicode_READ(kind, data, i);
+        if (ch < 128) {
+            char c = (char)ch;
             write(fd, &c, 1);
         }
-        else if (*u < 256) {
+        else if (ch < 256) {
             PUTS(fd, "\\x");
-            dump_hexadecimal(2, *u, fd);
+            dump_hexadecimal(2, ch, fd);
         }
         else
 #ifdef Py_UNICODE_WIDE
-        if (*u < 65536)
+        if (ch < 65536)
 #endif
         {
             PUTS(fd, "\\u");
-            dump_hexadecimal(4, *u, fd);
+            dump_hexadecimal(4, ch, fd);
 #ifdef Py_UNICODE_WIDE
         }
         else {
             PUTS(fd, "\\U");
-            dump_hexadecimal(8, *u, fd);
+            dump_hexadecimal(8, ch, fd);
 #endif
         }
     }
@@ -542,7 +559,7 @@
     }
 
     /* PyFrame_GetLineNumber() was introduced in Python 2.7.0 and 3.2.0 */
-    lineno = PyCode_Addr2Line(frame->f_code, frame->f_lasti);
+    lineno = PyCode_Addr2Line(code, frame->f_lasti);
     PUTS(fd, ", line ");
     dump_decimal(fd, lineno);
     PUTS(fd, " in ");
diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py
--- a/Tools/gdb/libpython.py
+++ b/Tools/gdb/libpython.py
@@ -51,6 +51,8 @@
 _type_void_ptr = gdb.lookup_type('void').pointer() # void*
 _type_size_t = gdb.lookup_type('size_t')
 
+_is_pep393 = 'data' in [f.name for f in gdb.lookup_type('PyUnicodeObject').target().fields()]
+
 SIZEOF_VOID_P = _type_void_ptr.sizeof
 
 
@@ -1123,11 +1125,30 @@
         #     Py_ssize_t length;  /* Length of raw Unicode data in buffer */
         #     Py_UNICODE *str;    /* Raw Unicode buffer */
         field_length = long(self.field('length'))
-        field_str = self.field('str')
+        if _is_pep393:
+            # Python 3.3 and newer
+            may_have_surrogates = False
+            field_state = long(self.field('state'))
+            repr_kind = (field_state & 0xC) >> 2
+            if repr_kind == 0:
+                # string is not ready
+                may_have_surrogates = True
+                field_str = self.field('wstr')
+                field_length = self.field('wstr_length')
+            elif repr_kind == 1:
+                field_str = self.field('data')['latin1']
+            elif repr_kind == 2:
+                field_str = self.field('data')['ucs2']
+            elif repr_kind == 3:
+                field_str = self.field('data')['ucs4']
+        else:
+            # Python 3.2 and earlier
+            field_str = self.field('str')
+            may_have_surrogates = self.char_width() == 2
 
         # Gather a list of ints from the Py_UNICODE array; these are either
-        # UCS-2 or UCS-4 code points:
-        if self.char_width() > 2:
+        # UCS-1, UCS-2 or UCS-4 code points:
+        if not may_have_surrogates:
             Py_UNICODEs = [int(field_str[i]) for i in safe_range(field_length)]
         else:
             # A more elaborate routine if sizeof(Py_UNICODE) is 2 in the
diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -767,7 +767,6 @@
 with_libm
 with_libc
 enable_big_digits
-with_wide_unicode
 with_computed_gotos
 '
       ac_precious_vars='build_alias
@@ -778,7 +777,8 @@
 LDFLAGS
 LIBS
 CPPFLAGS
-CPP'
+CPP
+CPPFLAGS'
 
 
 # Initialize some variables set by options.
@@ -1438,7 +1438,6 @@
   --with-fpectl           enable SIGFPE catching
   --with-libm=STRING      math library
   --with-libc=STRING      C library
-  --with-wide-unicode     Use 4-byte Unicode characters (default is 2 bytes)
   --with(out)-computed-gotos
                           Use computed gotos in evaluation loop (enabled by
                           default on supported compilers)
@@ -12370,65 +12369,19 @@
 $as_echo "$ac_cv_wchar_t_signed" >&6; }
 fi
 
-{ $as_echo "$as_me:${as_lineno-$LINENO}: checking what type to use for str" >&5
-$as_echo_n "checking what type to use for str... " >&6; }
-
-# Check whether --with-wide-unicode was given.
-if test "${with_wide_unicode+set}" = set; then :
-  withval=$with_wide_unicode;
-if test "$withval" != no
-then unicode_size="4"
-else unicode_size="2"
-fi
-
-else
-
-case "$have_ucs4_tcl" in
-  yes) unicode_size="4";;
-  *)   unicode_size="2" ;;
-esac
-
-fi
-
-
-
-case "$unicode_size" in
-  4)
-     $as_echo "#define Py_UNICODE_SIZE 4" >>confdefs.h
-
-     ABIFLAGS="${ABIFLAGS}u"
-     ;;
-  *) $as_echo "#define Py_UNICODE_SIZE 2" >>confdefs.h
- ;;
-esac
-
-
-
 # wchar_t is only usable if it maps to an unsigned type
-if test "$unicode_size" = "$ac_cv_sizeof_wchar_t" \
+if test "$ac_cv_sizeof_wchar_t" -ge 2 \
           -a "$ac_cv_wchar_t_signed" = "no"
 then
-  PY_UNICODE_TYPE="wchar_t"
+  HAVE_USABLE_WCHAR_T="yes"
 
 $as_echo "#define HAVE_USABLE_WCHAR_T 1" >>confdefs.h
 
-  $as_echo "#define PY_UNICODE_TYPE wchar_t" >>confdefs.h
-
-elif test "$ac_cv_sizeof_short" = "$unicode_size"
-then
-     PY_UNICODE_TYPE="unsigned short"
-     $as_echo "#define PY_UNICODE_TYPE unsigned short" >>confdefs.h
-
-elif test "$ac_cv_sizeof_long" = "$unicode_size"
-then
-     PY_UNICODE_TYPE="unsigned long"
-     $as_echo "#define PY_UNICODE_TYPE unsigned long" >>confdefs.h
-
-else
-     PY_UNICODE_TYPE="no type found"
-fi
-{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $PY_UNICODE_TYPE" >&5
-$as_echo "$PY_UNICODE_TYPE" >&6; }
+else
+  HAVE_USABLE_WCHAR_T="no usable wchar_t found"
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $HAVE_USABLE_WCHAR_T" >&5
+$as_echo "$HAVE_USABLE_WCHAR_T" >&6; }
 
 # check for endianness
  { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether byte ordering is bigendian" >&5
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -3615,57 +3615,19 @@
   AC_MSG_RESULT($ac_cv_wchar_t_signed)
 fi
 
-AC_MSG_CHECKING(what type to use for str)
-AC_ARG_WITH(wide-unicode, 
-            AS_HELP_STRING([--with-wide-unicode], [Use 4-byte Unicode characters (default is 2 bytes)]),
-[
-if test "$withval" != no
-then unicode_size="4"
-else unicode_size="2"
-fi
-],
-[
-case "$have_ucs4_tcl" in
-  yes) unicode_size="4";;
-  *)   unicode_size="2" ;;
-esac
-])
-
-AH_TEMPLATE(Py_UNICODE_SIZE,
-  [Define as the size of the unicode type.])
-case "$unicode_size" in
-  4)
-     AC_DEFINE(Py_UNICODE_SIZE, 4)
-     ABIFLAGS="${ABIFLAGS}u"
-     ;;
-  *) AC_DEFINE(Py_UNICODE_SIZE, 2) ;;
-esac
-
-AH_TEMPLATE(PY_UNICODE_TYPE,
-  [Define as the integral type used for Unicode representation.])
-
 # wchar_t is only usable if it maps to an unsigned type
-if test "$unicode_size" = "$ac_cv_sizeof_wchar_t" \
+if test "$ac_cv_sizeof_wchar_t" -ge 2 \
           -a "$ac_cv_wchar_t_signed" = "no"
 then
-  PY_UNICODE_TYPE="wchar_t"
+  HAVE_USABLE_WCHAR_T="yes"
   AC_DEFINE(HAVE_USABLE_WCHAR_T, 1,
   [Define if you have a useable wchar_t type defined in wchar.h; useable
    means wchar_t must be an unsigned type with at least 16 bits. (see
    Include/unicodeobject.h).])
-  AC_DEFINE(PY_UNICODE_TYPE,wchar_t)
-elif test "$ac_cv_sizeof_short" = "$unicode_size"
-then
-     PY_UNICODE_TYPE="unsigned short"
-     AC_DEFINE(PY_UNICODE_TYPE,unsigned short)
-elif test "$ac_cv_sizeof_long" = "$unicode_size"
-then
-     PY_UNICODE_TYPE="unsigned long"
-     AC_DEFINE(PY_UNICODE_TYPE,unsigned long)
 else
-     PY_UNICODE_TYPE="no type found"
+  HAVE_USABLE_WCHAR_T="no usable wchar_t found"
 fi
-AC_MSG_RESULT($PY_UNICODE_TYPE)
+AC_MSG_RESULT($HAVE_USABLE_WCHAR_T)
 
 # check for endianness
 AC_C_BIGENDIAN
diff --git a/pyconfig.h.in b/pyconfig.h.in
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -1111,18 +1111,12 @@
 /* Define to printf format modifier for Py_ssize_t */
 #undef PY_FORMAT_SIZE_T
 
-/* Define as the integral type used for Unicode representation. */
-#undef PY_UNICODE_TYPE
-
 /* Define if you want to build an interpreter with many run-time checks. */
 #undef Py_DEBUG
 
 /* Defined if Python is built as a shared library. */
 #undef Py_ENABLE_SHARED
 
-/* Define as the size of the unicode type. */
-#undef Py_UNICODE_SIZE
-
 /* assume C89 semantics that RETSIGTYPE is always void */
 #undef RETSIGTYPE
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 08:36:00 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Wed, 28 Sep 2011 08:36:00 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Update_for_PEP_393=2E?=
Message-ID: <E1R8nkO-0007ZF-O5@dinsdale.python.org>

http://hg.python.org/cpython/rev/69a6c56e8989
changeset:   72476:69a6c56e8989
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Wed Sep 28 08:35:25 2011 +0200
summary:
  Update for PEP 393.

files:
  Tools/gdb/libpython.py |  37 ++++++++++++++++++++---------
  1 files changed, 25 insertions(+), 12 deletions(-)


diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py
--- a/Tools/gdb/libpython.py
+++ b/Tools/gdb/libpython.py
@@ -50,6 +50,8 @@
 _type_unsigned_char_ptr = gdb.lookup_type('unsigned char').pointer() # unsigned char*
 _type_void_ptr = gdb.lookup_type('void').pointer() # void*
 _type_size_t = gdb.lookup_type('size_t')
+_type_unsigned_short_ptr = gdb.lookup_type('unsigned short').pointer()
+_type_unsigned_int_ptr = gdb.lookup_type('unsigned int').pointer()
 
 _is_pep393 = 'data' in [f.name for f in gdb.lookup_type('PyUnicodeObject').target().fields()]
 
@@ -1124,25 +1126,36 @@
         # From unicodeobject.h:
         #     Py_ssize_t length;  /* Length of raw Unicode data in buffer */
         #     Py_UNICODE *str;    /* Raw Unicode buffer */
-        field_length = long(self.field('length'))
         if _is_pep393:
             # Python 3.3 and newer
             may_have_surrogates = False
-            field_state = long(self.field('state'))
-            repr_kind = (field_state & 0xC) >> 2
-            if repr_kind == 0:
+            compact = self.field('_base')
+            ascii = compact['_base']
+            state = ascii['state']
+            field_length = long(ascii['length'])
+            if not int(state['ready']):
                 # string is not ready
                 may_have_surrogates = True
-                field_str = self.field('wstr')
-                field_length = self.field('wstr_length')
-            elif repr_kind == 1:
-                field_str = self.field('data')['latin1']
-            elif repr_kind == 2:
-                field_str = self.field('data')['ucs2']
-            elif repr_kind == 3:
-                field_str = self.field('data')['ucs4']
+                field_str = ascii['wstr']
+                if not int(state['ascii']):
+                    field_length = compact('wstr_length')
+            else:
+                if int(state['ascii']):
+                    field_str = ascii.address + 1
+                elif int(state['compact']):
+                    field_str = compact.address + 1
+                else:
+                    field_str = self.field('data')['any']
+                repr_kind = int(state['kind'])
+                if repr_kind == 1:
+                    field_str = field_str.cast(_type_unsigned_char_ptr)
+                elif repr_kind == 2:
+                    field_str = field_str.cast(_type_unsigned_short_ptr)
+                elif repr_kind == 3:
+                    field_str = field_str.cast(_type_unsigned_int_ptr)
         else:
             # Python 3.2 and earlier
+            field_length = long(self.field('length'))
             field_str = self.field('str')
             may_have_surrogates = self.char_width() == 2
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 08:54:26 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Wed, 28 Sep 2011 08:54:26 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_ACKS_for_PEP_393=2E?=
Message-ID: <E1R8o2E-0000dy-Du@dinsdale.python.org>

http://hg.python.org/cpython/rev/bbb0c4fdd90c
changeset:   72477:bbb0c4fdd90c
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Wed Sep 28 08:54:17 2011 +0200
summary:
  ACKS for PEP 393.

files:
  Misc/NEWS |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,7 +10,8 @@
 Core and Builtins
 -----------------
 
-- PEP 393: flexible string representation.
+- PEP 393: flexible string representation. Thanks to Torsten Becker for the
+  initial implementation, and Victor Stinner for various bug fixes.
 
 - Issue #13012: The 'keepends' parameter to str.splitlines may now be passed
   as a keyword argument:  "my_string.splitlines(keepends=True)".  The same

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 09:15:27 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Wed, 28 Sep 2011 09:15:27 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Use_eval_instead_of_codecs?=
 =?utf8?q?=2Elookup_to_trigger_UTF-8_generation=2E?=
Message-ID: <E1R8oMZ-0002bq-So@dinsdale.python.org>

http://hg.python.org/cpython/rev/6d2d45ab084d
changeset:   72478:6d2d45ab084d
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Wed Sep 28 09:15:11 2011 +0200
summary:
  Use eval instead of codecs.lookup to trigger UTF-8 generation.

files:
  Lib/test/test_sys.py |  10 ++++------
  1 files changed, 4 insertions(+), 6 deletions(-)


diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py
--- a/Lib/test/test_sys.py
+++ b/Lib/test/test_sys.py
@@ -857,12 +857,10 @@
         s = chr(0x4000)   # 4 bytes canonical representation
         check(s, size(compactfields) + 4)
         try:
-            # FIXME: codecs.lookup(str) calls encoding.search_function() which
-            # calls __import__ using str in the module name. __import__ encodes
-            # the module name to the file system encoding (which is the locale
-            # encoding), so test_sys fails if the locale encoding is not UTF-8.
-            codecs.lookup(s) # produces 4 bytes UTF-8
-        except LookupError:
+            # eval() will trigger the generation of the UTF-8 representation
+            # as a side effect
+            eval(s)
+        except NameError:
             check(s, size(compactfields) + 4 + 4)
         # TODO: add check that forces the presence of wchar_t representation
         # TODO: add check that forces layout of unicodefields

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 09:22:19 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Wed, 28 Sep 2011 09:22:19 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Use_compile=28=29_instead_o?=
	=?utf8?b?ZiBldmFsKCku?=
Message-ID: <E1R8oTD-0003Al-Qc@dinsdale.python.org>

http://hg.python.org/cpython/rev/636444a2827e
changeset:   72479:636444a2827e
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Wed Sep 28 09:22:13 2011 +0200
summary:
  Use compile() instead of eval().

files:
  Lib/test/test_sys.py |  10 ++++------
  1 files changed, 4 insertions(+), 6 deletions(-)


diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py
--- a/Lib/test/test_sys.py
+++ b/Lib/test/test_sys.py
@@ -856,12 +856,10 @@
         # verify that the UTF-8 size is accounted for
         s = chr(0x4000)   # 4 bytes canonical representation
         check(s, size(compactfields) + 4)
-        try:
-            # eval() will trigger the generation of the UTF-8 representation
-            # as a side effect
-            eval(s)
-        except NameError:
-            check(s, size(compactfields) + 4 + 4)
+        # compile() will trigger the generation of the UTF-8
+        # representation as a side effect
+        compile(s, "<stdin>", "eval")
+        check(s, size(compactfields) + 4 + 4)
         # TODO: add check that forces the presence of wchar_t representation
         # TODO: add check that forces layout of unicodefields
         # weakref

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 10:03:35 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Wed, 28 Sep 2011 10:03:35 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_struct_sizes=2E_Drop_-1?=
 =?utf8?q?=2C_since_the_resulting_string_was_actually_the_largest?=
Message-ID: <E1R8p79-0006rU-4M@dinsdale.python.org>

http://hg.python.org/cpython/rev/f19af08eef02
changeset:   72480:f19af08eef02
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Wed Sep 28 10:03:28 2011 +0200
summary:
  Fix struct sizes. Drop -1, since the resulting string was actually the largest one
that could be allocated.

files:
  Lib/test/test_unicode.py |  13 +++++++------
  1 files changed, 7 insertions(+), 6 deletions(-)


diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -1585,12 +1585,12 @@
     def test_raiseMemError(self):
         if struct.calcsize('P') == 8:
             # 64 bits pointers
-            ascii_struct_size = 64
-            compact_struct_size = 88
+            ascii_struct_size = 48
+            compact_struct_size = 72
         else:
             # 32 bits pointers
-            ascii_struct_size = 32
-            compact_struct_size = 44
+            ascii_struct_size = 24
+            compact_struct_size = 36
 
         for char in ('a', '\xe9', '\u20ac', '\U0010ffff'):
             code = ord(char)
@@ -1604,8 +1604,9 @@
                 char_size = 4  # sizeof(Py_UCS4)
                 struct_size = compact_struct_size
             # Note: sys.maxsize is half of the actual max allocation because of
-            # the signedness of Py_ssize_t. -1 because of the null character.
-            maxlen = ((sys.maxsize - struct_size) // char_size) - 1
+            # the signedness of Py_ssize_t. Strings of maxlen-1 should in principle
+            # be allocatable, given enough memory.
+            maxlen = ((sys.maxsize - struct_size) // char_size)
             alloc = lambda: char * maxlen
             self.assertRaises(MemoryError, alloc)
             self.assertRaises(MemoryError, alloc)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 13:36:10 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 28 Sep 2011 13:36:10 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_test=5Fctypes=3A_Windows_is?=
	=?utf8?q?_no_more_a_special_case?=
Message-ID: <E1R8sQs-000138-Np@dinsdale.python.org>

http://hg.python.org/cpython/rev/7f851d36c6ca
changeset:   72481:7f851d36c6ca
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 28 13:36:20 2011 +0200
summary:
  test_ctypes: Windows is no more a special case

files:
  Lib/ctypes/test/test_parameters.py |  9 +++------
  1 files changed, 3 insertions(+), 6 deletions(-)


diff --git a/Lib/ctypes/test/test_parameters.py b/Lib/ctypes/test/test_parameters.py
--- a/Lib/ctypes/test/test_parameters.py
+++ b/Lib/ctypes/test/test_parameters.py
@@ -73,13 +73,10 @@
         except ImportError:
 ##            print "(No c_wchar_p)"
             return
-        s = "123"
-        if sys.platform == "win32":
-            self.assertTrue(c_wchar_p.from_param(s)._obj is s)
-            self.assertRaises(TypeError, c_wchar_p.from_param, 42)
 
-            # new in 0.9.1: convert (decode) ascii to unicode
-            self.assertEqual(c_wchar_p.from_param("123")._obj, "123")
+        c_wchar_p.from_param("123")
+
+        self.assertRaises(TypeError, c_wchar_p.from_param, 42)
         self.assertRaises(TypeError, c_wchar_p.from_param, b"123\377")
 
         pa = c_wchar_p.from_param(c_wchar_p("123"))

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 14:09:11 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Wed, 28 Sep 2011 14:09:11 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_don=27t_check_that_the_firs?=
 =?utf8?q?t_character_is_XID=5FContinue?=
Message-ID: <E1R8swp-0004fU-K1@dinsdale.python.org>

http://hg.python.org/cpython/rev/37932fd9433a
changeset:   72482:37932fd9433a
user:        Benjamin Peterson <benjamin at python.org>
date:        Wed Sep 28 08:09:05 2011 -0400
summary:
  don't check that the first character is XID_Continue

Current, XID_Continue is a superset of XID_Start, but that may sometime change.

files:
  Lib/test/test_pep3131.py |  5 +----
  Objects/unicodeobject.c  |  2 +-
  2 files changed, 2 insertions(+), 5 deletions(-)


diff --git a/Lib/test/test_pep3131.py b/Lib/test/test_pep3131.py
--- a/Lib/test/test_pep3131.py
+++ b/Lib/test/test_pep3131.py
@@ -19,10 +19,7 @@
         ??????? = 1
         # On wide builds, this is normalized, but on narrow ones it is not. See
         # #12746.
-        try:
-            self.assertIn("???????", dir())
-        except AssertionError:
-            raise unittest.case._ExpectedFailure(sys.exc_info())
+        self.assertIn("Unicode", dir())
 
     def test_invalid(self):
         try:
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -10196,7 +10196,7 @@
     if (!_PyUnicode_IsXidStart(first) && first != 0x5F /* LOW LINE */)
         return 0;
 
-    for (i = 0; i < PyUnicode_GET_LENGTH(self); i++)
+    for (i = 1; i < PyUnicode_GET_LENGTH(self); i++)
         if (!_PyUnicode_IsXidContinue(PyUnicode_READ(kind, data, i)))
             return 0;
     return 1;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 14:20:26 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Wed, 28 Sep 2011 14:20:26 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_revert_unintended_change?=
Message-ID: <E1R8t7i-0006LW-4z@dinsdale.python.org>

http://hg.python.org/cpython/rev/d38a82804c1d
changeset:   72483:d38a82804c1d
user:        Benjamin Peterson <benjamin at python.org>
date:        Wed Sep 28 08:19:25 2011 -0400
summary:
  revert unintended change

files:
  Lib/test/test_pep3131.py |  5 ++++-
  1 files changed, 4 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_pep3131.py b/Lib/test/test_pep3131.py
--- a/Lib/test/test_pep3131.py
+++ b/Lib/test/test_pep3131.py
@@ -19,7 +19,10 @@
         ??????? = 1
         # On wide builds, this is normalized, but on narrow ones it is not. See
         # #12746.
-        self.assertIn("Unicode", dir())
+        try:
+            self.assertIn("???????", dir())
+        except AssertionError:
+            raise unittest.case._ExpectedFailure(sys.exc_info())
 
     def test_invalid(self):
         try:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 14:20:27 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Wed, 28 Sep 2011 14:20:27 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_this_test_works_as_expected?=
	=?utf8?q?_now?=
Message-ID: <E1R8t7j-0006Lk-13@dinsdale.python.org>

http://hg.python.org/cpython/rev/3548149a806c
changeset:   72484:3548149a806c
user:        Benjamin Peterson <benjamin at python.org>
date:        Wed Sep 28 08:20:00 2011 -0400
summary:
  this test works as expected now

files:
  Lib/test/test_pep3131.py |  7 +------
  1 files changed, 1 insertions(+), 6 deletions(-)


diff --git a/Lib/test/test_pep3131.py b/Lib/test/test_pep3131.py
--- a/Lib/test/test_pep3131.py
+++ b/Lib/test/test_pep3131.py
@@ -17,12 +17,7 @@
 
     def test_non_bmp_normalized(self):
         ??????? = 1
-        # On wide builds, this is normalized, but on narrow ones it is not. See
-        # #12746.
-        try:
-            self.assertIn("???????", dir())
-        except AssertionError:
-            raise unittest.case._ExpectedFailure(sys.exc_info())
+        self.assertIn("Unicode", dir())
 
     def test_invalid(self):
         try:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 16:38:06 2011
From: python-checkins at python.org (ezio.melotti)
Date: Wed, 28 Sep 2011 16:38:06 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_=2313012=3A_use_splitlines?=
 =?utf8?q?=28keepends=3DTrue/False=29_instead_of_splitlines=280/1=29=2E?=
Message-ID: <E1R8vGw-0003uz-If@dinsdale.python.org>

http://hg.python.org/cpython/rev/83f43b58c988
changeset:   72485:83f43b58c988
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Wed Sep 28 17:37:55 2011 +0300
summary:
  #13012: use splitlines(keepends=True/False) instead of splitlines(0/1).

files:
  Lib/argparse.py                 |   2 +-
  Lib/codecs.py                   |  14 +++++++-------
  Lib/collections/__init__.py     |   2 +-
  Lib/difflib.py                  |  20 ++++++++++----------
  Lib/doctest.py                  |   6 +++---
  Lib/lib2to3/refactor.py         |   4 ++--
  Lib/string.py                   |   2 +-
  Lib/test/test_calendar.py       |   2 +-
  Lib/test/test_email/__init__.py |   2 +-
  Lib/test/test_gzip.py           |   4 ++--
  Lib/test/test_inspect.py        |   2 +-
  Lib/test/test_io.py             |   4 ++--
  Lib/test/test_nntplib.py        |   4 ++--
  Lib/test/test_pydoc.py          |   4 ++--
  Lib/test/test_tokenize.py       |   2 +-
  Lib/unittest/case.py            |   4 ++--
  16 files changed, 39 insertions(+), 39 deletions(-)


diff --git a/Lib/argparse.py b/Lib/argparse.py
--- a/Lib/argparse.py
+++ b/Lib/argparse.py
@@ -641,7 +641,7 @@
     """
 
     def _fill_text(self, text, width, indent):
-        return ''.join([indent + line for line in text.splitlines(True)])
+        return ''.join(indent + line for line in text.splitlines(keepends=True))
 
 
 class RawTextHelpFormatter(RawDescriptionHelpFormatter):
diff --git a/Lib/codecs.py b/Lib/codecs.py
--- a/Lib/codecs.py
+++ b/Lib/codecs.py
@@ -484,7 +484,7 @@
                 if firstline:
                     newchars, decodedbytes = \
                         self.decode(data[:exc.start], self.errors)
-                    lines = newchars.splitlines(True)
+                    lines = newchars.splitlines(keepends=True)
                     if len(lines)<=1:
                         raise
                 else:
@@ -526,7 +526,7 @@
                 self.charbuffer = self.linebuffer[0]
                 self.linebuffer = None
             if not keepends:
-                line = line.splitlines(False)[0]
+                line = line.splitlines(keepends=False)[0]
             return line
 
         readsize = size or 72
@@ -543,7 +543,7 @@
                     data += self.read(size=1, chars=1)
 
             line += data
-            lines = line.splitlines(True)
+            lines = line.splitlines(keepends=True)
             if lines:
                 if len(lines) > 1:
                     # More than one line result; the first line is a full line
@@ -559,10 +559,10 @@
                         # only one remaining line, put it back into charbuffer
                         self.charbuffer = lines[0] + self.charbuffer
                     if not keepends:
-                        line = line.splitlines(False)[0]
+                        line = line.splitlines(keepends=False)[0]
                     break
                 line0withend = lines[0]
-                line0withoutend = lines[0].splitlines(False)[0]
+                line0withoutend = lines[0].splitlines(keepends=False)[0]
                 if line0withend != line0withoutend: # We really have a line end
                     # Put the rest back together and keep it until the next call
                     self.charbuffer = self._empty_charbuffer.join(lines[1:]) + \
@@ -575,7 +575,7 @@
             # we didn't get anything or this was our only try
             if not data or size is not None:
                 if line and not keepends:
-                    line = line.splitlines(False)[0]
+                    line = line.splitlines(keepends=False)[0]
                 break
             if readsize < 8000:
                 readsize *= 2
@@ -803,7 +803,7 @@
 
         data = self.reader.read()
         data, bytesencoded = self.encode(data, self.errors)
-        return data.splitlines(1)
+        return data.splitlines(keepends=True)
 
     def __next__(self):
 
diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py
--- a/Lib/collections/__init__.py
+++ b/Lib/collections/__init__.py
@@ -1045,7 +1045,7 @@
         return self.data.split(sep, maxsplit)
     def rsplit(self, sep=None, maxsplit=-1):
         return self.data.rsplit(sep, maxsplit)
-    def splitlines(self, keepends=0): return self.data.splitlines(keepends)
+    def splitlines(self, keepends=False): return self.data.splitlines(keepends)
     def startswith(self, prefix, start=0, end=_sys.maxsize):
         return self.data.startswith(prefix, start, end)
     def strip(self, chars=None): return self.__class__(self.data.strip(chars))
diff --git a/Lib/difflib.py b/Lib/difflib.py
--- a/Lib/difflib.py
+++ b/Lib/difflib.py
@@ -800,7 +800,7 @@
     ...   2. Explicit is better than implicit.
     ...   3. Simple is better than complex.
     ...   4. Complex is better than complicated.
-    ... '''.splitlines(1)
+    ... '''.splitlines(keepends=True)
     >>> len(text1)
     4
     >>> text1[0][-1]
@@ -809,7 +809,7 @@
     ...   3.   Simple is better than complex.
     ...   4. Complicated is better than complex.
     ...   5. Flat is better than nested.
-    ... '''.splitlines(1)
+    ... '''.splitlines(keepends=True)
 
     Next we instantiate a Differ object:
 
@@ -896,8 +896,8 @@
 
         Example:
 
-        >>> print(''.join(Differ().compare('one\ntwo\nthree\n'.splitlines(1),
-        ...                                'ore\ntree\nemu\n'.splitlines(1))),
+        >>> print(''.join(Differ().compare('one\ntwo\nthree\n'.splitlines(True),
+        ...                                'ore\ntree\nemu\n'.splitlines(True))),
         ...       end="")
         - one
         ?  ^
@@ -1269,8 +1269,8 @@
 
     Example:
 
-    >>> print(''.join(context_diff('one\ntwo\nthree\nfour\n'.splitlines(1),
-    ...       'zero\none\ntree\nfour\n'.splitlines(1), 'Original', 'Current')),
+    >>> print(''.join(context_diff('one\ntwo\nthree\nfour\n'.splitlines(True),
+    ...       'zero\none\ntree\nfour\n'.splitlines(True), 'Original', 'Current')),
     ...       end="")
     *** Original
     --- Current
@@ -1339,8 +1339,8 @@
 
     Example:
 
-    >>> diff = ndiff('one\ntwo\nthree\n'.splitlines(1),
-    ...              'ore\ntree\nemu\n'.splitlines(1))
+    >>> diff = ndiff('one\ntwo\nthree\n'.splitlines(keepends=True),
+    ...              'ore\ntree\nemu\n'.splitlines(keepends=True))
     >>> print(''.join(diff), end="")
     - one
     ?  ^
@@ -2034,8 +2034,8 @@
 
     Examples:
 
-    >>> diff = ndiff('one\ntwo\nthree\n'.splitlines(1),
-    ...              'ore\ntree\nemu\n'.splitlines(1))
+    >>> diff = ndiff('one\ntwo\nthree\n'.splitlines(keepends=True),
+    ...              'ore\ntree\nemu\n'.splitlines(keepends=True))
     >>> diff = list(diff)
     >>> print(''.join(restore(diff, 1)), end="")
     one
diff --git a/Lib/doctest.py b/Lib/doctest.py
--- a/Lib/doctest.py
+++ b/Lib/doctest.py
@@ -1332,7 +1332,7 @@
         m = self.__LINECACHE_FILENAME_RE.match(filename)
         if m and m.group('name') == self.test.name:
             example = self.test.examples[int(m.group('examplenum'))]
-            return example.source.splitlines(True)
+            return example.source.splitlines(keepends=True)
         else:
             return self.save_linecache_getlines(filename, module_globals)
 
@@ -1595,8 +1595,8 @@
         # Check if we should use diff.
         if self._do_a_fancy_diff(want, got, optionflags):
             # Split want & got into lines.
-            want_lines = want.splitlines(True)  # True == keep line ends
-            got_lines = got.splitlines(True)
+            want_lines = want.splitlines(keepends=True)
+            got_lines = got.splitlines(keepends=True)
             # Use difflib to find their differences.
             if optionflags & REPORT_UDIFF:
                 diff = difflib.unified_diff(want_lines, got_lines, n=2)
diff --git a/Lib/lib2to3/refactor.py b/Lib/lib2to3/refactor.py
--- a/Lib/lib2to3/refactor.py
+++ b/Lib/lib2to3/refactor.py
@@ -560,7 +560,7 @@
         block_lineno = None
         indent = None
         lineno = 0
-        for line in input.splitlines(True):
+        for line in input.splitlines(keepends=True):
             lineno += 1
             if line.lstrip().startswith(self.PS1):
                 if block is not None:
@@ -604,7 +604,7 @@
                            filename, lineno, err.__class__.__name__, err)
             return block
         if self.refactor_tree(tree, filename):
-            new = str(tree).splitlines(True)
+            new = str(tree).splitlines(keepends=True)
             # Undo the adjustment of the line numbers in wrap_toks() below.
             clipped, new = new[:lineno-1], new[lineno-1:]
             assert clipped == ["\n"] * (lineno-1), clipped
diff --git a/Lib/string.py b/Lib/string.py
--- a/Lib/string.py
+++ b/Lib/string.py
@@ -84,7 +84,7 @@
 
     def _invalid(self, mo):
         i = mo.start('invalid')
-        lines = self.template[:i].splitlines(True)
+        lines = self.template[:i].splitlines(keepends=True)
         if not lines:
             colno = 1
             lineno = 1
diff --git a/Lib/test/test_calendar.py b/Lib/test/test_calendar.py
--- a/Lib/test/test_calendar.py
+++ b/Lib/test/test_calendar.py
@@ -177,7 +177,7 @@
             return not c.isspace() and not c.isdigit()
 
         lines = []
-        for line in s.splitlines(False):
+        for line in s.splitlines(keepends=False):
             # Drop texts, as they are locale dependent
             if line and not filter(neitherspacenordigit, line):
                 lines.append(line)
diff --git a/Lib/test/test_email/__init__.py b/Lib/test/test_email/__init__.py
--- a/Lib/test/test_email/__init__.py
+++ b/Lib/test/test_email/__init__.py
@@ -38,7 +38,7 @@
             return email.message_from_file(fp)
 
     def _bytes_repr(self, b):
-        return [repr(x) for x in b.splitlines(True)]
+        return [repr(x) for x in b.splitlines(keepends=True)]
 
     def assertBytesEqual(self, first, second, msg):
         """Our byte strings are really encoded strings; improve diff output"""
diff --git a/Lib/test/test_gzip.py b/Lib/test/test_gzip.py
--- a/Lib/test/test_gzip.py
+++ b/Lib/test/test_gzip.py
@@ -139,7 +139,7 @@
             with io.BufferedReader(f) as r:
                 lines = [line for line in r]
 
-        self.assertEqual(lines, 50 * data1.splitlines(True))
+        self.assertEqual(lines, 50 * data1.splitlines(keepends=True))
 
     def test_readline(self):
         self.test_write()
@@ -340,7 +340,7 @@
 
     def test_textio_readlines(self):
         # Issue #10791: TextIOWrapper.readlines() fails when wrapping GzipFile.
-        lines = (data1 * 50).decode("ascii").splitlines(True)
+        lines = (data1 * 50).decode("ascii").splitlines(keepends=True)
         self.test_write()
         with gzip.GzipFile(self.filename, 'r') as f:
             with io.TextIOWrapper(f, encoding="ascii") as t:
diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py
--- a/Lib/test/test_inspect.py
+++ b/Lib/test/test_inspect.py
@@ -304,7 +304,7 @@
         getlines = linecache.getlines
         def monkey(filename, module_globals=None):
             if filename == fn:
-                return source.splitlines(True)
+                return source.splitlines(keepends=True)
             else:
                 return getlines(filename, module_globals)
         linecache.getlines = monkey
diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py
--- a/Lib/test/test_io.py
+++ b/Lib/test/test_io.py
@@ -1935,8 +1935,8 @@
         testdata = b"AAA\nBB\x00B\nCCC\rDDD\rEEE\r\nFFF\r\nGGG"
         normalized = testdata.replace(b"\r\n", b"\n").replace(b"\r", b"\n")
         for newline, expected in [
-            (None, normalized.decode("ascii").splitlines(True)),
-            ("", testdata.decode("ascii").splitlines(True)),
+            (None, normalized.decode("ascii").splitlines(keepends=True)),
+            ("", testdata.decode("ascii").splitlines(keepends=True)),
             ("\n", ["AAA\n", "BB\x00B\n", "CCC\rDDD\rEEE\r\n", "FFF\r\n", "GGG"]),
             ("\r\n", ["AAA\nBB\x00B\nCCC\rDDD\rEEE\r\n", "FFF\r\n", "GGG"]),
             ("\r",  ["AAA\nBB\x00B\nCCC\r", "DDD\r", "EEE\r", "\nFFF\r", "\nGGG"]),
diff --git a/Lib/test/test_nntplib.py b/Lib/test/test_nntplib.py
--- a/Lib/test/test_nntplib.py
+++ b/Lib/test/test_nntplib.py
@@ -1033,12 +1033,12 @@
         self.assertEqual(resp, success_resp)
         # With an iterable of terminated lines
         def iterlines(b):
-            return iter(b.splitlines(True))
+            return iter(b.splitlines(keepends=True))
         resp = self._check_post_ihave_sub(func, *args, file_factory=iterlines)
         self.assertEqual(resp, success_resp)
         # With an iterable of non-terminated lines
         def iterlines(b):
-            return iter(b.splitlines(False))
+            return iter(b.splitlines(keepends=False))
         resp = self._check_post_ihave_sub(func, *args, file_factory=iterlines)
         self.assertEqual(resp, success_resp)
 
diff --git a/Lib/test/test_pydoc.py b/Lib/test/test_pydoc.py
--- a/Lib/test/test_pydoc.py
+++ b/Lib/test/test_pydoc.py
@@ -238,8 +238,8 @@
 def print_diffs(text1, text2):
     "Prints unified diffs for two texts"
     # XXX now obsolete, use unittest built-in support
-    lines1 = text1.splitlines(True)
-    lines2 = text2.splitlines(True)
+    lines1 = text1.splitlines(keepends=True)
+    lines2 = text2.splitlines(keepends=True)
     diffs = difflib.unified_diff(lines1, lines2, n=0, fromfile='expected',
                                  tofile='got')
     print('\n' + ''.join(diffs))
diff --git a/Lib/test/test_tokenize.py b/Lib/test/test_tokenize.py
--- a/Lib/test/test_tokenize.py
+++ b/Lib/test/test_tokenize.py
@@ -600,7 +600,7 @@
         f.close()
     tokens1 = [tok[:2] for tok in token_list]
     new_bytes = untokenize(tokens1)
-    readline = (line for line in new_bytes.splitlines(1)).__next__
+    readline = (line for line in new_bytes.splitlines(keepends=True)).__next__
     tokens2 = [tok[:2] for tok in tokenize(readline)]
     return tokens1 == tokens2
 
diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py
--- a/Lib/unittest/case.py
+++ b/Lib/unittest/case.py
@@ -1010,8 +1010,8 @@
             if (len(first) > self._diffThreshold or
                 len(second) > self._diffThreshold):
                 self._baseAssertEqual(first, second, msg)
-            firstlines = first.splitlines(True)
-            secondlines = second.splitlines(True)
+            firstlines = first.splitlines(keepends=True)
+            secondlines = second.splitlines(keepends=True)
             if len(firstlines) == 1 and first.strip('\r\n') == first:
                 firstlines = [first + '\n']
                 secondlines = [second + '\n']

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 16:48:44 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Wed, 28 Sep 2011 16:48:44 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_this_isn=27t_fixed_on_windo?=
	=?utf8?b?d3MgeWV0Li4u?=
Message-ID: <E1R8vRE-0005I3-Ko@dinsdale.python.org>

http://hg.python.org/cpython/rev/347c950fb663
changeset:   72486:347c950fb663
parent:      72484:3548149a806c
user:        Benjamin Peterson <benjamin at python.org>
date:        Wed Sep 28 10:48:32 2011 -0400
summary:
  this isn't fixed on windows yet...

files:
  Lib/test/test_pep3131.py |  5 ++++-
  1 files changed, 4 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_pep3131.py b/Lib/test/test_pep3131.py
--- a/Lib/test/test_pep3131.py
+++ b/Lib/test/test_pep3131.py
@@ -17,7 +17,10 @@
 
     def test_non_bmp_normalized(self):
         ??????? = 1
-        self.assertIn("Unicode", dir())
+        try:
+            self.assertIn("Unicode", dir())
+        except AssertionError:
+            raise unittest._ExpectedFailure("doesn't work yet")
 
     def test_invalid(self):
         try:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 16:48:45 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Wed, 28 Sep 2011 16:48:45 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_merge_heads?=
Message-ID: <E1R8vRF-0005IP-QV@dinsdale.python.org>

http://hg.python.org/cpython/rev/5a12416890c0
changeset:   72487:5a12416890c0
parent:      72486:347c950fb663
parent:      72485:83f43b58c988
user:        Benjamin Peterson <benjamin at python.org>
date:        Wed Sep 28 10:48:40 2011 -0400
summary:
  merge heads

files:
  Lib/argparse.py                 |   2 +-
  Lib/codecs.py                   |  14 +++++++-------
  Lib/collections/__init__.py     |   2 +-
  Lib/difflib.py                  |  20 ++++++++++----------
  Lib/doctest.py                  |   6 +++---
  Lib/lib2to3/refactor.py         |   4 ++--
  Lib/string.py                   |   2 +-
  Lib/test/test_calendar.py       |   2 +-
  Lib/test/test_email/__init__.py |   2 +-
  Lib/test/test_gzip.py           |   4 ++--
  Lib/test/test_inspect.py        |   2 +-
  Lib/test/test_io.py             |   4 ++--
  Lib/test/test_nntplib.py        |   4 ++--
  Lib/test/test_pydoc.py          |   4 ++--
  Lib/test/test_tokenize.py       |   2 +-
  Lib/unittest/case.py            |   4 ++--
  16 files changed, 39 insertions(+), 39 deletions(-)


diff --git a/Lib/argparse.py b/Lib/argparse.py
--- a/Lib/argparse.py
+++ b/Lib/argparse.py
@@ -641,7 +641,7 @@
     """
 
     def _fill_text(self, text, width, indent):
-        return ''.join([indent + line for line in text.splitlines(True)])
+        return ''.join(indent + line for line in text.splitlines(keepends=True))
 
 
 class RawTextHelpFormatter(RawDescriptionHelpFormatter):
diff --git a/Lib/codecs.py b/Lib/codecs.py
--- a/Lib/codecs.py
+++ b/Lib/codecs.py
@@ -484,7 +484,7 @@
                 if firstline:
                     newchars, decodedbytes = \
                         self.decode(data[:exc.start], self.errors)
-                    lines = newchars.splitlines(True)
+                    lines = newchars.splitlines(keepends=True)
                     if len(lines)<=1:
                         raise
                 else:
@@ -526,7 +526,7 @@
                 self.charbuffer = self.linebuffer[0]
                 self.linebuffer = None
             if not keepends:
-                line = line.splitlines(False)[0]
+                line = line.splitlines(keepends=False)[0]
             return line
 
         readsize = size or 72
@@ -543,7 +543,7 @@
                     data += self.read(size=1, chars=1)
 
             line += data
-            lines = line.splitlines(True)
+            lines = line.splitlines(keepends=True)
             if lines:
                 if len(lines) > 1:
                     # More than one line result; the first line is a full line
@@ -559,10 +559,10 @@
                         # only one remaining line, put it back into charbuffer
                         self.charbuffer = lines[0] + self.charbuffer
                     if not keepends:
-                        line = line.splitlines(False)[0]
+                        line = line.splitlines(keepends=False)[0]
                     break
                 line0withend = lines[0]
-                line0withoutend = lines[0].splitlines(False)[0]
+                line0withoutend = lines[0].splitlines(keepends=False)[0]
                 if line0withend != line0withoutend: # We really have a line end
                     # Put the rest back together and keep it until the next call
                     self.charbuffer = self._empty_charbuffer.join(lines[1:]) + \
@@ -575,7 +575,7 @@
             # we didn't get anything or this was our only try
             if not data or size is not None:
                 if line and not keepends:
-                    line = line.splitlines(False)[0]
+                    line = line.splitlines(keepends=False)[0]
                 break
             if readsize < 8000:
                 readsize *= 2
@@ -803,7 +803,7 @@
 
         data = self.reader.read()
         data, bytesencoded = self.encode(data, self.errors)
-        return data.splitlines(1)
+        return data.splitlines(keepends=True)
 
     def __next__(self):
 
diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py
--- a/Lib/collections/__init__.py
+++ b/Lib/collections/__init__.py
@@ -1045,7 +1045,7 @@
         return self.data.split(sep, maxsplit)
     def rsplit(self, sep=None, maxsplit=-1):
         return self.data.rsplit(sep, maxsplit)
-    def splitlines(self, keepends=0): return self.data.splitlines(keepends)
+    def splitlines(self, keepends=False): return self.data.splitlines(keepends)
     def startswith(self, prefix, start=0, end=_sys.maxsize):
         return self.data.startswith(prefix, start, end)
     def strip(self, chars=None): return self.__class__(self.data.strip(chars))
diff --git a/Lib/difflib.py b/Lib/difflib.py
--- a/Lib/difflib.py
+++ b/Lib/difflib.py
@@ -800,7 +800,7 @@
     ...   2. Explicit is better than implicit.
     ...   3. Simple is better than complex.
     ...   4. Complex is better than complicated.
-    ... '''.splitlines(1)
+    ... '''.splitlines(keepends=True)
     >>> len(text1)
     4
     >>> text1[0][-1]
@@ -809,7 +809,7 @@
     ...   3.   Simple is better than complex.
     ...   4. Complicated is better than complex.
     ...   5. Flat is better than nested.
-    ... '''.splitlines(1)
+    ... '''.splitlines(keepends=True)
 
     Next we instantiate a Differ object:
 
@@ -896,8 +896,8 @@
 
         Example:
 
-        >>> print(''.join(Differ().compare('one\ntwo\nthree\n'.splitlines(1),
-        ...                                'ore\ntree\nemu\n'.splitlines(1))),
+        >>> print(''.join(Differ().compare('one\ntwo\nthree\n'.splitlines(True),
+        ...                                'ore\ntree\nemu\n'.splitlines(True))),
         ...       end="")
         - one
         ?  ^
@@ -1269,8 +1269,8 @@
 
     Example:
 
-    >>> print(''.join(context_diff('one\ntwo\nthree\nfour\n'.splitlines(1),
-    ...       'zero\none\ntree\nfour\n'.splitlines(1), 'Original', 'Current')),
+    >>> print(''.join(context_diff('one\ntwo\nthree\nfour\n'.splitlines(True),
+    ...       'zero\none\ntree\nfour\n'.splitlines(True), 'Original', 'Current')),
     ...       end="")
     *** Original
     --- Current
@@ -1339,8 +1339,8 @@
 
     Example:
 
-    >>> diff = ndiff('one\ntwo\nthree\n'.splitlines(1),
-    ...              'ore\ntree\nemu\n'.splitlines(1))
+    >>> diff = ndiff('one\ntwo\nthree\n'.splitlines(keepends=True),
+    ...              'ore\ntree\nemu\n'.splitlines(keepends=True))
     >>> print(''.join(diff), end="")
     - one
     ?  ^
@@ -2034,8 +2034,8 @@
 
     Examples:
 
-    >>> diff = ndiff('one\ntwo\nthree\n'.splitlines(1),
-    ...              'ore\ntree\nemu\n'.splitlines(1))
+    >>> diff = ndiff('one\ntwo\nthree\n'.splitlines(keepends=True),
+    ...              'ore\ntree\nemu\n'.splitlines(keepends=True))
     >>> diff = list(diff)
     >>> print(''.join(restore(diff, 1)), end="")
     one
diff --git a/Lib/doctest.py b/Lib/doctest.py
--- a/Lib/doctest.py
+++ b/Lib/doctest.py
@@ -1332,7 +1332,7 @@
         m = self.__LINECACHE_FILENAME_RE.match(filename)
         if m and m.group('name') == self.test.name:
             example = self.test.examples[int(m.group('examplenum'))]
-            return example.source.splitlines(True)
+            return example.source.splitlines(keepends=True)
         else:
             return self.save_linecache_getlines(filename, module_globals)
 
@@ -1595,8 +1595,8 @@
         # Check if we should use diff.
         if self._do_a_fancy_diff(want, got, optionflags):
             # Split want & got into lines.
-            want_lines = want.splitlines(True)  # True == keep line ends
-            got_lines = got.splitlines(True)
+            want_lines = want.splitlines(keepends=True)
+            got_lines = got.splitlines(keepends=True)
             # Use difflib to find their differences.
             if optionflags & REPORT_UDIFF:
                 diff = difflib.unified_diff(want_lines, got_lines, n=2)
diff --git a/Lib/lib2to3/refactor.py b/Lib/lib2to3/refactor.py
--- a/Lib/lib2to3/refactor.py
+++ b/Lib/lib2to3/refactor.py
@@ -560,7 +560,7 @@
         block_lineno = None
         indent = None
         lineno = 0
-        for line in input.splitlines(True):
+        for line in input.splitlines(keepends=True):
             lineno += 1
             if line.lstrip().startswith(self.PS1):
                 if block is not None:
@@ -604,7 +604,7 @@
                            filename, lineno, err.__class__.__name__, err)
             return block
         if self.refactor_tree(tree, filename):
-            new = str(tree).splitlines(True)
+            new = str(tree).splitlines(keepends=True)
             # Undo the adjustment of the line numbers in wrap_toks() below.
             clipped, new = new[:lineno-1], new[lineno-1:]
             assert clipped == ["\n"] * (lineno-1), clipped
diff --git a/Lib/string.py b/Lib/string.py
--- a/Lib/string.py
+++ b/Lib/string.py
@@ -84,7 +84,7 @@
 
     def _invalid(self, mo):
         i = mo.start('invalid')
-        lines = self.template[:i].splitlines(True)
+        lines = self.template[:i].splitlines(keepends=True)
         if not lines:
             colno = 1
             lineno = 1
diff --git a/Lib/test/test_calendar.py b/Lib/test/test_calendar.py
--- a/Lib/test/test_calendar.py
+++ b/Lib/test/test_calendar.py
@@ -177,7 +177,7 @@
             return not c.isspace() and not c.isdigit()
 
         lines = []
-        for line in s.splitlines(False):
+        for line in s.splitlines(keepends=False):
             # Drop texts, as they are locale dependent
             if line and not filter(neitherspacenordigit, line):
                 lines.append(line)
diff --git a/Lib/test/test_email/__init__.py b/Lib/test/test_email/__init__.py
--- a/Lib/test/test_email/__init__.py
+++ b/Lib/test/test_email/__init__.py
@@ -38,7 +38,7 @@
             return email.message_from_file(fp)
 
     def _bytes_repr(self, b):
-        return [repr(x) for x in b.splitlines(True)]
+        return [repr(x) for x in b.splitlines(keepends=True)]
 
     def assertBytesEqual(self, first, second, msg):
         """Our byte strings are really encoded strings; improve diff output"""
diff --git a/Lib/test/test_gzip.py b/Lib/test/test_gzip.py
--- a/Lib/test/test_gzip.py
+++ b/Lib/test/test_gzip.py
@@ -139,7 +139,7 @@
             with io.BufferedReader(f) as r:
                 lines = [line for line in r]
 
-        self.assertEqual(lines, 50 * data1.splitlines(True))
+        self.assertEqual(lines, 50 * data1.splitlines(keepends=True))
 
     def test_readline(self):
         self.test_write()
@@ -340,7 +340,7 @@
 
     def test_textio_readlines(self):
         # Issue #10791: TextIOWrapper.readlines() fails when wrapping GzipFile.
-        lines = (data1 * 50).decode("ascii").splitlines(True)
+        lines = (data1 * 50).decode("ascii").splitlines(keepends=True)
         self.test_write()
         with gzip.GzipFile(self.filename, 'r') as f:
             with io.TextIOWrapper(f, encoding="ascii") as t:
diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py
--- a/Lib/test/test_inspect.py
+++ b/Lib/test/test_inspect.py
@@ -304,7 +304,7 @@
         getlines = linecache.getlines
         def monkey(filename, module_globals=None):
             if filename == fn:
-                return source.splitlines(True)
+                return source.splitlines(keepends=True)
             else:
                 return getlines(filename, module_globals)
         linecache.getlines = monkey
diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py
--- a/Lib/test/test_io.py
+++ b/Lib/test/test_io.py
@@ -1935,8 +1935,8 @@
         testdata = b"AAA\nBB\x00B\nCCC\rDDD\rEEE\r\nFFF\r\nGGG"
         normalized = testdata.replace(b"\r\n", b"\n").replace(b"\r", b"\n")
         for newline, expected in [
-            (None, normalized.decode("ascii").splitlines(True)),
-            ("", testdata.decode("ascii").splitlines(True)),
+            (None, normalized.decode("ascii").splitlines(keepends=True)),
+            ("", testdata.decode("ascii").splitlines(keepends=True)),
             ("\n", ["AAA\n", "BB\x00B\n", "CCC\rDDD\rEEE\r\n", "FFF\r\n", "GGG"]),
             ("\r\n", ["AAA\nBB\x00B\nCCC\rDDD\rEEE\r\n", "FFF\r\n", "GGG"]),
             ("\r",  ["AAA\nBB\x00B\nCCC\r", "DDD\r", "EEE\r", "\nFFF\r", "\nGGG"]),
diff --git a/Lib/test/test_nntplib.py b/Lib/test/test_nntplib.py
--- a/Lib/test/test_nntplib.py
+++ b/Lib/test/test_nntplib.py
@@ -1033,12 +1033,12 @@
         self.assertEqual(resp, success_resp)
         # With an iterable of terminated lines
         def iterlines(b):
-            return iter(b.splitlines(True))
+            return iter(b.splitlines(keepends=True))
         resp = self._check_post_ihave_sub(func, *args, file_factory=iterlines)
         self.assertEqual(resp, success_resp)
         # With an iterable of non-terminated lines
         def iterlines(b):
-            return iter(b.splitlines(False))
+            return iter(b.splitlines(keepends=False))
         resp = self._check_post_ihave_sub(func, *args, file_factory=iterlines)
         self.assertEqual(resp, success_resp)
 
diff --git a/Lib/test/test_pydoc.py b/Lib/test/test_pydoc.py
--- a/Lib/test/test_pydoc.py
+++ b/Lib/test/test_pydoc.py
@@ -238,8 +238,8 @@
 def print_diffs(text1, text2):
     "Prints unified diffs for two texts"
     # XXX now obsolete, use unittest built-in support
-    lines1 = text1.splitlines(True)
-    lines2 = text2.splitlines(True)
+    lines1 = text1.splitlines(keepends=True)
+    lines2 = text2.splitlines(keepends=True)
     diffs = difflib.unified_diff(lines1, lines2, n=0, fromfile='expected',
                                  tofile='got')
     print('\n' + ''.join(diffs))
diff --git a/Lib/test/test_tokenize.py b/Lib/test/test_tokenize.py
--- a/Lib/test/test_tokenize.py
+++ b/Lib/test/test_tokenize.py
@@ -600,7 +600,7 @@
         f.close()
     tokens1 = [tok[:2] for tok in token_list]
     new_bytes = untokenize(tokens1)
-    readline = (line for line in new_bytes.splitlines(1)).__next__
+    readline = (line for line in new_bytes.splitlines(keepends=True)).__next__
     tokens2 = [tok[:2] for tok in tokenize(readline)]
     return tokens1 == tokens2
 
diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py
--- a/Lib/unittest/case.py
+++ b/Lib/unittest/case.py
@@ -1010,8 +1010,8 @@
             if (len(first) > self._diffThreshold or
                 len(second) > self._diffThreshold):
                 self._baseAssertEqual(first, second, msg)
-            firstlines = first.splitlines(True)
-            secondlines = second.splitlines(True)
+            firstlines = first.splitlines(keepends=True)
+            secondlines = second.splitlines(keepends=True)
             if len(firstlines) == 1 and first.strip('\r\n') == first:
                 firstlines = [first + '\n']
                 secondlines = [second + '\n']

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 20:29:28 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 28 Sep 2011 20:29:28 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Set_Py=5FUNICODE=5FREPLACEM?=
 =?utf8?q?ENT=5FCHARACTER_type_to_Py=5FUCS4=2C_instead_of_Py=5FUNICODE?=
Message-ID: <E1R8ysq-0003Ce-R6@dinsdale.python.org>

http://hg.python.org/cpython/rev/17626ba912fb
changeset:   72488:17626ba912fb
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 28 20:29:27 2011 +0200
summary:
  Set Py_UNICODE_REPLACEMENT_CHARACTER type to Py_UCS4, instead of Py_UNICODE

files:
  Include/unicodeobject.h |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -506,7 +506,7 @@
    Unicode character U+FFFD is the official REPLACEMENT CHARACTER in
    Unicode 3.0. */
 
-#define Py_UNICODE_REPLACEMENT_CHARACTER ((Py_UNICODE) 0xFFFD)
+#define Py_UNICODE_REPLACEMENT_CHARACTER ((Py_UCS4) 0xFFFD)
 
 /* === Public API ========================================================= */
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 21:48:29 2011
From: python-checkins at python.org (georg.brandl)
Date: Wed, 28 Sep 2011 21:48:29 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Rename_new_macros_to_confor?=
 =?utf8?q?m_to_naming_rules_=28function_macros_have_=22Py=22_prefix=2C?=
Message-ID: <E1R907J-0002mX-0r@dinsdale.python.org>

http://hg.python.org/cpython/rev/dacac31460c0
changeset:   72489:dacac31460c0
user:        Georg Brandl <georg at python.org>
date:        Wed Sep 28 21:49:49 2011 +0200
summary:
  Rename new macros to conform to naming rules (function macros have "Py" prefix, not "PY").

files:
  Include/Python.h        |   6 +++---
  Modules/arraymodule.c   |   2 +-
  Modules/zipimport.c     |   8 ++++----
  Objects/unicodeobject.c |  18 +++++++++---------
  Python/import.c         |   6 +++---
  5 files changed, 20 insertions(+), 20 deletions(-)


diff --git a/Include/Python.h b/Include/Python.h
--- a/Include/Python.h
+++ b/Include/Python.h
@@ -160,9 +160,9 @@
 #define PyDoc_STR(str) ""
 #endif
 
-#define PY_ARRAY_LENGTH(array) (sizeof(array) / sizeof((array)[0]))
+#define Py_ARRAY_LENGTH(array) (sizeof(array) / sizeof((array)[0]))
 
-#define PY_MIN(x, y) (((x) > (y)) ? (y) : (x))
-#define PY_MAX(x, y) (((x) > (y)) ? (x) : (y))
+#define Py_MIN(x, y) (((x) > (y)) ? (y) : (x))
+#define Py_MAX(x, y) (((x) > (y)) ? (x) : (y))
 
 #endif /* !Py_PYTHON_H */
diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c
--- a/Modules/arraymodule.c
+++ b/Modules/arraymodule.c
@@ -2810,7 +2810,7 @@
 PyInit_array(void)
 {
     PyObject *m;
-    char buffer[PY_ARRAY_LENGTH(descriptors)], *p;
+    char buffer[Py_ARRAY_LENGTH(descriptors)], *p;
     PyObject *typecodes;
     Py_ssize_t size = 0;
     struct arraydescr *descr;
diff --git a/Modules/zipimport.c b/Modules/zipimport.c
--- a/Modules/zipimport.c
+++ b/Modules/zipimport.c
@@ -88,7 +88,7 @@
                         "archive path too long");
         goto error;
     }
-    if (!PyUnicode_AsUCS4(pathobj, buf, PY_ARRAY_LENGTH(buf), 1))
+    if (!PyUnicode_AsUCS4(pathobj, buf, Py_ARRAY_LENGTH(buf), 1))
         goto error;
 
 #ifdef ALTSEP
@@ -473,7 +473,7 @@
         PyErr_SetString(ZipImportError, "path too long");
         return NULL;
     }
-    if (!PyUnicode_AsUCS4(pathobj, buf, PY_ARRAY_LENGTH(buf), 1))
+    if (!PyUnicode_AsUCS4(pathobj, buf, Py_ARRAY_LENGTH(buf), 1))
         return NULL;
     path = buf;
 #ifdef ALTSEP
@@ -484,7 +484,7 @@
 #endif
     len = PyUnicode_GET_LENGTH(self->archive);
     if ((size_t)len < Py_UCS4_strlen(path)) {
-        if (!PyUnicode_AsUCS4(self->archive, archive, PY_ARRAY_LENGTH(archive), 1))
+        if (!PyUnicode_AsUCS4(self->archive, archive, Py_ARRAY_LENGTH(archive), 1))
             return NULL;
         if (Py_UCS4_strncmp(path, archive, len) == 0 &&
             path[len] == SEP) {
@@ -771,7 +771,7 @@
                         "Zip path name is too long");
         return NULL;
     }
-    if (!PyUnicode_AsUCS4(archive, path, PY_ARRAY_LENGTH(path), 1))
+    if (!PyUnicode_AsUCS4(archive, path, Py_ARRAY_LENGTH(path), 1))
         return NULL;
 
     fp = _Py_fopen(archive, "rb");
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -1521,7 +1521,7 @@
             case 'c':
             {
                 Py_UCS4 ordinal = va_arg(count, int);
-                maxchar = PY_MAX(maxchar, ordinal);
+                maxchar = Py_MAX(maxchar, ordinal);
                 n++;
                 break;
             }
@@ -1617,7 +1617,7 @@
                 /* since PyUnicode_DecodeUTF8 returns already flexible
                    unicode objects, there is no need to call ready on them */
                 argmaxchar = PyUnicode_MAX_CHAR_VALUE(str);
-                maxchar = PY_MAX(maxchar, argmaxchar);
+                maxchar = Py_MAX(maxchar, argmaxchar);
                 n += PyUnicode_GET_LENGTH(str);
                 /* Remember the str and switch to the next slot */
                 *callresult++ = str;
@@ -1630,7 +1630,7 @@
                 if (PyUnicode_READY(obj) == -1)
                     goto fail;
                 argmaxchar = PyUnicode_MAX_CHAR_VALUE(obj);
-                maxchar = PY_MAX(maxchar, argmaxchar);
+                maxchar = Py_MAX(maxchar, argmaxchar);
                 n += PyUnicode_GET_LENGTH(obj);
                 break;
             }
@@ -1645,7 +1645,7 @@
                     if (PyUnicode_READY(obj) == -1)
                         goto fail;
                     argmaxchar = PyUnicode_MAX_CHAR_VALUE(obj);
-                    maxchar = PY_MAX(maxchar, argmaxchar);
+                    maxchar = Py_MAX(maxchar, argmaxchar);
                     n += PyUnicode_GET_LENGTH(obj);
                     *callresult++ = NULL;
                 }
@@ -1654,7 +1654,7 @@
                     if (!str_obj)
                         goto fail;
                     argmaxchar = PyUnicode_MAX_CHAR_VALUE(str_obj);
-                    maxchar = PY_MAX(maxchar, argmaxchar);
+                    maxchar = Py_MAX(maxchar, argmaxchar);
                     n += PyUnicode_GET_LENGTH(str_obj);
                     *callresult++ = str_obj;
                 }
@@ -1669,7 +1669,7 @@
                 if (!str || PyUnicode_READY(str) == -1)
                     goto fail;
                 argmaxchar = PyUnicode_MAX_CHAR_VALUE(str);
-                maxchar = PY_MAX(maxchar, argmaxchar);
+                maxchar = Py_MAX(maxchar, argmaxchar);
                 n += PyUnicode_GET_LENGTH(str);
                 /* Remember the str and switch to the next slot */
                 *callresult++ = str;
@@ -1684,7 +1684,7 @@
                 if (!repr || PyUnicode_READY(repr) == -1)
                     goto fail;
                 argmaxchar = PyUnicode_MAX_CHAR_VALUE(repr);
-                maxchar = PY_MAX(maxchar, argmaxchar);
+                maxchar = Py_MAX(maxchar, argmaxchar);
                 n += PyUnicode_GET_LENGTH(repr);
                 /* Remember the repr and switch to the next slot */
                 *callresult++ = repr;
@@ -1699,7 +1699,7 @@
                 if (!ascii || PyUnicode_READY(ascii) == -1)
                     goto fail;
                 argmaxchar = PyUnicode_MAX_CHAR_VALUE(ascii);
-                maxchar = PY_MAX(maxchar, argmaxchar);
+                maxchar = Py_MAX(maxchar, argmaxchar);
                 n += PyUnicode_GET_LENGTH(ascii);
                 /* Remember the repr and switch to the next slot */
                 *callresult++ = ascii;
@@ -11051,7 +11051,7 @@
 
     kind1 = PyUnicode_KIND(str_in);
     kind2 = PyUnicode_KIND(sep_obj);
-    kind = PY_MAX(kind1, kind2);
+    kind = Py_MAX(kind1, kind2);
     buf1 = PyUnicode_DATA(str_in);
     if (kind1 != kind)
         buf1 = _PyUnicode_AsKind(str_in, kind);
diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -1784,7 +1784,7 @@
         return 0;
 
     len = PyUnicode_GET_LENGTH(path_unicode);
-    if (!PyUnicode_AsUCS4(path_unicode, buf, PY_ARRAY_LENGTH(buf), 1)) {
+    if (!PyUnicode_AsUCS4(path_unicode, buf, Py_ARRAY_LENGTH(buf), 1)) {
         Py_DECREF(path_unicode);
         PyErr_Clear();
         return 0;
@@ -1828,7 +1828,7 @@
 #endif
         )
         buf[len++] = SEP;
-    if (!PyUnicode_AsUCS4(name, buf+len, PY_ARRAY_LENGTH(buf)-len, 1)) {
+    if (!PyUnicode_AsUCS4(name, buf+len, Py_ARRAY_LENGTH(buf)-len, 1)) {
         PyErr_Clear();
         return 0;
     }
@@ -2787,7 +2787,7 @@
     if (PyUnicode_READY(parent_name))
         return NULL;
     buflen = PyUnicode_GET_LENGTH(parent_name);
-    if (!PyUnicode_AsUCS4(parent_name, buf, PY_ARRAY_LENGTH(buf), 1)) {
+    if (!PyUnicode_AsUCS4(parent_name, buf, Py_ARRAY_LENGTH(buf), 1)) {
         Py_DECREF(parent_name);
         PyErr_SetString(PyExc_ValueError,
                         "Module name too long");

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 21:49:48 2011
From: python-checkins at python.org (georg.brandl)
Date: Wed, 28 Sep 2011 21:49:48 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_versionadded_directive_?=
	=?utf8?q?to_new_API_function=2E?=
Message-ID: <E1R908a-0002w8-A4@dinsdale.python.org>

http://hg.python.org/cpython/rev/1b16230f1611
changeset:   72490:1b16230f1611
user:        Georg Brandl <georg at python.org>
date:        Wed Sep 28 21:51:06 2011 +0200
summary:
  Add versionadded directive to new API function.

files:
  Doc/c-api/unicode.rst |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst
--- a/Doc/c-api/unicode.rst
+++ b/Doc/c-api/unicode.rst
@@ -1080,6 +1080,8 @@
    first match; a value of ``-1`` indicates that no match was found, and ``-2``
    indicates that an error occurred and an exception has been set.
 
+   .. versionadded:: 3.3
+
 
 .. c:function:: Py_ssize_t PyUnicode_Count(PyObject *str, PyObject *substr, Py_ssize_t start, Py_ssize_t end)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 22:08:35 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 28 Sep 2011 22:08:35 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Mark_PyUnicode=5FCONVERT=5F?=
	=?utf8?q?BYTES_as_private?=
Message-ID: <E1R90Ql-00053i-Ex@dinsdale.python.org>

http://hg.python.org/cpython/rev/256f7c55672f
changeset:   72491:256f7c55672f
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 28 21:39:49 2011 +0200
summary:
  Mark PyUnicode_CONVERT_BYTES as private

files:
  Include/unicodeobject.h |  15 ------------
  Objects/unicodeobject.c |  35 ++++++++++++++++++++--------
  2 files changed, 25 insertions(+), 25 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -468,21 +468,6 @@
      (PyUnicode_IS_READY(op) ?                          \
       0 : _PyUnicode_Ready((PyUnicodeObject *)(op))))
 
-/* Generic helper macro to convert characters of different types.
-   from_type and to_type have to be valid type names, begin and end
-   are pointers to the source characters which should be of type
-   "from_type *".  to is a pointer of type "to_type *" and points to the
-   buffer where the result characters are written to. */
-#define PyUnicode_CONVERT_BYTES(from_type, to_type, begin, end, to) \
-    do { \
-        const from_type *iter_; to_type *to_; \
-        for (iter_ = (begin), to_ = (to_type *)(to); \
-             iter_ < (end); \
-             ++iter_, ++to_) { \
-            *to_ = (to_type)*iter_; \
-        } \
-    } while (0)
-
 /* Return a maximum character value which is suitable for creating another
    string based on op.  This is always an approximation but more efficient
    than interating over the string. */
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -90,6 +90,21 @@
 extern "C" {
 #endif
 
+/* Generic helper macro to convert characters of different types.
+   from_type and to_type have to be valid type names, begin and end
+   are pointers to the source characters which should be of type
+   "from_type *".  to is a pointer of type "to_type *" and points to the
+   buffer where the result characters are written to. */
+#define _PyUnicode_CONVERT_BYTES(from_type, to_type, begin, end, to) \
+    do { \
+        const from_type *iter_; to_type *to_; \
+        for (iter_ = (begin), to_ = (to_type *)(to); \
+             iter_ < (end); \
+             ++iter_, ++to_) { \
+            *to_ = (to_type)*iter_; \
+        } \
+    } while (0)
+
 #define _PyUnicode_WSTR(op) (((PyASCIIObject*)(op))->wstr)
 #define _PyUnicode_WSTR_LENGTH(op) (((PyCompactUnicodeObject*)(op))->wstr_length)
 #define _PyUnicode_LENGTH(op) (((PyASCIIObject *)(op))->length)
@@ -622,7 +637,7 @@
         case PyUnicode_1BYTE_KIND:
             switch (to_kind) {
                 case PyUnicode_2BYTE_KIND:
-                    PyUnicode_CONVERT_BYTES(
+                    _PyUnicode_CONVERT_BYTES(
                         unsigned char, Py_UCS2,
                         PyUnicode_1BYTE_DATA(from) + from_start,
                         PyUnicode_1BYTE_DATA(from) + from_start + how_many,
@@ -630,7 +645,7 @@
                         );
                     break;
                 case PyUnicode_4BYTE_KIND:
-                    PyUnicode_CONVERT_BYTES(
+                    _PyUnicode_CONVERT_BYTES(
                         unsigned char, Py_UCS4,
                         PyUnicode_1BYTE_DATA(from) + from_start,
                         PyUnicode_1BYTE_DATA(from) + from_start + how_many,
@@ -644,7 +659,7 @@
         case PyUnicode_2BYTE_KIND:
             switch (to_kind) {
                 case PyUnicode_1BYTE_KIND:
-                    PyUnicode_CONVERT_BYTES(
+                    _PyUnicode_CONVERT_BYTES(
                         Py_UCS2, unsigned char,
                         PyUnicode_2BYTE_DATA(from) + from_start,
                         PyUnicode_2BYTE_DATA(from) + from_start + how_many,
@@ -652,7 +667,7 @@
                         );
                     break;
                 case PyUnicode_4BYTE_KIND:
-                    PyUnicode_CONVERT_BYTES(
+                    _PyUnicode_CONVERT_BYTES(
                         Py_UCS2, Py_UCS4,
                         PyUnicode_2BYTE_DATA(from) + from_start,
                         PyUnicode_2BYTE_DATA(from) + from_start + how_many,
@@ -666,7 +681,7 @@
         case PyUnicode_4BYTE_KIND:
             switch (to_kind) {
                 case PyUnicode_1BYTE_KIND:
-                    PyUnicode_CONVERT_BYTES(
+                    _PyUnicode_CONVERT_BYTES(
                         Py_UCS4, unsigned char,
                         PyUnicode_4BYTE_DATA(from) + from_start,
                         PyUnicode_4BYTE_DATA(from) + from_start + how_many,
@@ -674,7 +689,7 @@
                         );
                     break;
                 case PyUnicode_2BYTE_KIND:
-                    PyUnicode_CONVERT_BYTES(
+                    _PyUnicode_CONVERT_BYTES(
                         Py_UCS4, Py_UCS2,
                         PyUnicode_4BYTE_DATA(from) + from_start,
                         PyUnicode_4BYTE_DATA(from) + from_start + how_many,
@@ -792,7 +807,7 @@
             PyErr_NoMemory();
             return -1;
         }
-        PyUnicode_CONVERT_BYTES(wchar_t, unsigned char,
+        _PyUnicode_CONVERT_BYTES(wchar_t, unsigned char,
                                 _PyUnicode_WSTR(unicode), end,
                                 PyUnicode_1BYTE_DATA(unicode));
         PyUnicode_1BYTE_DATA(unicode)[_PyUnicode_WSTR_LENGTH(unicode)] = '\0';
@@ -834,7 +849,7 @@
                 PyErr_NoMemory();
                 return -1;
             }
-            PyUnicode_CONVERT_BYTES(wchar_t, Py_UCS2,
+            _PyUnicode_CONVERT_BYTES(wchar_t, Py_UCS2,
                                     _PyUnicode_WSTR(unicode), end,
                                     PyUnicode_2BYTE_DATA(unicode));
             PyUnicode_2BYTE_DATA(unicode)[_PyUnicode_WSTR_LENGTH(unicode)] = '\0';
@@ -1023,14 +1038,14 @@
 
     switch (PyUnicode_KIND(unicode)) {
     case PyUnicode_1BYTE_KIND:
-        PyUnicode_CONVERT_BYTES(Py_UNICODE, unsigned char,
+        _PyUnicode_CONVERT_BYTES(Py_UNICODE, unsigned char,
                                 u, u + size, PyUnicode_1BYTE_DATA(unicode));
         break;
     case PyUnicode_2BYTE_KIND:
 #if Py_UNICODE_SIZE == 2
         Py_MEMCPY(PyUnicode_2BYTE_DATA(unicode), u, size * 2);
 #else
-        PyUnicode_CONVERT_BYTES(Py_UNICODE, Py_UCS2,
+        _PyUnicode_CONVERT_BYTES(Py_UNICODE, Py_UCS2,
                                 u, u + size, PyUnicode_2BYTE_DATA(unicode));
 #endif
         break;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 22:08:36 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 28 Sep 2011 22:08:36 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_PyUnicode=5FCopyCharacters?=
 =?utf8?q?=28=29_checks_for_buffer_and_character_overflow?=
Message-ID: <E1R90Qm-00053s-CH@dinsdale.python.org>

http://hg.python.org/cpython/rev/e69b7f21823f
changeset:   72492:e69b7f21823f
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 28 21:37:03 2011 +0200
summary:
  PyUnicode_CopyCharacters() checks for buffer and character overflow

It now returns the number of written characters on success.

files:
  Include/unicodeobject.h |   18 ++-
  Objects/unicodeobject.c |  165 +++++++++++++--------------
  2 files changed, 95 insertions(+), 88 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -519,10 +519,22 @@
 #endif
 
 /* Copy character from one unicode object into another, this function performs
-   character conversion when nessesary and falls back to memcpy if possible.
-   Return -1 and raise an exception on error, return 0 on success. */
+   character conversion when necessary and falls back to memcpy if possible.
+
+   Fail if 'to' is smaller than how_many or smaller than len(from)-from_start,
+   or if kind(from[from_start:from_start+how_many]) > kind(to).
+
+   Return the number of written character, or return -1 and raise an exception
+   on error.
+
+   Pseudo-code:
+
+       how_many = min(how_many, len(from) - from_start)
+       to[to_start:to_start+how_many] = from[from_start:from_start+how_many]
+       return how_many
+   */
 #ifndef Py_LIMITED_API
-PyAPI_FUNC(int) PyUnicode_CopyCharacters(
+PyAPI_FUNC(Py_ssize_t) PyUnicode_CopyCharacters(
     PyObject *to,
     Py_ssize_t to_start,
     PyObject *from,
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -606,13 +606,13 @@
 }
 #endif
 
-int
+Py_ssize_t
 PyUnicode_CopyCharacters(PyObject *to, Py_ssize_t to_start,
                          PyObject *from, Py_ssize_t from_start,
                          Py_ssize_t how_many)
 {
-    int from_kind;
-    int to_kind;
+    unsigned int from_kind;
+    unsigned int to_kind;
 
     assert(PyUnicode_Check(from));
     assert(PyUnicode_Check(to));
@@ -622,94 +622,89 @@
     if (PyUnicode_READY(to))
         return -1;
 
+    how_many = PY_MIN(PyUnicode_GET_LENGTH(from), how_many);
+    if (to_start + how_many > PyUnicode_GET_LENGTH(to)) {
+        PyErr_Format(PyExc_ValueError,
+                     "Cannot write %zi characters at %zi "
+                     "in a string of %zi characters",
+                     how_many, to_start, PyUnicode_GET_LENGTH(to));
+        return -1;
+    }
+
     from_kind = PyUnicode_KIND(from);
     to_kind = PyUnicode_KIND(to);
 
     if (from_kind == to_kind) {
-        const Py_ssize_t char_size = PyUnicode_CHARACTER_SIZE(to);
-        Py_MEMCPY(PyUnicode_1BYTE_DATA(to) + (to_start * char_size),
-                  PyUnicode_1BYTE_DATA(from) + (from_start * char_size),
-                  how_many * char_size);
-        return 0;
-    }
-
-    switch (from_kind) {
-        case PyUnicode_1BYTE_KIND:
-            switch (to_kind) {
-                case PyUnicode_2BYTE_KIND:
-                    _PyUnicode_CONVERT_BYTES(
-                        unsigned char, Py_UCS2,
-                        PyUnicode_1BYTE_DATA(from) + from_start,
-                        PyUnicode_1BYTE_DATA(from) + from_start + how_many,
-                        PyUnicode_2BYTE_DATA(to) + to_start
-                        );
+        /* fast path */
+        Py_MEMCPY((char*)PyUnicode_DATA(to)
+                      + PyUnicode_KIND_SIZE(to_kind, to_start),
+                  (char*)PyUnicode_DATA(from)
+                      + PyUnicode_KIND_SIZE(from_kind, from_start),
+                  PyUnicode_KIND_SIZE(to_kind, how_many));
+        return how_many;
+    }
+    
+    if (from_kind > to_kind) {
+        /* slow path to check for character overflow */
+        const Py_UCS4 to_maxchar = PyUnicode_MAX_CHAR_VALUE(to);
+        void *from_data = PyUnicode_DATA(from);
+        void *to_data = PyUnicode_DATA(to);
+        Py_UCS4 ch, maxchar;
+        Py_ssize_t i;
+        int overflow;
+
+        maxchar = 0;
+        for (i=0; i < how_many; i++) {
+            ch = PyUnicode_READ(from_kind, from_data, from_start + i);
+            if (ch > maxchar) {
+                maxchar = ch;
+                if (maxchar > to_maxchar) {
+                    overflow = 1;
                     break;
-                case PyUnicode_4BYTE_KIND:
-                    _PyUnicode_CONVERT_BYTES(
-                        unsigned char, Py_UCS4,
-                        PyUnicode_1BYTE_DATA(from) + from_start,
-                        PyUnicode_1BYTE_DATA(from) + from_start + how_many,
-                        PyUnicode_4BYTE_DATA(to) + to_start
-                        );
-                    break;
-                default:
-                    goto invalid_state;
-            }
-            break;
-        case PyUnicode_2BYTE_KIND:
-            switch (to_kind) {
-                case PyUnicode_1BYTE_KIND:
-                    _PyUnicode_CONVERT_BYTES(
-                        Py_UCS2, unsigned char,
-                        PyUnicode_2BYTE_DATA(from) + from_start,
-                        PyUnicode_2BYTE_DATA(from) + from_start + how_many,
-                        PyUnicode_1BYTE_DATA(to) + to_start
-                        );
-                    break;
-                case PyUnicode_4BYTE_KIND:
-                    _PyUnicode_CONVERT_BYTES(
-                        Py_UCS2, Py_UCS4,
-                        PyUnicode_2BYTE_DATA(from) + from_start,
-                        PyUnicode_2BYTE_DATA(from) + from_start + how_many,
-                        PyUnicode_4BYTE_DATA(to) + to_start
-                        );
-                    break;
-                default:
-                    goto invalid_state;
-            }
-            break;
-        case PyUnicode_4BYTE_KIND:
-            switch (to_kind) {
-                case PyUnicode_1BYTE_KIND:
-                    _PyUnicode_CONVERT_BYTES(
-                        Py_UCS4, unsigned char,
-                        PyUnicode_4BYTE_DATA(from) + from_start,
-                        PyUnicode_4BYTE_DATA(from) + from_start + how_many,
-                        PyUnicode_1BYTE_DATA(to) + to_start
-                        );
-                    break;
-                case PyUnicode_2BYTE_KIND:
-                    _PyUnicode_CONVERT_BYTES(
-                        Py_UCS4, Py_UCS2,
-                        PyUnicode_4BYTE_DATA(from) + from_start,
-                        PyUnicode_4BYTE_DATA(from) + from_start + how_many,
-                        PyUnicode_2BYTE_DATA(to) + to_start
-                        );
-                    break;
-                default:
-                    goto invalid_state;
-            }
-            break;
-        default:
-            goto invalid_state;
-    }
-    return 0;
-
-invalid_state:
+                }
+            }
+            PyUnicode_WRITE(to_kind, to_data, to_start + i, ch);
+        }
+        if (!overflow)
+            return how_many;
+    }
+    else if (from_kind == PyUnicode_1BYTE_KIND && to_kind == PyUnicode_2BYTE_KIND)
+    {
+        _PyUnicode_CONVERT_BYTES(
+            Py_UCS1, Py_UCS2,
+            PyUnicode_1BYTE_DATA(from) + from_start,
+            PyUnicode_1BYTE_DATA(from) + from_start + how_many,
+            PyUnicode_2BYTE_DATA(to) + to_start
+            );
+        return how_many;
+    }
+    else if (from_kind == PyUnicode_1BYTE_KIND 
+             && to_kind == PyUnicode_4BYTE_KIND)
+    {
+        _PyUnicode_CONVERT_BYTES(
+            Py_UCS1, Py_UCS4,
+            PyUnicode_1BYTE_DATA(from) + from_start,
+            PyUnicode_1BYTE_DATA(from) + from_start + how_many,
+            PyUnicode_4BYTE_DATA(to) + to_start
+            );
+        return how_many;
+    }
+    else if (from_kind == PyUnicode_2BYTE_KIND
+             && to_kind == PyUnicode_4BYTE_KIND)
+    {
+        _PyUnicode_CONVERT_BYTES(
+            Py_UCS2, Py_UCS4,
+            PyUnicode_2BYTE_DATA(from) + from_start,
+            PyUnicode_2BYTE_DATA(from) + from_start + how_many,
+            PyUnicode_4BYTE_DATA(to) + to_start
+            );
+        return how_many;
+    }
     PyErr_Format(PyExc_ValueError,
-                 "Impossible kind state (from=%i, to=%i) "
-                 "in PyUnicode_CopyCharacters",
-                 from_kind, to_kind);
+                 "Cannot copy UCS%u characters "
+                 "into a string of UCS%u characters",
+                 1 << (from_kind - 1), 
+                 1 << (to_kind -1));
     return -1;
 }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 22:08:37 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 28 Sep 2011 22:08:37 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Check_for_PyUnicode=5FCopyC?=
	=?utf8?q?haracters=28=29_failure?=
Message-ID: <E1R90Qn-00053w-66@dinsdale.python.org>

http://hg.python.org/cpython/rev/c4dcb68e0711
changeset:   72493:c4dcb68e0711
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 28 21:39:17 2011 +0200
summary:
  Check for PyUnicode_CopyCharacters() failure

files:
  Objects/unicodeobject.c |  120 ++++++++++++++++++---------
  Python/compile.c        |   10 +-
  2 files changed, 87 insertions(+), 43 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -1784,9 +1784,10 @@
                 (void) va_arg(vargs, char *);
                 size = PyUnicode_GET_LENGTH(*callresult);
                 assert(PyUnicode_KIND(*callresult) <= PyUnicode_KIND(string));
-                PyUnicode_CopyCharacters((PyObject*)string, i,
-                                         *callresult, 0,
-                                         size);
+                if (PyUnicode_CopyCharacters((PyObject*)string, i,
+                                             *callresult, 0,
+                                             size) < 0)
+                    goto fail;
                 i += size;
                 /* We're done with the unicode()/repr() => forget it */
                 Py_DECREF(*callresult);
@@ -1800,9 +1801,10 @@
                 Py_ssize_t size;
                 assert(PyUnicode_KIND(obj) <= PyUnicode_KIND(string));
                 size = PyUnicode_GET_LENGTH(obj);
-                PyUnicode_CopyCharacters((PyObject*)string, i,
-                                         obj, 0,
-                                         size);
+                if (PyUnicode_CopyCharacters((PyObject*)string, i,
+                                             obj, 0,
+                                             size) < 0)
+                    goto fail;
                 i += size;
                 break;
             }
@@ -1814,17 +1816,19 @@
                 if (obj) {
                     size = PyUnicode_GET_LENGTH(obj);
                     assert(PyUnicode_KIND(obj) <= PyUnicode_KIND(string));
-                    PyUnicode_CopyCharacters((PyObject*)string, i,
-                                             obj, 0,
-                                             size);
+                    if (PyUnicode_CopyCharacters((PyObject*)string, i,
+                                                 obj, 0,
+                                                 size) < 0)
+                        goto fail;
                     i += size;
                 } else {
                     size = PyUnicode_GET_LENGTH(*callresult);
                     assert(PyUnicode_KIND(*callresult) <=
                            PyUnicode_KIND(string));
-                    PyUnicode_CopyCharacters((PyObject*)string, i,
-                                             *callresult,
-                                             0, size);
+                    if (PyUnicode_CopyCharacters((PyObject*)string, i,
+                                                 *callresult,
+                                                 0, size) < 0)
+                        goto fail;
                     i += size;
                     Py_DECREF(*callresult);
                 }
@@ -1838,9 +1842,10 @@
                 /* unused, since we already have the result */
                 (void) va_arg(vargs, PyObject *);
                 assert(PyUnicode_KIND(*callresult) <= PyUnicode_KIND(string));
-                PyUnicode_CopyCharacters((PyObject*)string, i,
-                                         *callresult, 0,
-                                         PyUnicode_GET_LENGTH(*callresult));
+                if (PyUnicode_CopyCharacters((PyObject*)string, i,
+                                             *callresult, 0,
+                                             PyUnicode_GET_LENGTH(*callresult)) < 0)
+                    goto fail;
                 i += PyUnicode_GET_LENGTH(*callresult);
                 /* We're done with the unicode()/repr() => forget it */
                 Py_DECREF(*callresult);
@@ -8141,8 +8146,7 @@
     else {
         /* In case the maximum character changed, we need to
            convert the string to the new category. */
-        PyObject *v = PyUnicode_New(
-                             PyUnicode_GET_LENGTH(self), maxchar_new);
+        PyObject *v = PyUnicode_New(PyUnicode_GET_LENGTH(self), maxchar_new);
         if (v == NULL) {
             Py_DECREF(u);
             return NULL;
@@ -8151,12 +8155,25 @@
             /* If the maxchar increased so that the kind changed, not all
                characters are representable anymore and we need to fix the
                string again. This only happens in very few cases. */
-            PyUnicode_CopyCharacters(v, 0, (PyObject*)self, 0, PyUnicode_GET_LENGTH(self));
+            if (PyUnicode_CopyCharacters(v, 0, 
+                                         (PyObject*)self, 0, 
+                                         PyUnicode_GET_LENGTH(self)) < 0)
+            {
+                Py_DECREF(u);
+                return NULL;
+            }
             maxchar_old = fixfct((PyUnicodeObject*)v);
             assert(maxchar_old > 0 && maxchar_old <= maxchar_new);
         }
-        else
-            PyUnicode_CopyCharacters(v, 0, u, 0, PyUnicode_GET_LENGTH(self));
+        else {
+            if (PyUnicode_CopyCharacters(v, 0, 
+                                         u, 0, 
+                                         PyUnicode_GET_LENGTH(self)) < 0)
+            {
+                Py_DECREF(u);
+                return NULL;
+            }
+        }
 
         Py_DECREF(u);
         return v;
@@ -8455,12 +8472,14 @@
         itemlen = PyUnicode_GET_LENGTH(item);
         /* Copy item, and maybe the separator. */
         if (i) {
-            PyUnicode_CopyCharacters(res, res_offset,
-                                     sep, 0, seplen);
+            if (PyUnicode_CopyCharacters(res, res_offset,
+                                         sep, 0, seplen) < 0)
+                goto onError;
             res_offset += seplen;
         }
-        PyUnicode_CopyCharacters(res, res_offset,
-                                 item, 0, itemlen);
+        if (PyUnicode_CopyCharacters(res, res_offset,
+                                     item, 0, itemlen) < 0)
+            goto onError;
         res_offset += itemlen;
     }
     assert(res_offset == PyUnicode_GET_LENGTH(res));
@@ -8508,6 +8527,8 @@
 {
     PyObject *u;
     Py_UCS4 maxchar;
+    int kind;
+    void *data;
 
     if (left < 0)
         left = 0;
@@ -8528,14 +8549,21 @@
     if (fill > maxchar)
         maxchar = fill;
     u = PyUnicode_New(left + _PyUnicode_LENGTH(self) + right, maxchar);
-    if (u) {
-        int kind = PyUnicode_KIND(u);
-        void *data = PyUnicode_DATA(u);
-        if (left)
-            FILL(kind, data, fill, 0, left);
-        if (right)
-            FILL(kind, data, fill, left + _PyUnicode_LENGTH(self), right);
-        PyUnicode_CopyCharacters(u, left, (PyObject*)self, 0, _PyUnicode_LENGTH(self));
+    if (!u)
+        return NULL;
+
+    kind = PyUnicode_KIND(u);
+    data = PyUnicode_DATA(u);
+    if (left)
+        FILL(kind, data, fill, 0, left);
+    if (right)
+        FILL(kind, data, fill, left + _PyUnicode_LENGTH(self), right);
+    if (PyUnicode_CopyCharacters(u, left, 
+                                 (PyObject*)self, 0, 
+                                 _PyUnicode_LENGTH(self)) < 0)
+    {
+        Py_DECREF(u);
+        return NULL;
     }
 
     return (PyUnicodeObject*)u;
@@ -8821,8 +8849,12 @@
             u = PyUnicode_New(slen, maxchar);
             if (!u)
                 goto error;
-            PyUnicode_CopyCharacters(u, 0,
-                                     (PyObject*)self, 0, slen);
+            if (PyUnicode_CopyCharacters(u, 0,
+                                         (PyObject*)self, 0, slen) < 0)
+            {
+                Py_DECREF(u);
+                return NULL;
+            }
             rkind = PyUnicode_KIND(u);
             for (i = 0; i < PyUnicode_GET_LENGTH(u); i++)
                 if (PyUnicode_READ(rkind, PyUnicode_DATA(u), i) == u1) {
@@ -9437,8 +9469,7 @@
         goto onError;
 
     maxchar = PyUnicode_MAX_CHAR_VALUE(u);
-    if (PyUnicode_MAX_CHAR_VALUE(v) > maxchar)
-        maxchar = PyUnicode_MAX_CHAR_VALUE(v);
+    maxchar = PY_MAX(maxchar, PyUnicode_MAX_CHAR_VALUE(v));
 
     /* Concat the two Unicode strings */
     w = PyUnicode_New(
@@ -9446,9 +9477,12 @@
         maxchar);
     if (w == NULL)
         goto onError;
-    PyUnicode_CopyCharacters(w, 0, u, 0, PyUnicode_GET_LENGTH(u));
-    PyUnicode_CopyCharacters(w, PyUnicode_GET_LENGTH(u), v, 0,
-                             PyUnicode_GET_LENGTH(v));
+    if (PyUnicode_CopyCharacters(w, 0, u, 0, PyUnicode_GET_LENGTH(u)) < 0)
+        goto onError;
+    if (PyUnicode_CopyCharacters(w, PyUnicode_GET_LENGTH(u), 
+                                 v, 0,
+                                 PyUnicode_GET_LENGTH(v)) < 0)
+        goto onError;
     Py_DECREF(u);
     Py_DECREF(v);
     return w;
@@ -10396,8 +10430,12 @@
     unicode = PyUnicode_New(len, maxchar);
     if (unicode == NULL)
         return NULL;
-    PyUnicode_CopyCharacters(unicode, 0,
-                             (PyObject*)self, start, len);
+    if (PyUnicode_CopyCharacters(unicode, 0,
+                                 (PyObject*)self, start, len) < 0)
+    {
+        Py_DECREF(unicode);
+        return NULL;
+    }
     return unicode;
 }
 
diff --git a/Python/compile.c b/Python/compile.c
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -245,8 +245,14 @@
         return 0;
     /* ident = "_" + priv[ipriv:] + ident # i.e. 1+plen+nlen bytes */
     PyUnicode_WRITE(PyUnicode_KIND(result), PyUnicode_DATA(result), 0, '_');
-    PyUnicode_CopyCharacters(result, 1, privateobj, ipriv, plen);
-    PyUnicode_CopyCharacters(result, plen+1, ident, 0, nlen);
+    if (PyUnicode_CopyCharacters(result, 1, privateobj, ipriv, plen) < 0) {
+        Py_DECREF(result);
+        return NULL;
+    }
+    if (PyUnicode_CopyCharacters(result, plen+1, ident, 0, nlen) < 0) {
+        Py_DECREF(result);
+        return NULL;
+    }
     return result;
 }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 22:08:37 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 28 Sep 2011 22:08:37 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Strip_trailing_spaces_in_un?=
	=?utf8?q?icodeobject=2E=5Bch=5D?=
Message-ID: <E1R90Qn-000540-Vf@dinsdale.python.org>

http://hg.python.org/cpython/rev/93cc591cffe4
changeset:   72494:93cc591cffe4
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 28 21:41:31 2011 +0200
summary:
  Strip trailing spaces in unicodeobject.[ch]

files:
  Include/unicodeobject.h |  10 +++++-----
  Objects/unicodeobject.c |  20 ++++++++++----------
  2 files changed, 15 insertions(+), 15 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -356,13 +356,13 @@
 #define PyUnicode_IS_COMPACT(op) \
     (((PyASCIIObject*)(op))->state.compact)
 
-/* Return one of the PyUnicode_*_KIND values defined above. */ 
+/* Return one of the PyUnicode_*_KIND values defined above. */
 #define PyUnicode_KIND(op) \
     (assert(PyUnicode_Check(op)), \
      assert(PyUnicode_IS_READY(op)),            \
      ((PyASCIIObject *)(op))->state.kind)
 
-/* Return a void pointer to the raw unicode buffer. */ 
+/* Return a void pointer to the raw unicode buffer. */
 #define _PyUnicode_COMPACT_DATA(op)                     \
     (PyUnicode_IS_COMPACT_ASCII(op) ?                   \
      ((void*)((PyASCIIObject*)(op) + 1)) :              \
@@ -509,7 +509,7 @@
 
 /* Initializes the canonical string representation from a the deprected
    wstr/Py_UNICODE representation.  This function is used to convert
-   unicode objects which were created using the old API to the new flexible 
+   unicode objects which were created using the old API to the new flexible
    format introduced with PEP 393.  The PyUnicode_READY() macro can be
    more efficient if the string is already ready. */
 #ifndef Py_LIMITED_API
@@ -641,7 +641,7 @@
     PyObject *unicode
 );
 
-/* Get the number of Py_UNICODE units in the 
+/* Get the number of Py_UNICODE units in the
    string representation. */
 
 PyAPI_FUNC(Py_ssize_t) PyUnicode_GetSize(
@@ -857,7 +857,7 @@
 
    In case of an error, no *size is set.
 
-   This funcation caches the UTF-8 encoded string in the unicodeobject 
+   This funcation caches the UTF-8 encoded string in the unicodeobject
    and subsequent calls will return the same string.  The memory is relased
    when the unicodeobject is deallocated.
 
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -643,7 +643,7 @@
                   PyUnicode_KIND_SIZE(to_kind, how_many));
         return how_many;
     }
-    
+
     if (from_kind > to_kind) {
         /* slow path to check for character overflow */
         const Py_UCS4 to_maxchar = PyUnicode_MAX_CHAR_VALUE(to);
@@ -678,7 +678,7 @@
             );
         return how_many;
     }
-    else if (from_kind == PyUnicode_1BYTE_KIND 
+    else if (from_kind == PyUnicode_1BYTE_KIND
              && to_kind == PyUnicode_4BYTE_KIND)
     {
         _PyUnicode_CONVERT_BYTES(
@@ -703,7 +703,7 @@
     PyErr_Format(PyExc_ValueError,
                  "Cannot copy UCS%u characters "
                  "into a string of UCS%u characters",
-                 1 << (from_kind - 1), 
+                 1 << (from_kind - 1),
                  1 << (to_kind -1));
     return -1;
 }
@@ -8155,8 +8155,8 @@
             /* If the maxchar increased so that the kind changed, not all
                characters are representable anymore and we need to fix the
                string again. This only happens in very few cases. */
-            if (PyUnicode_CopyCharacters(v, 0, 
-                                         (PyObject*)self, 0, 
+            if (PyUnicode_CopyCharacters(v, 0,
+                                         (PyObject*)self, 0,
                                          PyUnicode_GET_LENGTH(self)) < 0)
             {
                 Py_DECREF(u);
@@ -8166,8 +8166,8 @@
             assert(maxchar_old > 0 && maxchar_old <= maxchar_new);
         }
         else {
-            if (PyUnicode_CopyCharacters(v, 0, 
-                                         u, 0, 
+            if (PyUnicode_CopyCharacters(v, 0,
+                                         u, 0,
                                          PyUnicode_GET_LENGTH(self)) < 0)
             {
                 Py_DECREF(u);
@@ -8558,8 +8558,8 @@
         FILL(kind, data, fill, 0, left);
     if (right)
         FILL(kind, data, fill, left + _PyUnicode_LENGTH(self), right);
-    if (PyUnicode_CopyCharacters(u, left, 
-                                 (PyObject*)self, 0, 
+    if (PyUnicode_CopyCharacters(u, left,
+                                 (PyObject*)self, 0,
                                  _PyUnicode_LENGTH(self)) < 0)
     {
         Py_DECREF(u);
@@ -9479,7 +9479,7 @@
         goto onError;
     if (PyUnicode_CopyCharacters(w, 0, u, 0, PyUnicode_GET_LENGTH(u)) < 0)
         goto onError;
-    if (PyUnicode_CopyCharacters(w, PyUnicode_GET_LENGTH(u), 
+    if (PyUnicode_CopyCharacters(w, PyUnicode_GET_LENGTH(u),
                                  v, 0,
                                  PyUnicode_GET_LENGTH(v)) < 0)
         goto onError;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 22:08:38 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 28 Sep 2011 22:08:38 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_fill=5Fchar=28=29_can_now_p?=
	=?utf8?q?ropagate_an_error?=
Message-ID: <E1R90Qo-000544-Nb@dinsdale.python.org>

http://hg.python.org/cpython/rev/cf3557b65ae1
changeset:   72495:cf3557b65ae1
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 28 21:50:16 2011 +0200
summary:
  fill_char() can now propagate an error

files:
  Python/formatter_unicode.c |  72 ++++++++++++++++---------
  1 files changed, 47 insertions(+), 25 deletions(-)


diff --git a/Python/formatter_unicode.c b/Python/formatter_unicode.c
--- a/Python/formatter_unicode.c
+++ b/Python/formatter_unicode.c
@@ -341,7 +341,7 @@
 
     /* Pad on right. */
     if (n_rpadding)
-        unicode_fill(s, start + nchars + n_lpadding, 
+        unicode_fill(s, start + nchars + n_lpadding,
                      start + nchars + n_lpadding + n_rpadding, fill_char);
 
     /* Pointer to the user content. */
@@ -501,7 +501,7 @@
         spec->n_grouped_digits = 0;
     else
         spec->n_grouped_digits = _PyUnicode_InsertThousandsGrouping(
-            PyUnicode_1BYTE_KIND, NULL, 0, NULL, 
+            PyUnicode_1BYTE_KIND, NULL, 0, NULL,
             spec->n_digits, spec->n_min_width,
             locale->grouping, locale->thousands_sep);
 
@@ -541,11 +541,12 @@
 
 /* Fill in the digit parts of a numbers's string representation,
    as determined in calc_number_widths().
-   No error checking, since we know the buffer is the correct size. */
-static void
+   Return -1 on error, or 0 on success. */
+static int
 fill_number(PyObject *out, Py_ssize_t pos, const NumberFieldWidths *spec,
             PyObject *digits, Py_ssize_t d_start, Py_ssize_t d_end,
-            PyObject *prefix, Py_ssize_t p_start, Py_UCS4 fill_char,
+            PyObject *prefix, Py_ssize_t p_start,
+            Py_UCS4 fill_char,
             LocaleInfo *locale, int toupper)
 {
     /* Used to keep track of digits, decimal, and remainder. */
@@ -589,11 +590,8 @@
         char *pdigits = PyUnicode_DATA(digits);
         if (PyUnicode_KIND(digits) < kind) {
             pdigits = _PyUnicode_AsKind(digits, kind);
-            if (pdigits == NULL) {
-                /* XXX report exception */
-                Py_FatalError("out of memory");
-                return;
-            }
+            if (pdigits == NULL)
+                return -1;
         }
 #ifndef NDEBUG
         r =
@@ -640,6 +638,7 @@
         unicode_fill(out, pos, pos + spec->n_rpadding, fill_char);
         pos += spec->n_rpadding;
     }
+    return 0;
 }
 
 static char no_grouping[1] = {CHAR_MAX};
@@ -765,6 +764,7 @@
     Py_ssize_t prefix;
     NumberFieldWidths spec;
     long x;
+    int err;
 
     /* Locale settings, either from the actual locale or
        from a hard-code pseudo-locale */
@@ -886,10 +886,13 @@
         goto done;
 
     /* Populate the memory. */
-    fill_number(result, 0, &spec, tmp, inumeric_chars, inumeric_chars + n_digits,
-                tmp, prefix,
-                format->fill_char == '\0' ? ' ' : format->fill_char,
-                &locale, format->type == 'X');
+    err = fill_number(result, 0, &spec,
+                      tmp, inumeric_chars, inumeric_chars + n_digits,
+                      tmp, prefix,
+                      format->fill_char == '\0' ? ' ' : format->fill_char,
+                      &locale, format->type == 'X');
+    if (err)
+        Py_CLEAR(result);
 
 done:
     Py_XDECREF(tmp);
@@ -929,6 +932,7 @@
     Py_UCS4 sign_char = '\0';
     int float_type; /* Used to see if we have a nan, inf, or regular float. */
     PyObject *unicode_tmp = NULL;
+    int err;
 
     /* Locale settings, either from the actual locale or
        from a hard-code pseudo-locale */
@@ -1010,7 +1014,7 @@
                     &locale);
 
     /* Calculate how much memory we'll need. */
-    n_total = calc_number_widths(&spec, 0, sign_char, unicode_tmp, index, 
+    n_total = calc_number_widths(&spec, 0, sign_char, unicode_tmp, index,
                                  index + n_digits, n_remainder, has_decimal,
                                  &locale, format);
 
@@ -1020,10 +1024,13 @@
         goto done;
 
     /* Populate the memory. */
-    fill_number(result, 0, &spec, unicode_tmp, index, index + n_digits,
-                NULL, 0,
-                format->fill_char == '\0' ? ' ' : format->fill_char, &locale,
-                0);
+    err = fill_number(result, 0, &spec,
+                      unicode_tmp, index, index + n_digits,
+                      NULL, 0,
+                      format->fill_char == '\0' ? ' ' : format->fill_char,
+                      &locale, 0);
+    if (err)
+        Py_CLEAR(result);
 
 done:
     PyMem_Free(buf);
@@ -1077,6 +1084,7 @@
     Py_ssize_t total;
     PyObject *re_unicode_tmp = NULL;
     PyObject *im_unicode_tmp = NULL;
+    int err;
 
     /* Locale settings, either from the actual locale or
        from a hard-code pseudo-locale */
@@ -1170,9 +1178,9 @@
 
     /* Determine if we have any "remainder" (after the digits, might include
        decimal or exponent or both (or neither)) */
-    parse_number(re_unicode_tmp, i_re, i_re + n_re_digits, 
+    parse_number(re_unicode_tmp, i_re, i_re + n_re_digits,
                  &n_re_remainder, &re_has_decimal);
-    parse_number(im_unicode_tmp, i_im, i_im + n_im_digits, 
+    parse_number(im_unicode_tmp, i_im, i_im + n_im_digits,
                  &n_im_remainder, &im_has_decimal);
 
     /* Determine the grouping, separator, and decimal point, if any. */
@@ -1225,12 +1233,26 @@
         PyUnicode_WRITE(rkind, rdata, index++, '(');
 
     if (!skip_re) {
-        fill_number(result, index, &re_spec, re_unicode_tmp, 
-                    i_re, i_re + n_re_digits, NULL, 0, 0, &locale, 0);
+        err = fill_number(result, index, &re_spec,
+                          re_unicode_tmp, i_re, i_re + n_re_digits,
+                          NULL, 0,
+                          0,
+                          &locale, 0);
+        if (err) {
+            Py_CLEAR(result);
+            goto done;
+        }
         index += n_re_total;
     }
-    fill_number(result, index, &im_spec, im_unicode_tmp,
-                i_im, i_im + n_im_digits, NULL, 0, 0, &locale, 0);
+    err = fill_number(result, index, &im_spec,
+                      im_unicode_tmp, i_im, i_im + n_im_digits,
+                      NULL, 0,
+                      0,
+                      &locale, 0);
+    if (err) {
+        Py_CLEAR(result);
+        goto done;
+    }
     index += n_im_total;
     PyUnicode_WRITE(rkind, rdata, index++, 'j');
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 22:08:39 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 28 Sep 2011 22:08:39 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_fill=5Fnumber=28=29_ensures?=
 =?utf8?q?_that_the_=27digits=27_string_is_ready?=
Message-ID: <E1R90Qp-000548-FZ@dinsdale.python.org>

http://hg.python.org/cpython/rev/ad337be523bf
changeset:   72496:ad337be523bf
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 28 21:50:42 2011 +0200
summary:
  fill_number() ensures that the 'digits' string is ready

files:
  Python/formatter_unicode.c |  5 ++++-
  1 files changed, 4 insertions(+), 1 deletions(-)


diff --git a/Python/formatter_unicode.c b/Python/formatter_unicode.c
--- a/Python/formatter_unicode.c
+++ b/Python/formatter_unicode.c
@@ -587,7 +587,10 @@
     /* Only for type 'c' special case, it has no digits. */
     if (spec->n_digits != 0) {
         /* Fill the digits with InsertThousandsGrouping. */
-        char *pdigits = PyUnicode_DATA(digits);
+        char *pdigits;
+        if (PyUnicode_READY(digits))
+            return -1;
+        pdigits = PyUnicode_DATA(digits);
         if (PyUnicode_KIND(digits) < kind) {
             pdigits = _PyUnicode_AsKind(digits, kind);
             if (pdigits == NULL)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 22:08:40 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 28 Sep 2011 22:08:40 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_fill=5Fnumber=28=29_and_for?=
 =?utf8?q?mat=5Fstring=5Finternal=28=29_check_for_PyUnicode=5FCopyCharacte?=
 =?utf8?q?rs=28=29?=
Message-ID: <E1R90Qq-00054C-9g@dinsdale.python.org>

http://hg.python.org/cpython/rev/f05c2df39ea8
changeset:   72497:f05c2df39ea8
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 28 21:53:49 2011 +0200
summary:
  fill_number() and format_string_internal() check for PyUnicode_CopyCharacters() failure

files:
  Python/formatter_unicode.c |  11 ++++++++---
  1 files changed, 8 insertions(+), 3 deletions(-)


diff --git a/Python/formatter_unicode.c b/Python/formatter_unicode.c
--- a/Python/formatter_unicode.c
+++ b/Python/formatter_unicode.c
@@ -566,7 +566,10 @@
         PyUnicode_WRITE(kind, data, pos++, spec->sign);
     }
     if (spec->n_prefix) {
-        PyUnicode_CopyCharacters(out, pos, prefix, p_start, spec->n_prefix);
+        if (PyUnicode_CopyCharacters(out, pos,
+                                     prefix, p_start,
+                                     spec->n_prefix) < 0)
+            return -1;
         if (toupper) {
             Py_ssize_t t;
             /* XXX if the upper-case prefix is wider than the target
@@ -632,7 +635,8 @@
     }
 
     if (spec->n_remainder) {
-        PyUnicode_CopyCharacters(out, pos, digits, d_pos, spec->n_remainder);
+        if (PyUnicode_CopyCharacters(out, pos, digits, d_pos, spec->n_remainder) < 0)
+            return -1;
         pos += spec->n_remainder;
         d_pos += spec->n_remainder;
     }
@@ -735,7 +739,8 @@
                        lpad, rpad);
 
     /* Then the source string. */
-    PyUnicode_CopyCharacters(result, pos, value, 0, len);
+    if (PyUnicode_CopyCharacters(result, pos, value, 0, len) < 0)
+        Py_CLEAR(result);
 
 done:
     return result;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 22:17:16 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 28 Sep 2011 22:17:16 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Mark_=5FPyUnicode=5FFindMax?=
 =?utf8?q?CharAndNumSurrogatePairs=28=29_as_private?=
Message-ID: <E1R90ZA-00064w-BG@dinsdale.python.org>

http://hg.python.org/cpython/rev/e395f1436fe6
changeset:   72498:e395f1436fe6
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 28 22:15:37 2011 +0200
summary:
  Mark _PyUnicode_FindMaxCharAndNumSurrogatePairs() as private

files:
  Include/unicodeobject.h |  12 ------------
  Objects/unicodeobject.c |  17 ++++++++++-------
  2 files changed, 10 insertions(+), 19 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -543,18 +543,6 @@
     );
 #endif
 
-/* Find the maximum code point and count the number of surrogate pairs so a
-   correct string length can be computed before converting a string to UCS4.
-   This function counts single surrogates as a character and not as a pair. */
-#ifndef Py_LIMITED_API
-PyAPI_FUNC(int) _PyUnicode_FindMaxCharAndNumSurrogatePairs(
-    const wchar_t *begin,
-    const wchar_t *end,
-    Py_UCS4 *maxchar,
-    Py_ssize_t *num_surrogates
-    );
-#endif
-
 /* Create a Unicode Object from the Py_UNICODE buffer u of the given
    size.
 
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -708,11 +708,14 @@
     return -1;
 }
 
-int
-_PyUnicode_FindMaxCharAndNumSurrogatePairs(const wchar_t *begin,
-                                           const wchar_t *end,
-                                           Py_UCS4 *maxchar,
-                                           Py_ssize_t *num_surrogates)
+/* Find the maximum code point and count the number of surrogate pairs so a
+   correct string length can be computed before converting a string to UCS4.
+   This function counts single surrogates as a character and not as a pair.
+
+   Return 0 on success, or -1 on error. */
+static int
+find_maxchar_surrogates(const wchar_t *begin, const wchar_t *end,
+                        Py_UCS4 *maxchar, Py_ssize_t *num_surrogates)
 {
     const wchar_t *iter;
 
@@ -789,7 +792,7 @@
 #endif
 
     end = _PyUnicode_WSTR(unicode) + _PyUnicode_WSTR_LENGTH(unicode);
-    if (_PyUnicode_FindMaxCharAndNumSurrogatePairs(_PyUnicode_WSTR(unicode), end,
+    if (find_maxchar_surrogates(_PyUnicode_WSTR(unicode), end,
                                                   &maxchar,
                                                   &num_surrogates) == -1) {
         assert(0 && "PyUnicode_FindMaxCharAndNumSurrogatePairs failed");
@@ -1022,7 +1025,7 @@
 
     /* If not empty and not single character, copy the Unicode data
        into the new object */
-    if (_PyUnicode_FindMaxCharAndNumSurrogatePairs(u, u + size, &maxchar,
+    if (find_maxchar_surrogates(u, u + size, &maxchar,
                                                   &num_surrogates) == -1)
         return NULL;
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 22:17:17 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 28 Sep 2011 22:17:17 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Oops=2C_fix_Py=5FMIN/Py=5FM?=
	=?utf8?q?AX_case?=
Message-ID: <E1R90ZB-000656-1E@dinsdale.python.org>

http://hg.python.org/cpython/rev/9d06a9408ca3
changeset:   72499:9d06a9408ca3
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 28 22:17:19 2011 +0200
summary:
  Oops, fix Py_MIN/Py_MAX case

files:
  Objects/unicodeobject.c |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -622,7 +622,7 @@
     if (PyUnicode_READY(to))
         return -1;
 
-    how_many = PY_MIN(PyUnicode_GET_LENGTH(from), how_many);
+    how_many = Py_MIN(PyUnicode_GET_LENGTH(from), how_many);
     if (to_start + how_many > PyUnicode_GET_LENGTH(to)) {
         PyErr_Format(PyExc_ValueError,
                      "Cannot write %zi characters at %zi "
@@ -9472,7 +9472,7 @@
         goto onError;
 
     maxchar = PyUnicode_MAX_CHAR_VALUE(u);
-    maxchar = PY_MAX(maxchar, PyUnicode_MAX_CHAR_VALUE(v));
+    maxchar = Py_MAX(maxchar, PyUnicode_MAX_CHAR_VALUE(v));
 
     /* Concat the two Unicode strings */
     w = PyUnicode_New(

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 22:20:38 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 28 Sep 2011 22:20:38 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Mark_PyUnicode=5FFromUCS=5B?=
	=?utf8?q?124=5D_as_private?=
Message-ID: <E1R90cQ-0006PS-3Y@dinsdale.python.org>

http://hg.python.org/cpython/rev/347fb9006c6f
changeset:   72500:347fb9006c6f
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 28 22:20:48 2011 +0200
summary:
  Mark PyUnicode_FromUCS[124] as private

files:
  Objects/stringlib/ucs1lib.h |   2 +-
  Objects/stringlib/ucs2lib.h |   2 +-
  Objects/stringlib/ucs4lib.h |   2 +-
  Objects/unicodeobject.c     |  20 ++++++++++----------
  4 files changed, 13 insertions(+), 13 deletions(-)


diff --git a/Objects/stringlib/ucs1lib.h b/Objects/stringlib/ucs1lib.h
--- a/Objects/stringlib/ucs1lib.h
+++ b/Objects/stringlib/ucs1lib.h
@@ -19,7 +19,7 @@
 #define STRINGLIB_FILL           Py_UNICODE_FILL
 #define STRINGLIB_STR            PyUnicode_1BYTE_DATA
 #define STRINGLIB_LEN            PyUnicode_GET_LENGTH
-#define STRINGLIB_NEW            PyUnicode_FromUCS1
+#define STRINGLIB_NEW            _PyUnicode_FromUCS1
 #define STRINGLIB_RESIZE         not_supported
 #define STRINGLIB_CHECK          PyUnicode_Check
 #define STRINGLIB_CHECK_EXACT    PyUnicode_CheckExact
diff --git a/Objects/stringlib/ucs2lib.h b/Objects/stringlib/ucs2lib.h
--- a/Objects/stringlib/ucs2lib.h
+++ b/Objects/stringlib/ucs2lib.h
@@ -19,7 +19,7 @@
 #define STRINGLIB_FILL           Py_UNICODE_FILL
 #define STRINGLIB_STR            PyUnicode_1BYTE_DATA
 #define STRINGLIB_LEN            PyUnicode_GET_LENGTH
-#define STRINGLIB_NEW            PyUnicode_FromUCS2
+#define STRINGLIB_NEW            _PyUnicode_FromUCS2
 #define STRINGLIB_RESIZE         not_supported
 #define STRINGLIB_CHECK          PyUnicode_Check
 #define STRINGLIB_CHECK_EXACT    PyUnicode_CheckExact
diff --git a/Objects/stringlib/ucs4lib.h b/Objects/stringlib/ucs4lib.h
--- a/Objects/stringlib/ucs4lib.h
+++ b/Objects/stringlib/ucs4lib.h
@@ -19,7 +19,7 @@
 #define STRINGLIB_FILL           Py_UNICODE_FILL
 #define STRINGLIB_STR            PyUnicode_1BYTE_DATA
 #define STRINGLIB_LEN            PyUnicode_GET_LENGTH
-#define STRINGLIB_NEW            PyUnicode_FromUCS4
+#define STRINGLIB_NEW            _PyUnicode_FromUCS4
 #define STRINGLIB_RESIZE         not_supported
 #define STRINGLIB_CHECK          PyUnicode_Check
 #define STRINGLIB_CHECK_EXACT    PyUnicode_CheckExact
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -1117,8 +1117,8 @@
     return PyUnicode_FromStringAndSize(u, size);
 }
 
-PyObject*
-PyUnicode_FromUCS1(const unsigned char* u, Py_ssize_t size)
+static PyObject*
+_PyUnicode_FromUCS1(const unsigned char* u, Py_ssize_t size)
 {
     PyObject *res;
     unsigned char max = 127;
@@ -1136,8 +1136,8 @@
     return res;
 }
 
-PyObject*
-PyUnicode_FromUCS2(const Py_UCS2 *u, Py_ssize_t size)
+static PyObject*
+_PyUnicode_FromUCS2(const Py_UCS2 *u, Py_ssize_t size)
 {
     PyObject *res;
     Py_UCS2 max = 0;
@@ -1156,8 +1156,8 @@
     return res;
 }
 
-PyObject*
-PyUnicode_FromUCS4(const Py_UCS4 *u, Py_ssize_t size)
+static PyObject*
+_PyUnicode_FromUCS4(const Py_UCS4 *u, Py_ssize_t size)
 {
     PyObject *res;
     Py_UCS4 max = 0;
@@ -1184,11 +1184,11 @@
 {
     switch(kind) {
     case PyUnicode_1BYTE_KIND:
-        return PyUnicode_FromUCS1(buffer, size);
+        return _PyUnicode_FromUCS1(buffer, size);
     case PyUnicode_2BYTE_KIND:
-        return PyUnicode_FromUCS2(buffer, size);
+        return _PyUnicode_FromUCS2(buffer, size);
     case PyUnicode_4BYTE_KIND:
-        return PyUnicode_FromUCS4(buffer, size);
+        return _PyUnicode_FromUCS4(buffer, size);
     }
     assert(0);
     return NULL;
@@ -5645,7 +5645,7 @@
 		       const char *errors)
 {
     /* Latin-1 is equivalent to the first 256 ordinals in Unicode. */
-    return PyUnicode_FromUCS1((unsigned char*)s, size);
+    return _PyUnicode_FromUCS1((unsigned char*)s, size);
 }
 
 /* create or adjust a UnicodeEncodeError */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 22:27:53 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 28 Sep 2011 22:27:53 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_PyUnicode=5FCopyCharacters?=
	=?utf8?q?=28=29_initializes_overflow?=
Message-ID: <E1R90jR-00073e-E2@dinsdale.python.org>

http://hg.python.org/cpython/rev/fb70b59fa8f9
changeset:   72501:fb70b59fa8f9
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 28 22:28:04 2011 +0200
summary:
  PyUnicode_CopyCharacters() initializes overflow

files:
  Objects/unicodeobject.c |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -654,6 +654,7 @@
         int overflow;
 
         maxchar = 0;
+        overflow = 0;
         for (i=0; i < how_many; i++) {
             ch = PyUnicode_READ(from_kind, from_data, from_start + i);
             if (ch > maxchar) {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 22:34:21 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 28 Sep 2011 22:34:21 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Check_size_of_wchar=5Ft_usi?=
	=?utf8?q?ng_the_preprocessor?=
Message-ID: <E1R90ph-0007l1-LQ@dinsdale.python.org>

http://hg.python.org/cpython/rev/b021f1b03f96
changeset:   72502:b021f1b03f96
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 28 22:34:18 2011 +0200
summary:
  Check size of wchar_t using the preprocessor

files:
  Objects/unicodeobject.c |  58 ++++++++++++++---------------
  1 files changed, 28 insertions(+), 30 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -830,36 +830,34 @@
         assert(num_surrogates == 0 &&
                "FindMaxCharAndNumSurrogatePairs() messed up");
 
-        if (sizeof(wchar_t) == 2) {
-            /* We can share representations and are done. */
-            unicode->data.any = _PyUnicode_WSTR(unicode);
-            PyUnicode_2BYTE_DATA(unicode)[_PyUnicode_WSTR_LENGTH(unicode)] = '\0';
-            _PyUnicode_LENGTH(unicode) = _PyUnicode_WSTR_LENGTH(unicode);
-            _PyUnicode_STATE(unicode).kind = PyUnicode_2BYTE_KIND;
-            unicode->_base.utf8 = NULL;
-            unicode->_base.utf8_length = 0;
-        }
-        else {
-            assert(sizeof(wchar_t) == 4);
-
-            unicode->data.any = PyObject_MALLOC(
-                2 * (_PyUnicode_WSTR_LENGTH(unicode) + 1));
-            if (!unicode->data.any) {
-                PyErr_NoMemory();
-                return -1;
-            }
-            _PyUnicode_CONVERT_BYTES(wchar_t, Py_UCS2,
-                                    _PyUnicode_WSTR(unicode), end,
-                                    PyUnicode_2BYTE_DATA(unicode));
-            PyUnicode_2BYTE_DATA(unicode)[_PyUnicode_WSTR_LENGTH(unicode)] = '\0';
-            _PyUnicode_LENGTH(unicode) = _PyUnicode_WSTR_LENGTH(unicode);
-            _PyUnicode_STATE(unicode).kind = PyUnicode_2BYTE_KIND;
-            unicode->_base.utf8 = NULL;
-            unicode->_base.utf8_length = 0;
-            PyObject_FREE(_PyUnicode_WSTR(unicode));
-            _PyUnicode_WSTR(unicode) = NULL;
-            _PyUnicode_WSTR_LENGTH(unicode) = 0;
-        }
+#if SIZEOF_WCHAR_T == 2
+        /* We can share representations and are done. */
+        unicode->data.any = _PyUnicode_WSTR(unicode);
+        PyUnicode_2BYTE_DATA(unicode)[_PyUnicode_WSTR_LENGTH(unicode)] = '\0';
+        _PyUnicode_LENGTH(unicode) = _PyUnicode_WSTR_LENGTH(unicode);
+        _PyUnicode_STATE(unicode).kind = PyUnicode_2BYTE_KIND;
+        unicode->_base.utf8 = NULL;
+        unicode->_base.utf8_length = 0;
+#else
+        /* sizeof(wchar_t) == 4 */
+        unicode->data.any = PyObject_MALLOC(
+            2 * (_PyUnicode_WSTR_LENGTH(unicode) + 1));
+        if (!unicode->data.any) {
+            PyErr_NoMemory();
+            return -1;
+        }
+        _PyUnicode_CONVERT_BYTES(wchar_t, Py_UCS2,
+                                _PyUnicode_WSTR(unicode), end,
+                                PyUnicode_2BYTE_DATA(unicode));
+        PyUnicode_2BYTE_DATA(unicode)[_PyUnicode_WSTR_LENGTH(unicode)] = '\0';
+        _PyUnicode_LENGTH(unicode) = _PyUnicode_WSTR_LENGTH(unicode);
+        _PyUnicode_STATE(unicode).kind = PyUnicode_2BYTE_KIND;
+        unicode->_base.utf8 = NULL;
+        unicode->_base.utf8_length = 0;
+        PyObject_FREE(_PyUnicode_WSTR(unicode));
+        _PyUnicode_WSTR(unicode) = NULL;
+        _PyUnicode_WSTR_LENGTH(unicode) = 0;
+#endif
     }
     /* maxchar exeeds 16 bit, wee need 4 bytes for unicode characters */
     else {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 23:18:33 2011
From: python-checkins at python.org (ezio.melotti)
Date: Wed, 28 Sep 2011 23:18:33 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_=2313054=3A_sys=2Emaxunicod?=
 =?utf8?q?e_is_now_always_0x10FFFF=2E?=
Message-ID: <E1R91WT-0003fG-0b@dinsdale.python.org>

http://hg.python.org/cpython/rev/606652491366
changeset:   72503:606652491366
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Sep 29 00:18:19 2011 +0300
summary:
  #13054: sys.maxunicode is now always 0x10FFFF.

files:
  Doc/library/sys.rst     |  10 +++++++---
  Doc/whatsnew/3.3.rst    |  11 +++++++++++
  Lib/test/test_sys.py    |   1 +
  Objects/unicodeobject.c |   3 ++-
  Python/sysmodule.c      |   4 ++--
  5 files changed, 23 insertions(+), 6 deletions(-)


diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -625,9 +625,13 @@
 
 .. data:: maxunicode
 
-   An integer giving the largest supported code point for a Unicode character.  The
-   value of this depends on the configuration option that specifies whether Unicode
-   characters are stored as UCS-2 or UCS-4.
+   An integer giving the value of the largest Unicode code point,
+   i.e. ``1114111`` (``0x10FFFF`` in hexadecimal).
+
+   .. versionchanged:: 3.3
+      Before :pep:`393`, :data:`sys.maxunicode` used to return either ``0xFFFF``
+      or ``0x10FFFF``, depending on the configuration option that specified
+      whether Unicode characters were stored as UCS-2 or UCS-4.
 
 
 .. data:: meta_path
diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -55,6 +55,17 @@
 =============
 
 
+PEP 393: Flexible String Representation
+=======================================
+
+XXX Add list of changes introduced by :pep:`393` here:
+
+* The value of :data:`sys.maxunicode` is now always ``1114111`` (``0x10FFFF``
+  in hexadecimal).  The :c:func:`PyUnicode_GetMax` function still returns
+  either ``0xFFFF`` or ``0x10FFFF`` for backward compatibility, and it should
+  not be used with the new Unicode API (see :issue:`13054`).
+
+
 Other Language Changes
 ======================
 
diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py
--- a/Lib/test/test_sys.py
+++ b/Lib/test/test_sys.py
@@ -447,6 +447,7 @@
 
         self.assertIsInstance(sys.maxsize, int)
         self.assertIsInstance(sys.maxunicode, int)
+        self.assertEqual(sys.maxunicode, 0x10FFFF)
         self.assertIsInstance(sys.platform, str)
         self.assertIsInstance(sys.prefix, str)
         self.assertIsInstance(sys.version, str)
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -207,7 +207,8 @@
     0, 0, 0, 0, 0, 0, 0, 0
 };
 
-
+/* The max unicode value is always 0x10FFFF while using the PEP-393 API.
+   This function is kept for backward compatibility with the old API. */
 Py_UNICODE
 PyUnicode_GetMax(void)
 {
diff --git a/Python/sysmodule.c b/Python/sysmodule.c
--- a/Python/sysmodule.c
+++ b/Python/sysmodule.c
@@ -1261,7 +1261,7 @@
 hexversion -- version information encoded as a single integer\n\
 int_info -- a struct sequence with information about the int implementation.\n\
 maxsize -- the largest supported length of containers.\n\
-maxunicode -- the largest supported character\n\
+maxunicode -- the value of the largest Unicode codepoint\n\
 platform -- platform identifier\n\
 prefix -- prefix used to find the Python library\n\
 thread_info -- a struct sequence with information about the thread implementation.\n\
@@ -1536,7 +1536,7 @@
     SET_SYS_FROM_STRING("hash_info",
                         get_hash_info());
     SET_SYS_FROM_STRING("maxunicode",
-                        PyLong_FromLong(PyUnicode_GetMax()));
+                        PyLong_FromLong(0x10FFFF));
     SET_SYS_FROM_STRING("builtin_module_names",
                         list_builtin_module_names());
     {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Sep 28 23:59:11 2011
From: python-checkins at python.org (ezio.melotti)
Date: Wed, 28 Sep 2011 23:59:11 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Clean_up_a_few_tabs_that_we?=
	=?utf8?q?nt_in_with_PEP393=2E?=
Message-ID: <E1R929n-0007aJ-9k@dinsdale.python.org>

http://hg.python.org/cpython/rev/950678c769e6
changeset:   72504:950678c769e6
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Sep 29 00:58:57 2011 +0300
summary:
  Clean up a few tabs that went in with PEP393.

files:
  Include/Python.h        |    2 +-
  Include/floatobject.h   |   18 +-
  Include/longobject.h    |   16 +-
  Modules/_sre.c          |  226 ++++++++++++++--------------
  Objects/typeobject.c    |   16 +-
  Objects/unicodeobject.c |  130 ++++++++--------
  6 files changed, 204 insertions(+), 204 deletions(-)


diff --git a/Include/Python.h b/Include/Python.h
--- a/Include/Python.h
+++ b/Include/Python.h
@@ -141,7 +141,7 @@
 #endif
 
 /* Argument must be a char or an int in [-128, 127] or [0, 255]. */
-#define Py_CHARMASK(c)		((unsigned char)((c) & 0xff))
+#define Py_CHARMASK(c) ((unsigned char)((c) & 0xff))
 
 #include "pyfpe.h"
 
diff --git a/Include/floatobject.h b/Include/floatobject.h
--- a/Include/floatobject.h
+++ b/Include/floatobject.h
@@ -27,12 +27,12 @@
 #define Py_RETURN_NAN return PyFloat_FromDouble(Py_NAN)
 #endif
 
-#define Py_RETURN_INF(sign) do					\
-	if (copysign(1., sign) == 1.) {				\
-		return PyFloat_FromDouble(Py_HUGE_VAL);	\
-	} else {						\
-		return PyFloat_FromDouble(-Py_HUGE_VAL);	\
-	} while(0)
+#define Py_RETURN_INF(sign) do                     \
+    if (copysign(1., sign) == 1.) {                \
+        return PyFloat_FromDouble(Py_HUGE_VAL);    \
+    } else {                        \
+        return PyFloat_FromDouble(-Py_HUGE_VAL);   \
+    } while(0)
 
 PyAPI_FUNC(double) PyFloat_GetMax(void);
 PyAPI_FUNC(double) PyFloat_GetMin(void);
@@ -113,9 +113,9 @@
 /* Format the object based on the format_spec, as defined in PEP 3101
    (Advanced String Formatting). */
 PyAPI_FUNC(PyObject *) _PyFloat_FormatAdvanced(PyObject *obj,
-					       PyObject *format_spec,
-					       Py_ssize_t start,
-					       Py_ssize_t end);
+                                               PyObject *format_spec,
+                                               Py_ssize_t start,
+                                               Py_ssize_t end);
 #endif /* Py_LIMITED_API */
 
 #ifdef __cplusplus
diff --git a/Include/longobject.h b/Include/longobject.h
--- a/Include/longobject.h
+++ b/Include/longobject.h
@@ -12,7 +12,7 @@
 PyAPI_DATA(PyTypeObject) PyLong_Type;
 
 #define PyLong_Check(op) \
-		PyType_FastSubclass(Py_TYPE(op), Py_TPFLAGS_LONG_SUBCLASS)
+        PyType_FastSubclass(Py_TYPE(op), Py_TPFLAGS_LONG_SUBCLASS)
 #define PyLong_CheckExact(op) (Py_TYPE(op) == &PyLong_Type)
 
 PyAPI_FUNC(PyObject *) PyLong_FromLong(long);
@@ -122,8 +122,8 @@
      enough memory to create the Python long.
 */
 PyAPI_FUNC(PyObject *) _PyLong_FromByteArray(
-	const unsigned char* bytes, size_t n,
-	int little_endian, int is_signed);
+    const unsigned char* bytes, size_t n,
+    int little_endian, int is_signed);
 
 /* _PyLong_AsByteArray: Convert the least-significant 8*n bits of long
    v to a base-256 integer, stored in array bytes.  Normally return 0,
@@ -145,8 +145,8 @@
      case, but bytes holds the least-signficant n bytes of the true value.
 */
 PyAPI_FUNC(int) _PyLong_AsByteArray(PyLongObject* v,
-	unsigned char* bytes, size_t n,
-	int little_endian, int is_signed);
+    unsigned char* bytes, size_t n,
+    int little_endian, int is_signed);
 
 
 /* _PyLong_Format: Convert the long to a string object with given base,
@@ -156,9 +156,9 @@
 /* Format the object based on the format_spec, as defined in PEP 3101
    (Advanced String Formatting). */
 PyAPI_FUNC(PyObject *) _PyLong_FormatAdvanced(PyObject *obj,
-					      PyObject *format_spec,
-					      Py_ssize_t start,
-					      Py_ssize_t end);
+                                              PyObject *format_spec,
+                                              Py_ssize_t start,
+                                              Py_ssize_t end);
 #endif /* Py_LIMITED_API */
 
 /* These aren't really part of the long object, but they're handy. The
diff --git a/Modules/_sre.c b/Modules/_sre.c
--- a/Modules/_sre.c
+++ b/Modules/_sre.c
@@ -1739,16 +1739,16 @@
     if (!ptr)
         return NULL;
 
-	if (logical_charsize == 1 && pattern->logical_charsize > 1) {
-		PyErr_SetString(PyExc_TypeError,
-			"can't use a string pattern on a bytes-like object");
-		return NULL;
-	}
-	if (logical_charsize > 1 && pattern->logical_charsize == 1) {
-		PyErr_SetString(PyExc_TypeError,
-			"can't use a bytes pattern on a string-like object");
-		return NULL;
-	}
+        if (logical_charsize == 1 && pattern->logical_charsize > 1) {
+                PyErr_SetString(PyExc_TypeError,
+                        "can't use a string pattern on a bytes-like object");
+                return NULL;
+        }
+        if (logical_charsize > 1 && pattern->logical_charsize == 1) {
+                PyErr_SetString(PyExc_TypeError,
+                        "can't use a bytes pattern on a string-like object");
+                return NULL;
+        }
 
     /* adjust boundaries */
     if (start < 0)
@@ -2062,8 +2062,8 @@
             status = sre_usearch(&state, PatternObject_GetCode(self));
         }
 
-	if (PyErr_Occurred())
-	    goto error;
+        if (PyErr_Occurred())
+            goto error;
 
         if (status <= 0) {
             if (status == 0)
@@ -2190,8 +2190,8 @@
             status = sre_usearch(&state, PatternObject_GetCode(self));
         }
 
-	if (PyErr_Occurred())
-	    goto error;
+        if (PyErr_Occurred())
+            goto error;
 
         if (status <= 0) {
             if (status == 0)
@@ -2334,8 +2334,8 @@
             status = sre_usearch(&state, PatternObject_GetCode(self));
         }
 
-	if (PyErr_Occurred())
-	    goto error;
+        if (PyErr_Occurred())
+            goto error;
 
         if (status <= 0) {
             if (status == 0)
@@ -2553,20 +2553,20 @@
 
 static PyMethodDef pattern_methods[] = {
     {"match", (PyCFunction) pattern_match, METH_VARARGS|METH_KEYWORDS,
-	pattern_match_doc},
+        pattern_match_doc},
     {"search", (PyCFunction) pattern_search, METH_VARARGS|METH_KEYWORDS,
-	pattern_search_doc},
+        pattern_search_doc},
     {"sub", (PyCFunction) pattern_sub, METH_VARARGS|METH_KEYWORDS,
-	pattern_sub_doc},
+        pattern_sub_doc},
     {"subn", (PyCFunction) pattern_subn, METH_VARARGS|METH_KEYWORDS,
-	pattern_subn_doc},
+        pattern_subn_doc},
     {"split", (PyCFunction) pattern_split, METH_VARARGS|METH_KEYWORDS,
-	pattern_split_doc},
+        pattern_split_doc},
     {"findall", (PyCFunction) pattern_findall, METH_VARARGS|METH_KEYWORDS,
-	pattern_findall_doc},
+        pattern_findall_doc},
 #if PY_VERSION_HEX >= 0x02020000
     {"finditer", (PyCFunction) pattern_finditer, METH_VARARGS,
-	pattern_finditer_doc},
+        pattern_finditer_doc},
 #endif
     {"scanner", (PyCFunction) pattern_scanner, METH_VARARGS},
     {"__copy__", (PyCFunction) pattern_copy, METH_NOARGS},
@@ -2587,31 +2587,31 @@
     PyVarObject_HEAD_INIT(NULL, 0)
     "_" SRE_MODULE ".SRE_Pattern",
     sizeof(PatternObject), sizeof(SRE_CODE),
-    (destructor)pattern_dealloc,	/* tp_dealloc */
-    0,					/* tp_print */
-    0,					/* tp_getattr */
-    0,					/* tp_setattr */
-    0,					/* tp_reserved */
-    0,					/* tp_repr */
-    0,					/* tp_as_number */
-    0,					/* tp_as_sequence */
-    0,					/* tp_as_mapping */
-    0,					/* tp_hash */
-    0,					/* tp_call */
-    0,					/* tp_str */
-    0,					/* tp_getattro */
-    0,					/* tp_setattro */
-    0,					/* tp_as_buffer */
-    Py_TPFLAGS_DEFAULT,			/* tp_flags */
-    pattern_doc,			/* tp_doc */
-    0,					/* tp_traverse */
-    0,					/* tp_clear */
-    0,					/* tp_richcompare */
-    offsetof(PatternObject, weakreflist),	/* tp_weaklistoffset */
-    0,					/* tp_iter */
-    0,					/* tp_iternext */
-    pattern_methods,			/* tp_methods */
-    pattern_members,			/* tp_members */
+    (destructor)pattern_dealloc,        /* tp_dealloc */
+    0,                                  /* tp_print */
+    0,                                  /* tp_getattr */
+    0,                                  /* tp_setattr */
+    0,                                  /* tp_reserved */
+    0,                                  /* tp_repr */
+    0,                                  /* tp_as_number */
+    0,                                  /* tp_as_sequence */
+    0,                                  /* tp_as_mapping */
+    0,                                  /* tp_hash */
+    0,                                  /* tp_call */
+    0,                                  /* tp_str */
+    0,                                  /* tp_getattro */
+    0,                                  /* tp_setattro */
+    0,                                  /* tp_as_buffer */
+    Py_TPFLAGS_DEFAULT,                 /* tp_flags */
+    pattern_doc,                        /* tp_doc */
+    0,                                  /* tp_traverse */
+    0,                                  /* tp_clear */
+    0,                                  /* tp_richcompare */
+    offsetof(PatternObject, weakreflist),       /* tp_weaklistoffset */
+    0,                                  /* tp_iter */
+    0,                                  /* tp_iternext */
+    pattern_methods,                    /* tp_methods */
+    pattern_members,                    /* tp_members */
 };
 
 static int _validate(PatternObject *self); /* Forward */
@@ -3210,8 +3210,8 @@
     Py_ssize_t i;
 
     if (index == NULL)
-	/* Default value */
-	return 0;
+        /* Default value */
+        return 0;
 
     if (PyLong_Check(index))
         return PyLong_AsSsize_t(index);
@@ -3551,7 +3551,7 @@
 match_lastindex_get(MatchObject *self)
 {
     if (self->lastindex >= 0)
-	return Py_BuildValue("i", self->lastindex);
+        return Py_BuildValue("i", self->lastindex);
     Py_INCREF(Py_None);
     return Py_None;
 }
@@ -3604,32 +3604,32 @@
     PyVarObject_HEAD_INIT(NULL,0)
     "_" SRE_MODULE ".SRE_Match",
     sizeof(MatchObject), sizeof(Py_ssize_t),
-    (destructor)match_dealloc,	/* tp_dealloc */
-    0,				/* tp_print */
-    0,				/* tp_getattr */
-    0,				/* tp_setattr */
-    0,				/* tp_reserved */
-    0,				/* tp_repr */
-    0,				/* tp_as_number */
-    0,				/* tp_as_sequence */
-    0,				/* tp_as_mapping */
-    0,				/* tp_hash */
-    0,				/* tp_call */
-    0,				/* tp_str */
-    0,				/* tp_getattro */
-    0,				/* tp_setattro */
-    0,				/* tp_as_buffer */
-    Py_TPFLAGS_DEFAULT,		/* tp_flags */
-    0,				/* tp_doc */
-    0,				/* tp_traverse */
-    0,				/* tp_clear */
-    0,				/* tp_richcompare */
-    0,				/* tp_weaklistoffset */
-    0,				/* tp_iter */
-    0,				/* tp_iternext */
-    match_methods,		/* tp_methods */
-    match_members,		/* tp_members */
-    match_getset,  	        /* tp_getset */
+    (destructor)match_dealloc,  /* tp_dealloc */
+    0,                          /* tp_print */
+    0,                          /* tp_getattr */
+    0,                          /* tp_setattr */
+    0,                          /* tp_reserved */
+    0,                          /* tp_repr */
+    0,                          /* tp_as_number */
+    0,                          /* tp_as_sequence */
+    0,                          /* tp_as_mapping */
+    0,                          /* tp_hash */
+    0,                          /* tp_call */
+    0,                          /* tp_str */
+    0,                          /* tp_getattro */
+    0,                          /* tp_setattro */
+    0,                          /* tp_as_buffer */
+    Py_TPFLAGS_DEFAULT,         /* tp_flags */
+    0,                          /* tp_doc */
+    0,                          /* tp_traverse */
+    0,                          /* tp_clear */
+    0,                          /* tp_richcompare */
+    0,                          /* tp_weaklistoffset */
+    0,                          /* tp_iter */
+    0,                          /* tp_iternext */
+    match_methods,              /* tp_methods */
+    match_members,              /* tp_members */
+    match_getset,               /* tp_getset */
 };
 
 static PyObject*
@@ -3776,7 +3776,7 @@
 
 #define SCAN_OFF(x) offsetof(ScannerObject, x)
 static PyMemberDef scanner_members[] = {
-    {"pattern",	T_OBJECT,	SCAN_OFF(pattern),	READONLY},
+    {"pattern", T_OBJECT,       SCAN_OFF(pattern),      READONLY},
     {NULL}  /* Sentinel */
 };
 
@@ -3785,31 +3785,31 @@
     "_" SRE_MODULE ".SRE_Scanner",
     sizeof(ScannerObject), 0,
     (destructor)scanner_dealloc,/* tp_dealloc */
-    0,				/* tp_print */
-    0,				/* tp_getattr */
-    0,				/* tp_setattr */
-    0,				/* tp_reserved */
-    0,				/* tp_repr */
-    0,				/* tp_as_number */
-    0,				/* tp_as_sequence */
-    0,				/* tp_as_mapping */
-    0,				/* tp_hash */
-    0,				/* tp_call */
-    0,				/* tp_str */
-    0,				/* tp_getattro */
-    0,				/* tp_setattro */
-    0,				/* tp_as_buffer */
-    Py_TPFLAGS_DEFAULT,		/* tp_flags */
-    0,				/* tp_doc */
-    0,				/* tp_traverse */
-    0,				/* tp_clear */
-    0,				/* tp_richcompare */
-    0,				/* tp_weaklistoffset */
-    0,				/* tp_iter */
-    0,				/* tp_iternext */
-    scanner_methods,		/* tp_methods */
-    scanner_members,		/* tp_members */
-    0,				/* tp_getset */
+    0,                          /* tp_print */
+    0,                          /* tp_getattr */
+    0,                          /* tp_setattr */
+    0,                          /* tp_reserved */
+    0,                          /* tp_repr */
+    0,                          /* tp_as_number */
+    0,                          /* tp_as_sequence */
+    0,                          /* tp_as_mapping */
+    0,                          /* tp_hash */
+    0,                          /* tp_call */
+    0,                          /* tp_str */
+    0,                          /* tp_getattro */
+    0,                          /* tp_setattro */
+    0,                          /* tp_as_buffer */
+    Py_TPFLAGS_DEFAULT,         /* tp_flags */
+    0,                          /* tp_doc */
+    0,                          /* tp_traverse */
+    0,                          /* tp_clear */
+    0,                          /* tp_richcompare */
+    0,                          /* tp_weaklistoffset */
+    0,                          /* tp_iter */
+    0,                          /* tp_iternext */
+    scanner_methods,            /* tp_methods */
+    scanner_members,            /* tp_members */
+    0,                          /* tp_getset */
 };
 
 static PyObject*
@@ -3851,15 +3851,15 @@
 };
 
 static struct PyModuleDef sremodule = {
-	PyModuleDef_HEAD_INIT,
-	"_" SRE_MODULE,
-	NULL,
-	-1,
-	_functions,
-	NULL,
-	NULL,
-	NULL,
-	NULL
+        PyModuleDef_HEAD_INIT,
+        "_" SRE_MODULE,
+        NULL,
+        -1,
+        _functions,
+        NULL,
+        NULL,
+        NULL,
+        NULL
 };
 
 PyMODINIT_FUNC PyInit__sre(void)
@@ -3875,7 +3875,7 @@
 
     m = PyModule_Create(&sremodule);
     if (m == NULL)
-    	return NULL;
+        return NULL;
     d = PyModule_GetDict(m);
 
     x = PyLong_FromLong(SRE_MAGIC);
diff --git a/Objects/typeobject.c b/Objects/typeobject.c
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -2345,21 +2345,21 @@
       return NULL;
     res->ht_name = PyUnicode_FromString(spec->name);
     if (!res->ht_name)
-	goto fail;
+        goto fail;
     res->ht_type.tp_name = _PyUnicode_AsString(res->ht_name);
     if (!res->ht_type.tp_name)
-	goto fail;
+        goto fail;
 
     res->ht_type.tp_basicsize = spec->basicsize;
     res->ht_type.tp_itemsize = spec->itemsize;
     res->ht_type.tp_flags = spec->flags | Py_TPFLAGS_HEAPTYPE;
 
     for (slot = spec->slots; slot->slot; slot++) {
-	if (slot->slot >= sizeof(slotoffsets)/sizeof(slotoffsets[0])) {
-	    PyErr_SetString(PyExc_RuntimeError, "invalid slot offset");
-	    goto fail;
-	}
-	*(void**)(res_start + slotoffsets[slot->slot]) = slot->pfunc;
+        if (slot->slot >= sizeof(slotoffsets)/sizeof(slotoffsets[0])) {
+            PyErr_SetString(PyExc_RuntimeError, "invalid slot offset");
+            goto fail;
+        }
+        *(void**)(res_start + slotoffsets[slot->slot]) = slot->pfunc;
 
         /* need to make a copy of the docstring slot, which usually
            points to a static string literal */
@@ -2367,7 +2367,7 @@
             ssize_t len = strlen(slot->pfunc)+1;
             char *tp_doc = PyObject_MALLOC(len);
             if (tp_doc == NULL)
-	    	goto fail;
+                goto fail;
             memcpy(tp_doc, slot->pfunc, len);
             res->ht_type.tp_doc = tp_doc;
         }
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -174,10 +174,10 @@
 
 static void
 raise_encode_exception(PyObject **exceptionObject,
-		       const char *encoding,
-		       const Py_UNICODE *unicode, Py_ssize_t size,
-		       Py_ssize_t startpos, Py_ssize_t endpos,
-		       const char *reason);
+                       const char *encoding,
+                       const Py_UNICODE *unicode, Py_ssize_t size,
+                       Py_ssize_t startpos, Py_ssize_t endpos,
+                       const char *reason);
 
 /* Same for linebreaks */
 static unsigned char ascii_linebreak[] = {
@@ -2035,8 +2035,8 @@
 
 PyObject *
 PyUnicode_FromEncodedObject(register PyObject *obj,
-			    const char *encoding,
-			    const char *errors)
+                            const char *encoding,
+                            const char *errors)
 {
     Py_buffer buffer;
     PyObject *v;
@@ -2121,9 +2121,9 @@
 
 PyObject *
 PyUnicode_Decode(const char *s,
-		 Py_ssize_t size,
-		 const char *encoding,
-		 const char *errors)
+                 Py_ssize_t size,
+                 const char *encoding,
+                 const char *errors)
 {
     PyObject *buffer = NULL, *unicode;
     Py_buffer info;
@@ -2184,8 +2184,8 @@
 
 PyObject *
 PyUnicode_AsDecodedObject(PyObject *unicode,
-			  const char *encoding,
-			  const char *errors)
+                          const char *encoding,
+                          const char *errors)
 {
     PyObject *v;
 
@@ -2209,8 +2209,8 @@
 
 PyObject *
 PyUnicode_AsDecodedUnicode(PyObject *unicode,
-			   const char *encoding,
-			   const char *errors)
+                           const char *encoding,
+                           const char *errors)
 {
     PyObject *v;
 
@@ -2241,9 +2241,9 @@
 
 PyObject *
 PyUnicode_Encode(const Py_UNICODE *s,
-		 Py_ssize_t size,
-		 const char *encoding,
-		 const char *errors)
+                 Py_ssize_t size,
+                 const char *encoding,
+                 const char *errors)
 {
     PyObject *v, *unicode;
 
@@ -2257,8 +2257,8 @@
 
 PyObject *
 PyUnicode_AsEncodedObject(PyObject *unicode,
-			  const char *encoding,
-			  const char *errors)
+                          const char *encoding,
+                          const char *errors)
 {
     PyObject *v;
 
@@ -2345,8 +2345,8 @@
 
 PyObject *
 PyUnicode_AsEncodedString(PyObject *unicode,
-			  const char *encoding,
-			  const char *errors)
+                          const char *encoding,
+                          const char *errors)
 {
     PyObject *v;
     char lower[11];  /* Enough for any encoding shortcut */
@@ -2423,8 +2423,8 @@
 
 PyObject *
 PyUnicode_AsEncodedUnicode(PyObject *unicode,
-			   const char *encoding,
-			   const char *errors)
+                           const char *encoding,
+                           const char *errors)
 {
     PyObject *v;
 
@@ -2841,10 +2841,10 @@
 
 static int
 unicode_decode_call_errorhandler(const char *errors, PyObject **errorHandler,
-				 const char *encoding, const char *reason,
-				 const char **input, const char **inend, Py_ssize_t *startinpos,
-				 Py_ssize_t *endinpos, PyObject **exceptionObject, const char **inptr,
-				 PyUnicodeObject **output, Py_ssize_t *outpos, Py_UNICODE **outptr)
+                                 const char *encoding, const char *reason,
+                                 const char **input, const char **inend, Py_ssize_t *startinpos,
+                                 Py_ssize_t *endinpos, PyObject **exceptionObject, const char **inptr,
+                                 PyUnicodeObject **output, Py_ssize_t *outpos, Py_UNICODE **outptr)
 {
     static char *argparse = "O!n;decoding error handler must return (str, int) tuple";
 
@@ -3016,8 +3016,8 @@
 
 PyObject *
 PyUnicode_DecodeUTF7(const char *s,
-		     Py_ssize_t size,
-		     const char *errors)
+                     Py_ssize_t size,
+                     const char *errors)
 {
     return PyUnicode_DecodeUTF7Stateful(s, size, errors, NULL);
 }
@@ -3031,9 +3031,9 @@
 
 PyObject *
 PyUnicode_DecodeUTF7Stateful(const char *s,
-			     Py_ssize_t size,
-			     const char *errors,
-			     Py_ssize_t *consumed)
+                             Py_ssize_t size,
+                             const char *errors,
+                             Py_ssize_t *consumed)
 {
     const char *starts = s;
     Py_ssize_t startinpos;
@@ -3226,10 +3226,10 @@
 
 PyObject *
 PyUnicode_EncodeUTF7(const Py_UNICODE *s,
-		     Py_ssize_t size,
-		     int base64SetO,
-		     int base64WhiteSpace,
-		     const char *errors)
+                     Py_ssize_t size,
+                     int base64SetO,
+                     int base64WhiteSpace,
+                     const char *errors)
 {
     PyObject *v;
     /* It might be possible to tighten this worst case */
@@ -3352,8 +3352,8 @@
 
 PyObject *
 PyUnicode_DecodeUTF8(const char *s,
-		     Py_ssize_t size,
-		     const char *errors)
+                     Py_ssize_t size,
+                     const char *errors)
 {
     return PyUnicode_DecodeUTF8Stateful(s, size, errors, NULL);
 }
@@ -4868,8 +4868,8 @@
 
 PyObject *
 PyUnicode_DecodeUnicodeEscape(const char *s,
-			      Py_ssize_t size,
-			      const char *errors)
+                              Py_ssize_t size,
+                              const char *errors) 
 {
     const char *starts = s;
     Py_ssize_t startinpos;
@@ -5172,7 +5172,7 @@
 
 PyObject *
 PyUnicode_EncodeUnicodeEscape(const Py_UNICODE *s,
-			      Py_ssize_t size)
+                              Py_ssize_t size)
 {
     PyObject *repr;
     char *p;
@@ -5329,8 +5329,8 @@
 
 PyObject *
 PyUnicode_DecodeRawUnicodeEscape(const char *s,
-				 Py_ssize_t size,
-				 const char *errors)
+                                 Py_ssize_t size,
+                                 const char *errors)
 {
     const char *starts = s;
     Py_ssize_t startinpos;
@@ -5451,7 +5451,7 @@
 
 PyObject *
 PyUnicode_EncodeRawUnicodeEscape(const Py_UNICODE *s,
-				 Py_ssize_t size)
+                                 Py_ssize_t size)
 {
     PyObject *repr;
     char *p;
@@ -5556,8 +5556,8 @@
 
 PyObject *
 _PyUnicode_DecodeUnicodeInternal(const char *s,
-				 Py_ssize_t size,
-				 const char *errors)
+                                 Py_ssize_t size,
+                                 const char *errors)
 {
     const char *starts = s;
     Py_ssize_t startinpos;
@@ -5641,8 +5641,8 @@
 
 PyObject *
 PyUnicode_DecodeLatin1(const char *s,
-		       Py_ssize_t size,
-		       const char *errors)
+                       Py_ssize_t size,
+                       const char *errors)
 {
     /* Latin-1 is equivalent to the first 256 ordinals in Unicode. */
     return _PyUnicode_FromUCS1((unsigned char*)s, size);
@@ -5651,10 +5651,10 @@
 /* create or adjust a UnicodeEncodeError */
 static void
 make_encode_exception(PyObject **exceptionObject,
-		      const char *encoding,
-		      const Py_UNICODE *unicode, Py_ssize_t size,
-		      Py_ssize_t startpos, Py_ssize_t endpos,
-		      const char *reason)
+                      const char *encoding,
+                      const Py_UNICODE *unicode, Py_ssize_t size,
+                      Py_ssize_t startpos, Py_ssize_t endpos,
+                      const char *reason)
 {
     if (*exceptionObject == NULL) {
         *exceptionObject = PyUnicodeEncodeError_Create(
@@ -5677,10 +5677,10 @@
 /* raises a UnicodeEncodeError */
 static void
 raise_encode_exception(PyObject **exceptionObject,
-		       const char *encoding,
-		       const Py_UNICODE *unicode, Py_ssize_t size,
-		       Py_ssize_t startpos, Py_ssize_t endpos,
-		       const char *reason)
+                       const char *encoding,
+                       const Py_UNICODE *unicode, Py_ssize_t size,
+                       Py_ssize_t startpos, Py_ssize_t endpos,
+                       const char *reason)
 {
     make_encode_exception(exceptionObject,
                           encoding, unicode, size, startpos, endpos, reason);
@@ -5694,11 +5694,11 @@
    has to be freed by the caller */
 static PyObject *
 unicode_encode_call_errorhandler(const char *errors,
-				 PyObject **errorHandler,
-				 const char *encoding, const char *reason,
-				 const Py_UNICODE *unicode, Py_ssize_t size, PyObject **exceptionObject,
-				 Py_ssize_t startpos, Py_ssize_t endpos,
-				 Py_ssize_t *newpos)
+                                 PyObject **errorHandler,
+                                 const char *encoding, const char *reason,
+                                 const Py_UNICODE *unicode, Py_ssize_t size, PyObject **exceptionObject,
+                                 Py_ssize_t startpos, Py_ssize_t endpos,
+                                 Py_ssize_t *newpos)
 {
     static char *argparse = "On;encoding error handler must return (str/bytes, int) tuple";
 
@@ -5749,9 +5749,9 @@
 
 static PyObject *
 unicode_encode_ucs1(const Py_UNICODE *p,
-		    Py_ssize_t size,
-		    const char *errors,
-		    int limit)
+                    Py_ssize_t size,
+                    const char *errors,
+                    int limit)
 {
     /* output object */
     PyObject *res;
@@ -5946,8 +5946,8 @@
 
 PyObject *
 PyUnicode_EncodeLatin1(const Py_UNICODE *p,
-		       Py_ssize_t size,
-		       const char *errors)
+                       Py_ssize_t size,
+                       const char *errors)
 {
     return unicode_encode_ucs1(p, size, errors, 256);
 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 00:00:37 2011
From: python-checkins at python.org (ezio.melotti)
Date: Thu, 29 Sep 2011 00:00:37 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_whitespace=2E?=
Message-ID: <E1R92BB-0007oE-DG@dinsdale.python.org>

http://hg.python.org/cpython/rev/addbb996f885
changeset:   72505:addbb996f885
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Sep 29 01:00:19 2011 +0300
summary:
  Fix whitespace.

files:
  Modules/_sre.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Modules/_sre.c b/Modules/_sre.c
--- a/Modules/_sre.c
+++ b/Modules/_sre.c
@@ -3776,7 +3776,7 @@
 
 #define SCAN_OFF(x) offsetof(ScannerObject, x)
 static PyMemberDef scanner_members[] = {
-    {"pattern", T_OBJECT,       SCAN_OFF(pattern),      READONLY},
+    {"pattern", T_OBJECT, SCAN_OFF(pattern), READONLY},
     {NULL}  /* Sentinel */
 };
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 00:01:28 2011
From: python-checkins at python.org (ezio.melotti)
Date: Thu, 29 Sep 2011 00:01:28 +0200
Subject: [Python-checkins] =?utf8?q?peps=3A_Add_PyUnicode=5FGetMax_to_the_?=
 =?utf8?q?list_of_deprecated_functions=2E?=
Message-ID: <E1R92C0-0007tc-JT@dinsdale.python.org>

http://hg.python.org/peps/rev/9a154edf18e6
changeset:   3950:9a154edf18e6
user:        Ezio Melotti
date:        Thu Sep 29 01:01:23 2011 +0300
summary:
  Add PyUnicode_GetMax to the list of deprecated functions.

files:
  pep-0393.txt |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/pep-0393.txt b/pep-0393.txt
--- a/pep-0393.txt
+++ b/pep-0393.txt
@@ -300,6 +300,7 @@
   PyUnicode_TransformDecimalToASCII
 - Py_UNICODE_{strlen, strcat, strcpy, strcmp, strchr, strrchr}
 - PyUnicode_AsUnicodeCopy
+- PyUnicode_GetMax
 
 _PyUnicode_AsDefaultEncodedString is removed. It previously returned a
 borrowed reference to an UTF-8-encoded bytes object. Since the unicode

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Thu Sep 29 00:01:51 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 00:01:51 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_PyUnicode=5FCopyCharacters?=
 =?utf8?q?=28=29_fails_if_=27to=27_has_more_than_1_reference?=
Message-ID: <E1R92CN-0007vG-NW@dinsdale.python.org>

http://hg.python.org/cpython/rev/9f0e1f44ebac
changeset:   72506:9f0e1f44ebac
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 28 23:54:59 2011 +0200
summary:
  PyUnicode_CopyCharacters() fails if 'to' has more than 1 reference

files:
  Include/unicodeobject.h |  3 ++-
  Objects/unicodeobject.c |  8 ++++++++
  2 files changed, 10 insertions(+), 1 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -522,7 +522,8 @@
    character conversion when necessary and falls back to memcpy if possible.
 
    Fail if 'to' is smaller than how_many or smaller than len(from)-from_start,
-   or if kind(from[from_start:from_start+how_many]) > kind(to).
+   or if kind(from[from_start:from_start+how_many]) > kind(to), or if to has
+   more than 1 reference.
 
    Return the number of written character, or return -1 and raise an exception
    on error.
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -631,6 +631,14 @@
                      how_many, to_start, PyUnicode_GET_LENGTH(to));
         return -1;
     }
+    if (how_many == 0)
+        return 0;
+
+    if (Py_REFCNT(to) != 1) {
+        PyErr_SetString(PyExc_ValueError,
+                        "Cannot modify a string having more than 1 reference");
+        return -1;
+    }
 
     from_kind = PyUnicode_KIND(from);
     to_kind = PyUnicode_KIND(to);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 00:01:52 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 00:01:52 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_PyUnicode=5FCopyCharacters?=
 =?utf8?q?=28=29_marks_the_string_as_dirty_=28reset_the_hash=29?=
Message-ID: <E1R92CO-0007vK-Gi@dinsdale.python.org>

http://hg.python.org/cpython/rev/29ef50337bdb
changeset:   72507:29ef50337bdb
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Sep 28 23:59:20 2011 +0200
summary:
  PyUnicode_CopyCharacters() marks the string as dirty (reset the hash)

files:
  Objects/unicodeobject.c |  6 +++++-
  1 files changed, 5 insertions(+), 1 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -117,6 +117,9 @@
     (assert(PyUnicode_Check(op)),               \
      ((PyASCIIObject *)(op))->length)
 
+/* The Unicode string has been modified: reset the hash */
+#define _PyUnicode_DIRTY(op) do { _PyUnicode_HASH(op) = -1; } while (0)
+
 
 /* This dictionary holds all interned unicode strings.  Note that references
    to strings in this dictionary are *not* counted in the string's ob_refcnt.
@@ -356,7 +359,7 @@
         _PyUnicode_STATE(unicode).interned = _PyUnicode_STATE(unicode).interned;
         _PyUnicode_STATE(unicode).kind = PyUnicode_WCHAR_KIND;
     }
-    _PyUnicode_HASH(unicode) = -1;
+    _PyUnicode_DIRTY(unicode);
 
     return 0;
 }
@@ -639,6 +642,7 @@
                         "Cannot modify a string having more than 1 reference");
         return -1;
     }
+    _PyUnicode_DIRTY(unicode);
 
     from_kind = PyUnicode_KIND(from);
     to_kind = PyUnicode_KIND(to);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 00:16:51 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 00:16:51 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Oops=2C_fix_my_previous_com?=
	=?utf8?q?mit=3A_unicode_=3D=3E_to?=
Message-ID: <E1R92Qt-0000pv-LU@dinsdale.python.org>

http://hg.python.org/cpython/rev/8d4252f0bf14
changeset:   72508:8d4252f0bf14
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 00:16:58 2011 +0200
summary:
  Oops, fix my previous commit: unicode => to

files:
  Objects/unicodeobject.c |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -642,7 +642,7 @@
                         "Cannot modify a string having more than 1 reference");
         return -1;
     }
-    _PyUnicode_DIRTY(unicode);
+    _PyUnicode_DIRTY(to);
 
     from_kind = PyUnicode_KIND(from);
     to_kind = PyUnicode_KIND(to);
@@ -4881,7 +4881,7 @@
 PyObject *
 PyUnicode_DecodeUnicodeEscape(const char *s,
                               Py_ssize_t size,
-                              const char *errors) 
+                              const char *errors)
 {
     const char *starts = s;
     Py_ssize_t startinpos;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 00:42:23 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 00:42:23 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_=27c=27_format_of_PyUni?=
	=?utf8?q?code=5FFormat=28=29?=
Message-ID: <E1R92pb-00035j-7b@dinsdale.python.org>

http://hg.python.org/cpython/rev/876a36feb508
changeset:   72509:876a36feb508
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 00:39:24 2011 +0200
summary:
  Fix 'c' format of PyUnicode_Format()

formatbuf is now an array of Py_UCS4, not of Py_UNICODE

files:
  Objects/unicodeobject.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -12254,7 +12254,7 @@
             case 'c':
                 pbuf = formatbuf;
                 kind = PyUnicode_4BYTE_KIND;
-                len = formatchar(pbuf, sizeof(formatbuf)/sizeof(Py_UNICODE), v);
+                len = formatchar(pbuf, Py_ARRAY_LENGTH(formatbuf), v);
                 if (len < 0)
                     goto onError;
                 break;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 00:42:24 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 00:42:24 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Use_the_new_Py=5FARRAY=5FLE?=
	=?utf8?q?NGTH_macro?=
Message-ID: <E1R92pc-00035u-5l@dinsdale.python.org>

http://hg.python.org/cpython/rev/b534b56379cb
changeset:   72510:b534b56379cb
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 00:42:28 2011 +0200
summary:
  Use the new Py_ARRAY_LENGTH macro

files:
  Modules/_testcapimodule.c |   2 +-
  Modules/faulthandler.c    |   2 +-
  Modules/mathmodule.c      |   2 +-
  Modules/ossaudiodev.c     |   6 +++---
  Modules/posixmodule.c     |  10 +++++-----
  Modules/socketmodule.c    |   4 ++--
  Modules/unicodedata.c     |   2 +-
  Objects/longobject.c      |   3 +--
  Objects/typeobject.c      |   4 ++--
  Objects/unicodeobject.c   |   2 +-
  PC/frozen_dllmain.c       |   4 ++--
  PC/getpathp.c             |   6 +++---
  Python/codecs.c           |   2 +-
  Python/dynload_aix.c      |   3 +--
  Python/sysmodule.c        |   2 +-
  15 files changed, 26 insertions(+), 28 deletions(-)


diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -1593,7 +1593,7 @@
              {-0xfffffffL, 28, -1}};
     int i;
 
-    for (i = 0; i < sizeof(testcases) / sizeof(struct triple); ++i) {
+    for (i = 0; i < Py_ARRAY_LENGTH(testcases); ++i) {
         PyObject *plong = PyLong_FromLong(testcases[i].input);
         size_t nbits = _PyLong_NumBits(plong);
         int sign = _PyLong_Sign(plong);
diff --git a/Modules/faulthandler.c b/Modules/faulthandler.c
--- a/Modules/faulthandler.c
+++ b/Modules/faulthandler.c
@@ -112,7 +112,7 @@
     {SIGSEGV, 0, "Segmentation fault", }
 };
 static const unsigned char faulthandler_nsignals = \
-    sizeof(faulthandler_handlers) / sizeof(faulthandler_handlers[0]);
+    Py_ARRAY_LENGTH(faulthandler_handlers);
 
 #ifdef HAVE_SIGALTSTACK
 static stack_t stack;
diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c
--- a/Modules/mathmodule.c
+++ b/Modules/mathmodule.c
@@ -1435,7 +1435,7 @@
     }
 
     /* use lookup table if x is small */
-    if (x < (long)(sizeof(SmallFactorials)/sizeof(SmallFactorials[0])))
+    if (x < (long)Py_ARRAY_LENGTH(SmallFactorials))
         return PyLong_FromUnsignedLong(SmallFactorials[x]);
 
     /* else express in the form odd_part * 2**two_valuation, and compute as
diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c
--- a/Modules/ossaudiodev.c
+++ b/Modules/ossaudiodev.c
@@ -530,7 +530,7 @@
     return self;
 }
 
-static PyObject * 
+static PyObject *
 oss_exit(PyObject *self, PyObject *unused)
 {
     PyObject *ret = PyObject_CallMethod(self, "close", NULL);
@@ -1061,8 +1061,8 @@
     int num_controls;
     int i;
 
-    num_controls = sizeof(control_labels) / sizeof(control_labels[0]);
-    assert(num_controls == sizeof(control_names) / sizeof(control_names[0]));
+    num_controls = Py_ARRAY_LENGTH(control_labels);
+    assert(num_controls == Py_ARRAY_LENGTH(control_names));
 
     labels = PyList_New(num_controls);
     names = PyList_New(num_controls);
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -2895,9 +2895,9 @@
         DWORD result;
         PyObject *v;
         result = GetFullPathNameW(wpath,
-                                  sizeof(woutbuf)/sizeof(woutbuf[0]),
+                                  Py_ARRAY_LENGTH(woutbuf),
                                   woutbuf, &wtemp);
-        if (result > sizeof(woutbuf)/sizeof(woutbuf[0])) {
+        if (result > Py_ARRAY_LENGTH(woutbuf)) {
             woutbufp = malloc(result * sizeof(Py_UNICODE));
             if (!woutbufp)
                 return PyErr_NoMemory();
@@ -2920,7 +2920,7 @@
                            PyUnicode_FSConverter, &opath))
         return NULL;
     path = PyBytes_AsString(opath);
-    if (!GetFullPathName(path, sizeof(outbuf)/sizeof(outbuf[0]),
+    if (!GetFullPathName(path, Py_ARRAY_LENGTH(outbuf),
                          outbuf, &temp)) {
         win32_error("GetFullPathName", path);
         Py_DECREF(opath);
@@ -4903,7 +4903,7 @@
 cpu_set_repr(Py_cpu_set *set)
 {
     return PyUnicode_FromFormat("<cpu_set with %li entries>", set->ncpus);
-} 
+}
 
 static Py_ssize_t
 cpu_set_len(Py_cpu_set *set)
@@ -5656,7 +5656,7 @@
     PyObject *result = NULL;
 #ifdef MS_WINDOWS
     wchar_t user_name[UNLEN + 1];
-    DWORD num_chars = sizeof(user_name)/sizeof(user_name[0]);
+    DWORD num_chars = Py_ARRAY_LENGTH(user_name);
 
     if (GetUserNameW(user_name, &num_chars)) {
         /* num_chars is the number of unicode chars plus null terminator */
diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c
--- a/Modules/socketmodule.c
+++ b/Modules/socketmodule.c
@@ -3812,7 +3812,7 @@
        version of the hostname, whereas we need a Unicode string.
        Otherwise, gethostname apparently also returns the DNS name. */
     wchar_t buf[MAX_COMPUTERNAME_LENGTH + 1];
-    DWORD size = sizeof(buf) / sizeof(wchar_t);
+    DWORD size = Py_ARRAY_LENGTH(buf);
     PyObject *result;
     if (!GetComputerNameExW(ComputerNamePhysicalDnsHostname, buf, &size)) {
         if (GetLastError() == ERROR_MORE_DATA) {
@@ -6281,7 +6281,7 @@
         DWORD codes[] = {SIO_RCVALL, SIO_KEEPALIVE_VALS};
         const char *names[] = {"SIO_RCVALL", "SIO_KEEPALIVE_VALS"};
         int i;
-        for(i = 0; i<sizeof(codes)/sizeof(*codes); ++i) {
+        for(i = 0; i<Py_ARRAY_LENGTH(codes); ++i) {
             PyObject *tmp;
             tmp = PyLong_FromUnsignedLong(codes[i]);
             if (tmp == NULL)
diff --git a/Modules/unicodedata.c b/Modules/unicodedata.c
--- a/Modules/unicodedata.c
+++ b/Modules/unicodedata.c
@@ -439,7 +439,7 @@
        from Tools/unicode/makeunicodedata.py, it should not be possible
        to overflow decomp_prefix. */
     prefix_index = decomp_data[index] & 255;
-    assert(prefix_index < (sizeof(decomp_prefix)/sizeof(*decomp_prefix)));
+    assert(prefix_index < Py_ARRAY_LENGTH(decomp_prefix));
 
     /* copy prefix */
     i = strlen(decomp_prefix[prefix_index]);
diff --git a/Objects/longobject.c b/Objects/longobject.c
--- a/Objects/longobject.c
+++ b/Objects/longobject.c
@@ -2444,8 +2444,7 @@
                     break;
                 }
     }
-    assert(1 <= x_size &&
-           x_size <= (Py_ssize_t)(sizeof(x_digits)/sizeof(digit)));
+    assert(1 <= x_size && x_size <= (Py_ssize_t)Py_ARRAY_LENGTH(x_digits));
 
     /* Round, and convert to double. */
     x_digits[0] += half_even_correction[x_digits[0] & 7];
diff --git a/Objects/typeobject.c b/Objects/typeobject.c
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -2355,7 +2355,7 @@
     res->ht_type.tp_flags = spec->flags | Py_TPFLAGS_HEAPTYPE;
 
     for (slot = spec->slots; slot->slot; slot++) {
-        if (slot->slot >= sizeof(slotoffsets)/sizeof(slotoffsets[0])) {
+        if (slot->slot >= Py_ARRAY_LENGTH(slotoffsets)) {
             PyErr_SetString(PyExc_RuntimeError, "invalid slot offset");
             goto fail;
         }
@@ -2583,7 +2583,7 @@
     return PyDict_New();
 }
 
-/* 
+/*
    Merge the __dict__ of aclass into dict, and recursively also all
    the __dict__s of aclass's base classes.  The order of merging isn't
    defined, as it's expected that only the final set of dict keys is
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -12552,7 +12552,7 @@
     /* initialize the linebreak bloom filter */
     bloom_linebreak = make_bloom_mask(
         PyUnicode_2BYTE_KIND, linebreak,
-        sizeof(linebreak) / sizeof(linebreak[0]));
+        Py_ARRAY_LENGTH(linebreak));
 
     PyType_Ready(&EncodingMapType);
 }
diff --git a/PC/frozen_dllmain.c b/PC/frozen_dllmain.c
--- a/PC/frozen_dllmain.c
+++ b/PC/frozen_dllmain.c
@@ -77,7 +77,7 @@
 {
     // Must go backwards
     char **modName;
-    for (modName = possibleModules+(sizeof(possibleModules) / sizeof(char *))-2;
+    for (modName = possibleModules+Py_ARRAY_LENGTH(possibleModules)-2;
          modName >= possibleModules;
          *modName--) {
 /*              printf("Terminating '%s'\n", *modName);*/
@@ -103,7 +103,7 @@
         {
             // Must go backwards
             char **modName;
-            for (modName = possibleModules+(sizeof(possibleModules) / sizeof(char *))-2;
+            for (modName = possibleModules+Py_ARRAY_LENGTH(possibleModules)-2;
                  modName >= possibleModules;
                  *modName--)
                 CallModuleDllMain(*modName, DLL_PROCESS_DETACH);
diff --git a/PC/getpathp.c b/PC/getpathp.c
--- a/PC/getpathp.c
+++ b/PC/getpathp.c
@@ -251,7 +251,7 @@
     if (keyBuf==NULL) goto done;
 
     memcpy(keyBufPtr, keyPrefix, sizeof(keyPrefix)-sizeof(WCHAR));
-    keyBufPtr += sizeof(keyPrefix)/sizeof(WCHAR) - 1;
+    keyBufPtr += Py_ARRAY_LENGTH(keyPrefix) - 1;
     mbstowcs(keyBufPtr, PyWin_DLLVersionString, versionLen);
     keyBufPtr += versionLen;
     /* NULL comes with this one! */
@@ -708,8 +708,8 @@
     return progpath;
 }
 
-/* Load python3.dll before loading any extension module that might refer 
-   to it. That way, we can be sure that always the python3.dll corresponding 
+/* Load python3.dll before loading any extension module that might refer
+   to it. That way, we can be sure that always the python3.dll corresponding
    to this python DLL is loaded, not a python3.dll that might be on the path
    by chance.
    Return whether the DLL was found.
diff --git a/Python/codecs.c b/Python/codecs.c
--- a/Python/codecs.c
+++ b/Python/codecs.c
@@ -1044,7 +1044,7 @@
     interp->codec_error_registry = PyDict_New();
 
     if (interp->codec_error_registry) {
-        for (i = 0; i < sizeof(methods)/sizeof(methods[0]); ++i) {
+        for (i = 0; i < Py_ARRAY_LENGTH(methods); ++i) {
             PyObject *func = PyCFunction_New(&methods[i].def, NULL);
             int res;
             if (!func)
diff --git a/Python/dynload_aix.c b/Python/dynload_aix.c
--- a/Python/dynload_aix.c
+++ b/Python/dynload_aix.c
@@ -129,7 +129,6 @@
         {L_ERROR_ERRNO,                 NULL}
     };
 
-#define LOAD_ERRTAB_LEN (sizeof(load_errtab)/sizeof(load_errtab[0]))
 #define ERRBUF_APPEND(s) strncat(errbuf, s, sizeof(errbuf)-strlen(errbuf)-1)
 
     PyOS_snprintf(errbuf, sizeof(errbuf), "from module %.200s ", pathname);
@@ -140,7 +139,7 @@
     }
     for(i = 0; message[i] && *message[i]; i++) {
         int nerr = atoi(message[i]);
-        for (j=0; j<LOAD_ERRTAB_LEN ; j++) {
+        for (j=0; j < Py_ARRAY_LENGTH(load_errtab); j++) {
             if (nerr == load_errtab[j].errNo && load_errtab[j].errstr)
             ERRBUF_APPEND(load_errtab[j].errstr);
         }
diff --git a/Python/sysmodule.c b/Python/sysmodule.c
--- a/Python/sysmodule.c
+++ b/Python/sysmodule.c
@@ -1771,7 +1771,7 @@
         the argument must be the full path anyway. */
         wchar_t *ptemp;
         if (GetFullPathNameW(argv0,
-                           sizeof(fullpath)/sizeof(fullpath[0]),
+                           Py_ARRAY_LENGTH(fullpath),
                            fullpath,
                            &ptemp)) {
             argv0 = fullpath;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 01:05:42 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 01:05:42 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Move_code_related_to_compil?=
 =?utf8?q?e_from_Python=2Eh_to_compile=2Eh?=
Message-ID: <E1R93CA-0005BE-M4@dinsdale.python.org>

http://hg.python.org/cpython/rev/775977c601cd
changeset:   72511:775977c601cd
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 01:04:08 2011 +0200
summary:
  Move code related to compile from Python.h to compile.h

files:
  Include/Python.h  |  19 -------------------
  Include/compile.h |  14 ++++++++++++--
  2 files changed, 12 insertions(+), 21 deletions(-)


diff --git a/Include/Python.h b/Include/Python.h
--- a/Include/Python.h
+++ b/Include/Python.h
@@ -127,30 +127,11 @@
 #include "dtoa.h"
 #include "fileutils.h"
 
-#ifdef __cplusplus
-extern "C" {
-#endif
-
-/* _Py_Mangle is defined in compile.c */
-#ifndef Py_LIMITED_API
-PyAPI_FUNC(PyObject*) _Py_Mangle(PyObject *p, PyObject *name);
-#endif
-
-#ifdef __cplusplus
-}
-#endif
-
 /* Argument must be a char or an int in [-128, 127] or [0, 255]. */
 #define Py_CHARMASK(c) ((unsigned char)((c) & 0xff))
 
 #include "pyfpe.h"
 
-/* These definitions must match corresponding definitions in graminit.h.
-   There's code in compile.c that checks that they are the same. */
-#define Py_single_input 256
-#define Py_file_input 257
-#define Py_eval_input 258
-
 /* Define macros for inline documentation. */
 #define PyDoc_VAR(name) static char name[]
 #define PyDoc_STRVAR(name,str) PyDoc_VAR(name) = PyDoc_STR(str)
diff --git a/Include/compile.h b/Include/compile.h
--- a/Include/compile.h
+++ b/Include/compile.h
@@ -1,7 +1,7 @@
-#ifndef Py_LIMITED_API
 #ifndef Py_COMPILE_H
 #define Py_COMPILE_H
 
+#ifndef Py_LIMITED_API
 #include "code.h"
 
 #ifdef __cplusplus
@@ -38,9 +38,19 @@
     PyArena *arena);
 PyAPI_FUNC(PyFutureFeatures *) PyFuture_FromAST(struct _mod *, const char *);
 
+/* _Py_Mangle is defined in compile.c */
+PyAPI_FUNC(PyObject*) _Py_Mangle(PyObject *p, PyObject *name);
 
 #ifdef __cplusplus
 }
 #endif
+
+#endif /* !Py_LIMITED_API */
+
+/* These definitions must match corresponding definitions in graminit.h.
+   There's code in compile.c that checks that they are the same. */
+#define Py_single_input 256
+#define Py_file_input 257
+#define Py_eval_input 258
+
 #endif /* !Py_COMPILE_H */
-#endif /* !Py_LIMITED_API */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 01:12:12 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 01:12:12 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Enhance_Py=5FARRAY=5FLENGTH?=
 =?utf8?q?=28=29=3A_fail_at_build_time_if_the_argument_is_not_an_array?=
Message-ID: <E1R93IS-0005jZ-LZ@dinsdale.python.org>

http://hg.python.org/cpython/rev/36fc514de7f0
changeset:   72512:36fc514de7f0
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 01:12:24 2011 +0200
summary:
  Enhance Py_ARRAY_LENGTH(): fail at build time if the argument is not an array

Move other various macros to pymcacro.h

Thanks Rusty Russell for having written these amazing C macros!

files:
  Include/Python.h          |  19 +--------
  Include/pymacro.h         |  57 +++++++++++++++++++++++++++
  Makefile.pre.in           |   3 +-
  Misc/ACKS                 |   1 +
  PCbuild/pythoncore.vcproj |   4 +
  5 files changed, 65 insertions(+), 19 deletions(-)


diff --git a/Include/Python.h b/Include/Python.h
--- a/Include/Python.h
+++ b/Include/Python.h
@@ -48,6 +48,7 @@
 #include <assert.h>
 
 #include "pyport.h"
+#include "pymacro.h"
 
 #include "pyatomic.h"
 
@@ -126,24 +127,6 @@
 #include "pystrcmp.h"
 #include "dtoa.h"
 #include "fileutils.h"
-
-/* Argument must be a char or an int in [-128, 127] or [0, 255]. */
-#define Py_CHARMASK(c) ((unsigned char)((c) & 0xff))
-
 #include "pyfpe.h"
 
-/* Define macros for inline documentation. */
-#define PyDoc_VAR(name) static char name[]
-#define PyDoc_STRVAR(name,str) PyDoc_VAR(name) = PyDoc_STR(str)
-#ifdef WITH_DOC_STRINGS
-#define PyDoc_STR(str) str
-#else
-#define PyDoc_STR(str) ""
-#endif
-
-#define Py_ARRAY_LENGTH(array) (sizeof(array) / sizeof((array)[0]))
-
-#define Py_MIN(x, y) (((x) > (y)) ? (y) : (x))
-#define Py_MAX(x, y) (((x) > (y)) ? (x) : (y))
-
 #endif /* !Py_PYTHON_H */
diff --git a/Include/pymacro.h b/Include/pymacro.h
new file mode 100644
--- /dev/null
+++ b/Include/pymacro.h
@@ -0,0 +1,57 @@
+#ifndef Py_PYMACRO_H
+#define Py_PYMACRO_H
+
+#define Py_MIN(x, y) (((x) > (y)) ? (y) : (x))
+#define Py_MAX(x, y) (((x) > (y)) ? (x) : (y))
+
+/* Argument must be a char or an int in [-128, 127] or [0, 255]. */
+#define Py_CHARMASK(c) ((unsigned char)((c) & 0xff))
+
+
+/* Assert a build-time dependency, as an expression.
+
+   Your compile will fail if the condition isn't true, or can't be evaluated
+   by the compiler. This can be used in an expression: its value is 0.
+
+   Example:
+
+   #define foo_to_char(foo)  \
+       ((char *)(foo)        \
+        + Py_BUILD_ASSERT(offsetof(struct foo, string) == 0))
+
+   Written by Rusty Russell, public domain. */
+#define Py_BUILD_ASSERT(cond) \
+    (sizeof(char [1 - 2*!(cond)]) - 1)
+
+#if defined(__GNUC__)
+/* Two gcc extensions.
+   &a[0] degrades to a pointer: a different type from an array */
+#define _Py_ARRAY_LENGTH_CHECK(array) \
+    Py_BUILD_ASSERT(!__builtin_types_compatible_p(typeof(array), \
+                                                  typeof(&(array)[0])))
+#else
+#define _Py_ARRAY_LENGTH_CHECK(array) 0
+#endif
+
+
+/* Get the number of elements in a visible array
+
+   This does not work on pointers, or arrays declared as [], or function
+   parameters. With correct compiler support, such usage will cause a build
+   error (see Py_BUILD_ASSERT).
+
+   Written by Rusty Russell, public domain. */
+#define Py_ARRAY_LENGTH(array) \
+    (sizeof(array) / sizeof((array)[0]) + _Py_ARRAY_LENGTH_CHECK(array))
+
+
+/* Define macros for inline documentation. */
+#define PyDoc_VAR(name) static char name[]
+#define PyDoc_STRVAR(name,str) PyDoc_VAR(name) = PyDoc_STR(str)
+#ifdef WITH_DOC_STRINGS
+#define PyDoc_STR(str) str
+#else
+#define PyDoc_STR(str) ""
+#endif
+
+#endif /* Py_PYMACRO_H */
diff --git a/Makefile.pre.in b/Makefile.pre.in
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -493,7 +493,7 @@
 		-install_name $(DESTDIR)$(PYTHONFRAMEWORKINSTALLDIR)/Versions/$(VERSION)/$(PYTHONFRAMEWORK) \
 		-compatibility_version $(VERSION) \
 		-current_version $(VERSION) \
-		-framework CoreFoundation $(LIBS); 
+		-framework CoreFoundation $(LIBS);
 	$(INSTALL) -d -m $(DIRMODE)  \
 		$(PYTHONFRAMEWORKDIR)/Versions/$(VERSION)/Resources/English.lproj
 	$(INSTALL_DATA) $(RESSRCDIR)/Info.plist \
@@ -718,6 +718,7 @@
 		Include/pyfpe.h \
 		Include/pymath.h \
 		Include/pygetopt.h \
+		Include/pymacro.h \
 		Include/pymem.h \
 		Include/pyport.h \
 		Include/pystate.h \
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -825,6 +825,7 @@
 Jeff Rush
 Sam Rushing
 Mark Russell
+Rusty Russell
 Nick Russo
 Patrick Sabin
 S?bastien Sabl?
diff --git a/PCbuild/pythoncore.vcproj b/PCbuild/pythoncore.vcproj
--- a/PCbuild/pythoncore.vcproj
+++ b/PCbuild/pythoncore.vcproj
@@ -887,6 +887,10 @@
 				>
 			</File>
 			<File
+				RelativePath="..\Include\pymacro.h"
+				>
+			</File>
+			<File
 				RelativePath="..\Include\pymem.h"
 				>
 			</File>

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 01:55:52 2011
From: python-checkins at python.org (senthil.kumaran)
Date: Thu, 29 Sep 2011 01:55:52 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Doc_fix=2E_Math?=
 =?utf8?q?ematically_correct_sentence=2E?=
Message-ID: <E1R93yi-0000tO-0D@dinsdale.python.org>

http://hg.python.org/cpython/rev/f612b783bc17
changeset:   72513:f612b783bc17
branch:      2.7
parent:      72472:1fb5b0cc6367
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Thu Sep 29 07:52:46 2011 +0800
summary:
  Doc fix. Mathematically correct sentence.

files:
  Doc/tutorial/datastructures.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst
--- a/Doc/tutorial/datastructures.rst
+++ b/Doc/tutorial/datastructures.rst
@@ -170,8 +170,8 @@
 ``filter(function, sequence)`` returns a sequence consisting of those items from
 the sequence for which ``function(item)`` is true. If *sequence* is a
 :class:`string` or :class:`tuple`, the result will be of the same type;
-otherwise, it is always a :class:`list`. For example, to compute primes up
-to 25::
+otherwise, it is always a :class:`list`. For example, to compute a sequence of
+numbers not divisible by 2 and 3::
 
    >>> def f(x): return x % 2 != 0 and x % 3 != 0
    ...

-- 
Repository URL: http://hg.python.org/cpython

From benjamin at python.org  Thu Sep 29 02:07:02 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 28 Sep 2011 20:07:02 -0400
Subject: [Python-checkins] cpython: Enhance Py_ARRAY_LENGTH(): fail at
 build time if the argument is not an array
In-Reply-To: <E1R93IS-0005jZ-LZ@dinsdale.python.org>
References: <E1R93IS-0005jZ-LZ@dinsdale.python.org>
Message-ID: <CAPZV6o8fDKZoipB-3-r+k6pQEQsURWQJfLmBxJOMivrh-M=RkA@mail.gmail.com>

2011/9/28 victor.stinner <python-checkins at python.org>:
> http://hg.python.org/cpython/rev/36fc514de7f0
> changeset: ? 72512:36fc514de7f0
> user: ? ? ? ?Victor Stinner <victor.stinner at haypocalc.com>
> date: ? ? ? ?Thu Sep 29 01:12:24 2011 +0200
> summary:
> ?Enhance Py_ARRAY_LENGTH(): fail at build time if the argument is not an array
>
> Move other various macros to pymcacro.h
>
> Thanks Rusty Russell for having written these amazing C macros!
>
> files:
> ?Include/Python.h ? ? ? ? ?| ?19 +--------
> ?Include/pymacro.h ? ? ? ? | ?57 +++++++++++++++++++++++++++

Do we really need a new file? Why not pyport.h where other compiler stuff goes?


-- 
Regards,
Benjamin

From python-checkins at python.org  Thu Sep 29 02:15:08 2011
From: python-checkins at python.org (ezio.melotti)
Date: Thu, 29 Sep 2011 02:15:08 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Move_UCS4-specific_tests_wi?=
 =?utf8?q?th_the_=22normal=22_tests=2E?=
Message-ID: <E1R94HM-0002mN-3P@dinsdale.python.org>

http://hg.python.org/cpython/rev/9f41acdb3150
changeset:   72514:9f41acdb3150
parent:      72512:36fc514de7f0
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Sep 29 03:14:56 2011 +0300
summary:
  Move UCS4-specific tests with the "normal" tests.

files:
  Lib/test/test_unicodedata.py |  12 ++++--------
  1 files changed, 4 insertions(+), 8 deletions(-)


diff --git a/Lib/test/test_unicodedata.py b/Lib/test/test_unicodedata.py
--- a/Lib/test/test_unicodedata.py
+++ b/Lib/test/test_unicodedata.py
@@ -108,6 +108,7 @@
         self.assertEqual(self.db.digit('\u215b', None), None)
         self.assertEqual(self.db.digit('\u2468'), 9)
         self.assertEqual(self.db.digit('\U00020000', None), None)
+        self.assertEqual(self.db.digit('\U0001D7FD'), 7)
 
         self.assertRaises(TypeError, self.db.digit)
         self.assertRaises(TypeError, self.db.digit, 'xx')
@@ -120,6 +121,7 @@
         self.assertEqual(self.db.numeric('\u2468'), 9.0)
         self.assertEqual(self.db.numeric('\ua627'), 7.0)
         self.assertEqual(self.db.numeric('\U00020000', None), None)
+        self.assertEqual(self.db.numeric('\U0001012A'), 9000)
 
         self.assertRaises(TypeError, self.db.numeric)
         self.assertRaises(TypeError, self.db.numeric, 'xx')
@@ -131,6 +133,7 @@
         self.assertEqual(self.db.decimal('\u215b', None), None)
         self.assertEqual(self.db.decimal('\u2468', None), None)
         self.assertEqual(self.db.decimal('\U00020000', None), None)
+        self.assertEqual(self.db.decimal('\U0001D7FD'), 7)
 
         self.assertRaises(TypeError, self.db.decimal)
         self.assertRaises(TypeError, self.db.decimal, 'xx')
@@ -141,6 +144,7 @@
         self.assertEqual(self.db.category('a'), 'Ll')
         self.assertEqual(self.db.category('A'), 'Lu')
         self.assertEqual(self.db.category('\U00020000'), 'Lo')
+        self.assertEqual(self.db.category('\U0001012A'), 'No')
 
         self.assertRaises(TypeError, self.db.category)
         self.assertRaises(TypeError, self.db.category, 'xx')
@@ -308,14 +312,6 @@
                 self.assertEqual(len(lines), 1,
                                  r"\u%.4x should not be a linebreak" % i)
 
-    def test_UCS4(self):
-        # unicodedata should work with code points outside the BMP
-        # even on a narrow Unicode build
-        self.assertEqual(self.db.category("\U0001012A"), "No")
-        self.assertEqual(self.db.numeric("\U0001012A"), 9000)
-        self.assertEqual(self.db.decimal("\U0001D7FD"), 7)
-        self.assertEqual(self.db.digit("\U0001D7FD"), 7)
-
 def test_main():
     test.support.run_unittest(
         UnicodeMiscTest,

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 02:56:16 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 02:56:16 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Complete_What=27s_New_in_3?=
	=?utf8?q?=2E3_about_PEP_393?=
Message-ID: <E1R94vA-00068Q-0o@dinsdale.python.org>

http://hg.python.org/cpython/rev/991ef9253263
changeset:   72515:991ef9253263
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 02:56:16 2011 +0200
summary:
  Complete What's New in 3.3 about PEP 393

files:
  Doc/whatsnew/3.3.rst |  28 ++++++++++++++++++++++++++++
  1 files changed, 28 insertions(+), 0 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -65,6 +65,28 @@
   either ``0xFFFF`` or ``0x10FFFF`` for backward compatibility, and it should
   not be used with the new Unicode API (see :issue:`13054`).
 
+* Non-BMP characters (U+10000-U+10FFFF range) are no more special cases.
+  ``'\U0010FFFF'[0]`` is now ``'\U0010FFFF'`` on any platform, instead of
+  ``'\uDFFF'`` on narrow build or ``'\U0010FFFF'`` on wide build. And
+  ``len('\U0010FFFF')`` is now ``1`` on any platform, instead of ``2`` on
+  narrow build or ``1`` on wide build. More generally, most bugs related to
+  non-BMP characters are now fixed. For example, :func:`unicodedata.normalize`
+  handles correctly non-BMP characters on all platforms.
+
+* The storage of Unicode string is now adapted on the content of the string.
+  Pure ASCII and Latin1 strings (U+0000-U+00FF) use 1 byte per character, BMP
+  strings (U+0000-U+FFFF) use 2 bytes per character, and non-BMP characters
+  (U+10000-U+10FFFF range) use 4 bytes per characters. The memory usage of
+  Python 3.3 is two to three times smaller than Python 3.2, and a little bit
+  better than Python 2.7, on a `Django benchmark
+  <http://mail.python.org/pipermail/python-dev/2011-September/113714.html>`_.
+
+* The PEP 393 is fully backward compatible. The legacy API should remain
+  available at least five years. Applications using the legacy API will not
+  fully benefit of the memory reduction, or worse may use a little bit more
+  memory, because Python may have to maintain two versions of each string (in
+  the legacy format and in the new efficient storage).
+
 
 Other Language Changes
 ======================
@@ -334,3 +356,9 @@
 .. Issue #10998: -Q command-line flags are related artifacts have been
    removed.  Code checking sys.flags.division_warning will need updating.
    Contributed by ?ric Araujo.
+
+* :pep:`393`: The :c:type:`Py_UNICODE` type and all functions using this type
+  are deprecated. To fully benefit of the memory footprint reduction provided
+  by the PEP 393, you have to convert your code to the new Unicode API. Read
+  the porting guide: XXX.
+

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 03:36:47 2011
From: python-checkins at python.org (ezio.melotti)
Date: Thu, 29 Sep 2011 03:36:47 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_now_useless_redefini?=
 =?utf8?q?tion_of_chr/ord_for_narrow_builds_in?=
Message-ID: <E1R95YN-0000kd-Lq@dinsdale.python.org>

http://hg.python.org/cpython/rev/90fef68f06e7
changeset:   72516:90fef68f06e7
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Sep 29 04:36:38 2011 +0300
summary:
  Remove now useless redefinition of chr/ord for narrow builds in test_multibytecodec_support.py.

files:
  Lib/test/test_multibytecodec_support.py |  15 -------------
  1 files changed, 0 insertions(+), 15 deletions(-)


diff --git a/Lib/test/test_multibytecodec_support.py b/Lib/test/test_multibytecodec_support.py
--- a/Lib/test/test_multibytecodec_support.py
+++ b/Lib/test/test_multibytecodec_support.py
@@ -264,21 +264,6 @@
 
                 self.assertEqual(ostream.getvalue(), self.tstring[0])
 
-if len('\U00012345') == 2: # ucs2 build
-    _unichr = chr
-    def chr(v):
-        if v >= 0x10000:
-            return _unichr(0xd800 + ((v - 0x10000) >> 10)) + \
-                   _unichr(0xdc00 + ((v - 0x10000) & 0x3ff))
-        else:
-            return _unichr(v)
-    _ord = ord
-    def ord(c):
-        if len(c) == 2:
-            return 0x10000 + ((_ord(c[0]) - 0xd800) << 10) + \
-                          (ord(c[1]) - 0xdc00)
-        else:
-            return _ord(c)
 
 class TestBase_Mapping(unittest.TestCase):
     pass_enctest = []

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 03:56:17 2011
From: python-checkins at python.org (ezio.melotti)
Date: Thu, 29 Sep 2011 03:56:17 +0200
Subject: [Python-checkins] =?utf8?q?peps=3A_Fix_typos_and_whitespace_in_PE?=
	=?utf8?q?P-393=2E?=
Message-ID: <E1R95rF-0002FU-64@dinsdale.python.org>

http://hg.python.org/peps/rev/48b81244ae34
changeset:   3951:48b81244ae34
user:        Ezio Melotti
date:        Thu Sep 29 04:56:12 2011 +0300
summary:
  Fix typos and whitespace in PEP-393.

files:
  pep-0393.txt |  16 ++++++++--------
  1 files changed, 8 insertions(+), 8 deletions(-)


diff --git a/pep-0393.txt b/pep-0393.txt
--- a/pep-0393.txt
+++ b/pep-0393.txt
@@ -118,7 +118,7 @@
   (implies ready)
 - ascii: the object uses the PyASCIIObject representation
   (implies compact and ready)
-- ready: the canonical represenation is ready to be accessed through
+- ready: the canonical representation is ready to be accessed through
   PyUnicode_DATA and PyUnicode_GET_LENGTH. This is set either if the
   object is compact, or the data pointer and length have been
   initialized.
@@ -160,7 +160,7 @@
 In particular, codecs using this API must compute both the number of
 characters and the maximum character in advance. An string is
 allocated according to the specified size and character range and is
-null-terminated; the actual characters in it may be unitialized.
+null-terminated; the actual characters in it may be uninitialized.
 
 PyUnicode_FromString and PyUnicode_FromStringAndSize remain supported
 for processing UTF-8 input; the input is decoded, and the UTF-8
@@ -188,7 +188,7 @@
 PyUnicode_2BYTE_KIND (2), or PyUnicode_4BYTE_KIND (3). PyUnicode_DATA
 gives the void pointer to the data. Access to individual characters
 should use PyUnicode_{READ|WRITE}[_CHAR]:
- 
+
  - PyUnciode_READ(kind, data, index)
  - PyUnicode_WRITE(kind, data, index, value)
  - PyUnicode_READ_CHAR(unicode, index)
@@ -240,7 +240,7 @@
 
 Character access utility functions:
 
-- PyUnicode_GetLength(o), PyUnicode_ReadChar(o, index), 
+- PyUnicode_GetLength(o), PyUnicode_ReadChar(o, index),
   PyUnicode_WriteChar(o, index, character)
 - PyUnicode_CopyCharacters(to, to_start, from, from_start, how_many)
 - PyUnicode_FindChar(str, ch, start, end, direction)
@@ -296,7 +296,7 @@
   PyUnicode_EncodeUnicodeEscape, PyUnicode_EncodeRawUnicodeEscape,
   PyUnicode_EncodeLatin1, PyUnicode_EncodeASCII,
   PyUnicode_EncodeCharmap, PyUnicode_TranslateCharmap,
-  PyUnicode_EncodeMBCS, PyUnicode_EncodeDecimal, 
+  PyUnicode_EncodeMBCS, PyUnicode_EncodeDecimal,
   PyUnicode_TransformDecimalToASCII
 - Py_UNICODE_{strlen, strcat, strcpy, strcmp, strchr, strrchr}
 - PyUnicode_AsUnicodeCopy
@@ -314,7 +314,7 @@
 pointer. Such code will break with this PEP. The code was already
 flawed in 3.2, as there is was no explicit guarantee that the
 PyUnicode_AS_UNICODE result would stay valid after an API call (due to
-the possiblity of string resizing). Modules that face this issue
+the possibility of string resizing). Modules that face this issue
 need to re-fetch the Py_UNICODE pointer after API calls; doing
 so will continue to work correctly in earlier Python versions.
 
@@ -395,7 +395,7 @@
 indexing works well for PyUnicode_READ(_CHAR) and PyUnicode_WRITE. Use
 void* as the buffer type for characters to let the compiler detect
 invalid dereferencing operations. If you do want to use pointer
-arithmentics (e.g. when converting existing code), use (unsigned)
+arithmetics (e.g. when converting existing code), use (unsigned)
 char* as the buffer type, and keep the element size (1, 2, or 4) in a
 variable. Notice that (1<<(kind-1)) will produce the element size
 given a buffer kind.
@@ -428,7 +428,7 @@
 For common tasks, direct access to the string representation may not
 be necessary: PyUnicode_Find, PyUnicode_FindChar, PyUnicode_Ord, and
 PyUnicode_CopyCharacters help in analyzing and creating string
-objects, operating on indices instead of data pointers.
+objects, operating on indexes instead of data pointers.
 
 References
 ==========

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Thu Sep 29 04:03:19 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 04:03:19 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_=5Fsre=3A_don=27t_use_Py=5F?=
	=?utf8?q?UNICODE_anymore?=
Message-ID: <E1R95y3-0002oK-Ek@dinsdale.python.org>

http://hg.python.org/cpython/rev/e4a80a1a9e18
changeset:   72517:e4a80a1a9e18
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 03:27:47 2011 +0200
summary:
  _sre: don't use Py_UNICODE anymore

 * Downcasting from Py_UCS4 to Py_UNICODE is wrong is Py_UNICODE is 16-bit
   wchar_t
 * Remove old special case in getstring(), unicode is now handled separetely

files:
  Modules/_sre.c |  16 +++++++---------
  1 files changed, 7 insertions(+), 9 deletions(-)


diff --git a/Modules/_sre.c b/Modules/_sre.c
--- a/Modules/_sre.c
+++ b/Modules/_sre.c
@@ -163,15 +163,15 @@
 
 /* unicode-specific character predicates */
 
-#define SRE_UNI_IS_DIGIT(ch) Py_UNICODE_ISDECIMAL((Py_UNICODE)(ch))
-#define SRE_UNI_IS_SPACE(ch) Py_UNICODE_ISSPACE((Py_UNICODE)(ch))
-#define SRE_UNI_IS_LINEBREAK(ch) Py_UNICODE_ISLINEBREAK((Py_UNICODE)(ch))
-#define SRE_UNI_IS_ALNUM(ch) Py_UNICODE_ISALNUM((Py_UNICODE)(ch))
-#define SRE_UNI_IS_WORD(ch) (SRE_UNI_IS_ALNUM((ch)) || (ch) == '_')
+#define SRE_UNI_IS_DIGIT(ch) Py_UNICODE_ISDECIMAL(ch)
+#define SRE_UNI_IS_SPACE(ch) Py_UNICODE_ISSPACE(ch)
+#define SRE_UNI_IS_LINEBREAK(ch) Py_UNICODE_ISLINEBREAK(ch)
+#define SRE_UNI_IS_ALNUM(ch) Py_UNICODE_ISALNUM(ch)
+#define SRE_UNI_IS_WORD(ch) (SRE_UNI_IS_ALNUM(ch) || (ch) == '_')
 
 static unsigned int sre_lower_unicode(unsigned int ch)
 {
-    return (unsigned int) Py_UNICODE_TOLOWER((Py_UNICODE)(ch));
+    return (unsigned int) Py_UNICODE_TOLOWER(ch);
 }
 
 LOCAL(int)
@@ -1674,7 +1674,7 @@
         return ptr;
     }
 
-    /* get pointer to string buffer */
+    /* get pointer to byte string buffer */
     view.len = -1;
     buffer = Py_TYPE(string)->tp_as_buffer;
     if (!buffer || !buffer->bf_getbuffer ||
@@ -1702,8 +1702,6 @@
 
     if (PyBytes_Check(string) || bytes == size)
         charsize = 1;
-    else if (bytes == (Py_ssize_t) (size * sizeof(Py_UNICODE)))
-        charsize = sizeof(Py_UNICODE);
     else {
         PyErr_SetString(PyExc_TypeError, "buffer size mismatch");
         return NULL;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 04:03:20 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 04:03:20 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_=5Fio=2Etextio=3A_fix_chara?=
 =?utf8?q?cter_type=2C_use_Py=5FUCS4_instead_of_Py=5FUNICODE?=
Message-ID: <E1R95y4-0002oU-6x@dinsdale.python.org>

http://hg.python.org/cpython/rev/8f4b162f9f0e
changeset:   72518:8f4b162f9f0e
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 03:28:17 2011 +0200
summary:
  _io.textio: fix character type, use Py_UCS4 instead of Py_UNICODE

files:
  Modules/_io/textio.c |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c
--- a/Modules/_io/textio.c
+++ b/Modules/_io/textio.c
@@ -347,7 +347,7 @@
                 memchr(in_str, '\n', PyUnicode_KIND_SIZE(kind, len)) != NULL) {
                 Py_ssize_t i = 0;
                 for (;;) {
-                    Py_UNICODE c;
+                    Py_UCS4 c;
                     /* Fast loop for non-control characters */
                     while (PyUnicode_READ(kind, in_str, i) > '\n')
                         i++;
@@ -1570,7 +1570,7 @@
 }
 
 
-/* NOTE: `end` must point to the real end of the Py_UNICODE storage,
+/* NOTE: `end` must point to the real end of the Py_UCS4 storage,
    that is to the NUL character. Otherwise the function will produce
    incorrect results. */
 static char *
@@ -1614,7 +1614,7 @@
         for (;;) {
             Py_UCS4 ch;
             /* Fast path for non-control chars. The loop always ends
-               since the Py_UNICODE storage is NUL-terminated. */
+               since the Unicode string is NUL-terminated. */
             while (PyUnicode_READ(kind, s, 0) > '\r')
                 s += size;
             if (s >= end) {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 04:03:20 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 04:03:20 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_modsupport=2Ec_reuses_Py=5F?=
	=?utf8?q?UNICODE=5Fstrlen=28=29?=
Message-ID: <E1R95y4-0002ob-Ur@dinsdale.python.org>

http://hg.python.org/cpython/rev/204d9391889a
changeset:   72519:204d9391889a
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 04:01:43 2011 +0200
summary:
  modsupport.c reuses Py_UNICODE_strlen()

files:
  Python/modsupport.c |  11 +----------
  1 files changed, 1 insertions(+), 10 deletions(-)


diff --git a/Python/modsupport.c b/Python/modsupport.c
--- a/Python/modsupport.c
+++ b/Python/modsupport.c
@@ -148,15 +148,6 @@
     return v;
 }
 
-static int
-_ustrlen(Py_UNICODE *u)
-{
-    int i = 0;
-    Py_UNICODE *v = u;
-    while (*v != 0) { i++; v++; }
-    return i;
-}
-
 static PyObject *
 do_mktuple(const char **p_format, va_list *p_va, int endchar, int n, int flags)
 {
@@ -269,7 +260,7 @@
             }
             else {
                 if (n < 0)
-                    n = _ustrlen(u);
+                    n = Py_UNICODE_strlen(u);
                 v = PyUnicode_FromUnicode(u, n);
             }
             return v;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 04:03:21 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 04:03:21 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_hex=5Fdigit=5Fto=5Fint?=
 =?utf8?q?=28=29_prototype=3A_expect_Py=5FUCS4=2C_not_Py=5FUNICODE?=
Message-ID: <E1R95y5-0002of-N4@dinsdale.python.org>

http://hg.python.org/cpython/rev/e6a2b54c1d16
changeset:   72520:e6a2b54c1d16
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 04:02:13 2011 +0200
summary:
  Fix hex_digit_to_int() prototype: expect Py_UCS4, not Py_UNICODE

files:
  Objects/bytearrayobject.c |  4 ++--
  Objects/bytesobject.c     |  4 ++--
  2 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c
--- a/Objects/bytearrayobject.c
+++ b/Objects/bytearrayobject.c
@@ -1147,7 +1147,7 @@
 \n\
 Remove all items from B.");
 
-static PyObject * 
+static PyObject *
 bytearray_clear(PyByteArrayObject *self)
 {
     if (PyByteArray_Resize((PyObject *)self, 0) < 0)
@@ -2629,7 +2629,7 @@
 Example: bytearray.fromhex('B9 01EF') -> bytearray(b'\\xb9\\x01\\xef').");
 
 static int
-hex_digit_to_int(Py_UNICODE c)
+hex_digit_to_int(Py_UCS4 c)
 {
     if (c >= 128)
         return -1;
diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -593,7 +593,7 @@
         quote = '"';
     if (squotes && quote == '\'')
         newsize += squotes;
-    
+
     if (newsize > (PY_SSIZE_T_MAX - sizeof(PyUnicodeObject) - 1)) {
         PyErr_SetString(PyExc_OverflowError,
             "bytes object is too large to make repr");
@@ -2330,7 +2330,7 @@
 Example: bytes.fromhex('B9 01EF') -> b'\\xb9\\x01\\xef'.");
 
 static int
-hex_digit_to_int(Py_UNICODE c)
+hex_digit_to_int(Py_UCS4 c)
 {
     if (c >= 128)
         return -1;

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Thu Sep 29 05:25:14 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Thu, 29 Sep 2011 05:25:14 +0200
Subject: [Python-checkins] Daily reference leaks (991ef9253263): sum=0
Message-ID: <E1R97FK-0000iQ-VA@ap.vmr.nerim.net>

results for 991ef9253263 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogIi1ufB', '-x']

From python-checkins at python.org  Thu Sep 29 07:36:34 2011
From: python-checkins at python.org (ezio.melotti)
Date: Thu, 29 Sep 2011 07:36:34 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Update_and_reorganize_the_w?=
 =?utf8?q?hatsnew_entry_for_PEP_393=2E?=
Message-ID: <E1R99IQ-0005LN-LK@dinsdale.python.org>

http://hg.python.org/cpython/rev/ba6ee5cc9ed6
changeset:   72521:ba6ee5cc9ed6
parent:      72516:90fef68f06e7
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Sep 29 08:34:36 2011 +0300
summary:
  Update and reorganize the whatsnew entry for PEP 393.

files:
  Doc/whatsnew/3.3.rst |  63 +++++++++++++++++++++----------
  1 files changed, 42 insertions(+), 21 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -58,35 +58,56 @@
 PEP 393: Flexible String Representation
 =======================================
 
+XXX Give a short introduction about :pep:`393`.
+
+PEP 393 is fully backward compatible. The legacy API should remain
+available at least five years. Applications using the legacy API will not
+fully benefit of the memory reduction, or worse may use a little bit more
+memory, because Python may have to maintain two versions of each string (in
+the legacy format and in the new efficient storage).
+
 XXX Add list of changes introduced by :pep:`393` here:
 
+* Python now always supports the full range of Unicode codepoints, including
+  non-BMP ones (i.e. from ``U+0000`` to ``U+10FFFF``).  The distinction between
+  narrow and wide builds no longer exists and Python now behaves like a wide
+  build.
+
+* The storage of Unicode strings now depends on the highest codepoint in the string:
+
+  * pure ASCII and Latin1 strings (``U+0000-U+00FF``) use 1 byte per codepoint;
+
+  * BMP strings (``U+0000-U+FFFF``) use 2 bytes per codepoint;
+
+  * non-BMP strings (``U+10000-U+10FFFF``) use 4 bytes per codepoint.
+
+.. The memory usage of Python 3.3 is two to three times smaller than Python 3.2,
+   and a little bit better than Python 2.7, on a `Django benchmark
+   <http://mail.python.org/pipermail/python-dev/2011-September/113714.html>`_.
+   XXX The result should be moved in the PEP and a small summary about
+   performances and a link to the PEP should be added here.
+
+* Some of the problems visible on narrow builds have been fixed, for example:
+
+  * :func:`len` now always returns 1 for non-BMP characters,
+    so ``len('\U0010FFFF') == 1``;
+
+  * surrogate pairs are not recombined in string literals,
+    so ``'\uDBFF\uDFFF' != '\U0010FFFF'``;
+
+  * indexing or slicing a non-BMP characters doesn't return surrogates anymore,
+    so ``'\U0010FFFF'[0]`` now returns ``'\U0010FFFF'`` and not ``'\uDBFF'``;
+
+  * several other functions in the stdlib now handle correctly non-BMP codepoints.
+
 * The value of :data:`sys.maxunicode` is now always ``1114111`` (``0x10FFFF``
   in hexadecimal).  The :c:func:`PyUnicode_GetMax` function still returns
   either ``0xFFFF`` or ``0x10FFFF`` for backward compatibility, and it should
   not be used with the new Unicode API (see :issue:`13054`).
 
-* Non-BMP characters (U+10000-U+10FFFF range) are no more special cases.
-  ``'\U0010FFFF'[0]`` is now ``'\U0010FFFF'`` on any platform, instead of
-  ``'\uDFFF'`` on narrow build or ``'\U0010FFFF'`` on wide build. And
-  ``len('\U0010FFFF')`` is now ``1`` on any platform, instead of ``2`` on
-  narrow build or ``1`` on wide build. More generally, most bugs related to
-  non-BMP characters are now fixed. For example, :func:`unicodedata.normalize`
-  handles correctly non-BMP characters on all platforms.
+* The :file:`./configure` flag ``--with-wide-unicode`` has been removed.
 
-* The storage of Unicode string is now adapted on the content of the string.
-  Pure ASCII and Latin1 strings (U+0000-U+00FF) use 1 byte per character, BMP
-  strings (U+0000-U+FFFF) use 2 bytes per character, and non-BMP characters
-  (U+10000-U+10FFFF range) use 4 bytes per characters. The memory usage of
-  Python 3.3 is two to three times smaller than Python 3.2, and a little bit
-  better than Python 2.7, on a `Django benchmark
-  <http://mail.python.org/pipermail/python-dev/2011-September/113714.html>`_.
-
-* The PEP 393 is fully backward compatible. The legacy API should remain
-  available at least five years. Applications using the legacy API will not
-  fully benefit of the memory reduction, or worse may use a little bit more
-  memory, because Python may have to maintain two versions of each string (in
-  the legacy format and in the new efficient storage).
-
+XXX mention new and deprecated functions and macros
 
 Other Language Changes
 ======================

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 07:36:35 2011
From: python-checkins at python.org (ezio.melotti)
Date: Thu, 29 Sep 2011 07:36:35 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?b?KTogTWVyZ2UgaGVhZHMu?=
Message-ID: <E1R99IR-0005LR-DR@dinsdale.python.org>

http://hg.python.org/cpython/rev/e7672fe3cd35
changeset:   72522:e7672fe3cd35
parent:      72520:e6a2b54c1d16
parent:      72521:ba6ee5cc9ed6
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Sep 29 08:36:23 2011 +0300
summary:
  Merge heads.

files:
  Doc/whatsnew/3.3.rst |  63 +++++++++++++++++++++----------
  1 files changed, 42 insertions(+), 21 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -58,35 +58,56 @@
 PEP 393: Flexible String Representation
 =======================================
 
+XXX Give a short introduction about :pep:`393`.
+
+PEP 393 is fully backward compatible. The legacy API should remain
+available at least five years. Applications using the legacy API will not
+fully benefit of the memory reduction, or worse may use a little bit more
+memory, because Python may have to maintain two versions of each string (in
+the legacy format and in the new efficient storage).
+
 XXX Add list of changes introduced by :pep:`393` here:
 
+* Python now always supports the full range of Unicode codepoints, including
+  non-BMP ones (i.e. from ``U+0000`` to ``U+10FFFF``).  The distinction between
+  narrow and wide builds no longer exists and Python now behaves like a wide
+  build.
+
+* The storage of Unicode strings now depends on the highest codepoint in the string:
+
+  * pure ASCII and Latin1 strings (``U+0000-U+00FF``) use 1 byte per codepoint;
+
+  * BMP strings (``U+0000-U+FFFF``) use 2 bytes per codepoint;
+
+  * non-BMP strings (``U+10000-U+10FFFF``) use 4 bytes per codepoint.
+
+.. The memory usage of Python 3.3 is two to three times smaller than Python 3.2,
+   and a little bit better than Python 2.7, on a `Django benchmark
+   <http://mail.python.org/pipermail/python-dev/2011-September/113714.html>`_.
+   XXX The result should be moved in the PEP and a small summary about
+   performances and a link to the PEP should be added here.
+
+* Some of the problems visible on narrow builds have been fixed, for example:
+
+  * :func:`len` now always returns 1 for non-BMP characters,
+    so ``len('\U0010FFFF') == 1``;
+
+  * surrogate pairs are not recombined in string literals,
+    so ``'\uDBFF\uDFFF' != '\U0010FFFF'``;
+
+  * indexing or slicing a non-BMP characters doesn't return surrogates anymore,
+    so ``'\U0010FFFF'[0]`` now returns ``'\U0010FFFF'`` and not ``'\uDBFF'``;
+
+  * several other functions in the stdlib now handle correctly non-BMP codepoints.
+
 * The value of :data:`sys.maxunicode` is now always ``1114111`` (``0x10FFFF``
   in hexadecimal).  The :c:func:`PyUnicode_GetMax` function still returns
   either ``0xFFFF`` or ``0x10FFFF`` for backward compatibility, and it should
   not be used with the new Unicode API (see :issue:`13054`).
 
-* Non-BMP characters (U+10000-U+10FFFF range) are no more special cases.
-  ``'\U0010FFFF'[0]`` is now ``'\U0010FFFF'`` on any platform, instead of
-  ``'\uDFFF'`` on narrow build or ``'\U0010FFFF'`` on wide build. And
-  ``len('\U0010FFFF')`` is now ``1`` on any platform, instead of ``2`` on
-  narrow build or ``1`` on wide build. More generally, most bugs related to
-  non-BMP characters are now fixed. For example, :func:`unicodedata.normalize`
-  handles correctly non-BMP characters on all platforms.
+* The :file:`./configure` flag ``--with-wide-unicode`` has been removed.
 
-* The storage of Unicode string is now adapted on the content of the string.
-  Pure ASCII and Latin1 strings (U+0000-U+00FF) use 1 byte per character, BMP
-  strings (U+0000-U+FFFF) use 2 bytes per character, and non-BMP characters
-  (U+10000-U+10FFFF range) use 4 bytes per characters. The memory usage of
-  Python 3.3 is two to three times smaller than Python 3.2, and a little bit
-  better than Python 2.7, on a `Django benchmark
-  <http://mail.python.org/pipermail/python-dev/2011-September/113714.html>`_.
-
-* The PEP 393 is fully backward compatible. The legacy API should remain
-  available at least five years. Applications using the legacy API will not
-  fully benefit of the memory reduction, or worse may use a little bit more
-  memory, because Python may have to maintain two versions of each string (in
-  the legacy format and in the new efficient storage).
-
+XXX mention new and deprecated functions and macros
 
 Other Language Changes
 ======================

-- 
Repository URL: http://hg.python.org/cpython

From ezio.melotti at gmail.com  Thu Sep 29 09:54:37 2011
From: ezio.melotti at gmail.com (Ezio Melotti)
Date: Thu, 29 Sep 2011 10:54:37 +0300
Subject: [Python-checkins] Hg tips (was Re: cpython (merge default ->
 default): Merge heads.)
Message-ID: <CACBhJdFH8jaiEakk8FX94+W3qwiSLkyn4_U3U9_gYCde9wmyHg@mail.gmail.com>

Tip 1 -- merging heads:

A while ago ?ric suggested a nice tip to make merges easier and since I
haven't seen many people using it and now I got a chance to use it again, I
think it might be worth showing it once more:

# so assume you just committed some changes:
$ hg ci Doc/whatsnew/3.3.rst -m 'Update and reorganize the whatsnew entry
for PEP 393.'
# you push them, but someone else pushed something in the meanwhile, so the
push fails
$ hg push
pushing to ssh://hg at hg.python.org/cpython
searching for changes
abort: push creates new remote heads on branch 'default'!
(you should pull and merge or use push -f to force)
# so you pull the other changes
$ hg pull -u
pulling from ssh://hg at hg.python.org/cpython
searching for changes
adding changesets
adding manifests
adding file changes
added 4 changesets with 5 changes to 5 files (+1 heads)
not updating, since new heads added
(run 'hg heads' to see heads, 'hg merge' to merge)
# and use "hg heads ." to see the two heads (yours and the one you pulled)
in the current branch
$ hg heads .
changeset:   72521:e6a2b54c1d16
tag:         tip
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 04:02:13 2011 +0200
summary:     Fix hex_digit_to_int() prototype: expect Py_UCS4, not
Py_UNICODE

changeset:   72517:ba6ee5cc9ed6
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Sep 29 08:34:36 2011 +0300
summary:     Update and reorganize the whatsnew entry for PEP 393.
# here comes the tip: before merging you switch to the other head (i.e. the
one pushed by Victor),
# if you don't switch, you'll be merging Victor changeset and in case of
conflicts you will have to review
# and modify his code (e.g. put a Misc/NEWS entry in the right section or
something more complicated)
$ hg up e6a2b54c1d16
6 files updated, 0 files merged, 0 files removed, 0 files unresolved
# after the switch you will merge the changeset you just committed, so in
case of conflicts
# reviewing and merging is much easier because you know the changes already
$ hg merge
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
# here everything went fine and there were no conflicts, and in the diff I
can see my last changeset
$ hg di
diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
[...]
# everything looks fine, so I can commit the merge and push
$ hg ci -m 'Merge heads.'
$ hg push
pushing to ssh://hg at hg.python.org/cpython
searching for changes
remote: adding
changesets

remote: adding manifests
remote: adding file changes
remote: added 2 changesets with 1 changes to 1 files
remote: buildbot: 2 changes sent successfully
remote: notified python-checkins at python.org of incoming changeset
ba6ee5cc9ed6
remote: notified python-checkins at python.org of incoming changeset
e7672fe3cd35

This tip is not only useful while merging, but it's also useful for
python-checkins reviews, because the "merge" mail has the same diff  of the
previous mail rather than having 15 unrelated changesets from the last week
because the committer didn't pull in a while.


Tip 2 -- extended diffs:

If you haven't already, enable git diffs, adding to your ~/.hgrc the
following two lines:

> [diff]
> git = True
>
(this is already in the devguide, even if 'git = on' is used there. The
mercurial website uses git = True too.)
More info:  http://hgtip.com/tips/beginner/2009-10-22-always-use-git-diffs/


Tip 3 -- extensions:

I personally like the 'color' extension, it makes the output of commands
like 'hg diff' and 'hg stat' more readable (e.g. it shows removed lines in
red and added ones in green).
If you want to give it a try, add to your ~/.hgrc the following two lines:

> [extensions]
> color =
>

If you find operations like pulling, updating or cloning too slow, you might
also want to look at the 'progress' extension, which displays a progress bar
during these operations:

> [extensions]
> progress =
>


Tip 4 -- porting from 2.7 to 3.2:

The devguide suggests:
>
> hg export a7df1a869e4a | hg import --no-commit -
>
but it's not always necessary to copy the changeset number manually.
If you are porting your last commit you can just use 'hg export 2.7' (or any
other branch name):
* using the one-dir-per-branch setup:
  wolf at hp:~/dev/py/2.7$ hg ci -m 'Fix some bug.'
  wolf at hp:~/dev/py/2.7$ cd ../3.2
  wolf at hp:~/dev/py/3.2$ hg pull -u ../2.7
  wolf at hp:~/dev/py/3.2$ hg export 2.7 | hg import --no-commit -
* using the single-dir setup:
  wolf at hp:~/dev/python$ hg branch
  2.7
  wolf at hp:~/dev/python$ hg ci -m 'Fix some bug.'
  wolf at hp:~/dev/python$ hg up 3.2  # here you might enjoy the progress
extension
  wolf at hp:~/dev/python$ hg export 2.7 | hg import --no-commit -
And then you can check that everything is fine, and commit on 3.2 too.
Of course it works the other way around (from 3.2 to 2.7) too.


I hope you'll find these tips useful.

Best Regards,
Ezio Melotti


On Thu, Sep 29, 2011 at 8:36 AM, ezio.melotti <python-checkins at python.org>wrote:

> http://hg.python.org/cpython/rev/e7672fe3cd35
> changeset:   72522:e7672fe3cd35
> parent:      72520:e6a2b54c1d16
> parent:      72521:ba6ee5cc9ed6
> user:        Ezio Melotti <ezio.melotti at gmail.com>
> date:        Thu Sep 29 08:36:23 2011 +0300
> summary:
>  Merge heads.
>
> files:
>  Doc/whatsnew/3.3.rst |  63 +++++++++++++++++++++----------
>  1 files changed, 42 insertions(+), 21 deletions(-)
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-checkins/attachments/20110929/e0e02c8b/attachment.html>

From python-checkins at python.org  Thu Sep 29 12:13:24 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 12:13:24 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_pymacro=2Eh=3A_Inline_=5FPy?=
 =?utf8?q?=5FARRAY=5FLENGTH=5FCHECK=28=29_and_add_http=3A//ccodearchive=2E?=
 =?utf8?q?net/?=
Message-ID: <E1R9DcK-0004Z4-U7@dinsdale.python.org>

http://hg.python.org/cpython/rev/d2bac6e41f71
changeset:   72523:d2bac6e41f71
user:        Victor Stinner <vstinner at wyplay.com>
date:        Thu Sep 29 12:12:39 2011 +0200
summary:
  pymacro.h: Inline _Py_ARRAY_LENGTH_CHECK() and add http://ccodearchive.net/

files:
  Include/pymacro.h |  25 ++++++++++++-------------
  1 files changed, 12 insertions(+), 13 deletions(-)


diff --git a/Include/pymacro.h b/Include/pymacro.h
--- a/Include/pymacro.h
+++ b/Include/pymacro.h
@@ -19,20 +19,10 @@
        ((char *)(foo)        \
         + Py_BUILD_ASSERT(offsetof(struct foo, string) == 0))
 
-   Written by Rusty Russell, public domain. */
+   Written by Rusty Russell, public domain, http://ccodearchive.net/ */
 #define Py_BUILD_ASSERT(cond) \
     (sizeof(char [1 - 2*!(cond)]) - 1)
 
-#if defined(__GNUC__)
-/* Two gcc extensions.
-   &a[0] degrades to a pointer: a different type from an array */
-#define _Py_ARRAY_LENGTH_CHECK(array) \
-    Py_BUILD_ASSERT(!__builtin_types_compatible_p(typeof(array), \
-                                                  typeof(&(array)[0])))
-#else
-#define _Py_ARRAY_LENGTH_CHECK(array) 0
-#endif
-
 
 /* Get the number of elements in a visible array
 
@@ -40,9 +30,18 @@
    parameters. With correct compiler support, such usage will cause a build
    error (see Py_BUILD_ASSERT).
 
-   Written by Rusty Russell, public domain. */
+   Written by Rusty Russell, public domain, http://ccodearchive.net/ */
+#if defined(__GNUC__)
+/* Two gcc extensions.
+   &a[0] degrades to a pointer: a different type from an array */
 #define Py_ARRAY_LENGTH(array) \
-    (sizeof(array) / sizeof((array)[0]) + _Py_ARRAY_LENGTH_CHECK(array))
+    (sizeof(array) / sizeof((array)[0]) \
+     + Py_BUILD_ASSERT(!__builtin_types_compatible_p(typeof(array), \
+                                                     typeof(&(array)[0]))))
+#else
+#define Py_ARRAY_LENGTH(array) \
+    (sizeof(array) / sizeof((array)[0]))
+#endif
 
 
 /* Define macros for inline documentation. */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 12:44:29 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 12:44:29 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Rename_Py=5FBUILD=5FASSERT_?=
 =?utf8?q?to_Py=5FBUILD=5FASSERT=5FEXPR?=
Message-ID: <E1R9E6P-0007NS-54@dinsdale.python.org>

http://hg.python.org/cpython/rev/60b76397080d
changeset:   72524:60b76397080d
user:        Victor Stinner <vstinner at wyplay.com>
date:        Thu Sep 29 12:43:18 2011 +0200
summary:
  Rename Py_BUILD_ASSERT to Py_BUILD_ASSERT_EXPR

To make it clearer that Py_BUILD_ASSERT_EXPR(cond) cannot be used as
assert(cond).

files:
  Include/pymacro.h |  11 +++++------
  1 files changed, 5 insertions(+), 6 deletions(-)


diff --git a/Include/pymacro.h b/Include/pymacro.h
--- a/Include/pymacro.h
+++ b/Include/pymacro.h
@@ -17,18 +17,17 @@
 
    #define foo_to_char(foo)  \
        ((char *)(foo)        \
-        + Py_BUILD_ASSERT(offsetof(struct foo, string) == 0))
+        + Py_BUILD_ASSERT_EXPR(offsetof(struct foo, string) == 0))
 
    Written by Rusty Russell, public domain, http://ccodearchive.net/ */
-#define Py_BUILD_ASSERT(cond) \
+#define Py_BUILD_ASSERT_EXPR(cond) \
     (sizeof(char [1 - 2*!(cond)]) - 1)
 
-
 /* Get the number of elements in a visible array
 
    This does not work on pointers, or arrays declared as [], or function
    parameters. With correct compiler support, such usage will cause a build
-   error (see Py_BUILD_ASSERT).
+   error (see Py_BUILD_ASSERT_EXPR).
 
    Written by Rusty Russell, public domain, http://ccodearchive.net/ */
 #if defined(__GNUC__)
@@ -36,8 +35,8 @@
    &a[0] degrades to a pointer: a different type from an array */
 #define Py_ARRAY_LENGTH(array) \
     (sizeof(array) / sizeof((array)[0]) \
-     + Py_BUILD_ASSERT(!__builtin_types_compatible_p(typeof(array), \
-                                                     typeof(&(array)[0]))))
+     + Py_BUILD_ASSERT_EXPR(!__builtin_types_compatible_p(typeof(array), \
+                                                          typeof(&(array)[0]))))
 #else
 #define Py_ARRAY_LENGTH(array) \
     (sizeof(array) / sizeof((array)[0]))

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 13:40:13 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Thu, 29 Sep 2011 13:40:13 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Port_normalization_to_new_A?=
	=?utf8?b?UEku?=
Message-ID: <E1R9EyL-0003y5-HH@dinsdale.python.org>

http://hg.python.org/cpython/rev/40cf7204ca2b
changeset:   72525:40cf7204ca2b
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Thu Sep 29 13:39:38 2011 +0200
summary:
  Port normalization to new API.

files:
  Modules/unicodedata.c |  188 ++++++++++++++++++-----------
  1 files changed, 118 insertions(+), 70 deletions(-)


diff --git a/Modules/unicodedata.c b/Modules/unicodedata.c
--- a/Modules/unicodedata.c
+++ b/Modules/unicodedata.c
@@ -494,36 +494,44 @@
 nfd_nfkd(PyObject *self, PyObject *input, int k)
 {
     PyObject *result;
-    Py_UNICODE *i, *end, *o;
+    Py_UCS4 *output;
+    Py_ssize_t i, o, osize;
+    int kind;
+    void *data;
     /* Longest decomposition in Unicode 3.2: U+FDFA */
-    Py_UNICODE stack[20];
+    Py_UCS4 stack[20];
     Py_ssize_t space, isize;
     int index, prefix, count, stackptr;
     unsigned char prev, cur;
 
     stackptr = 0;
-    isize = PyUnicode_GET_SIZE(input);
+    isize = PyUnicode_GET_LENGTH(input);
     /* Overallocate atmost 10 characters. */
     space = (isize > 10 ? 10 : isize) + isize;
-    result = PyUnicode_FromUnicode(NULL, space);
-    if (!result)
+    osize = space;
+    output = PyMem_Malloc(space * sizeof(Py_UCS4));
+    if (!output) {
+        PyErr_NoMemory();
         return NULL;
-    i = PyUnicode_AS_UNICODE(input);
-    end = i + isize;
-    o = PyUnicode_AS_UNICODE(result);
+    }
+    i = o = 0;
+    kind = PyUnicode_KIND(input);
+    data = PyUnicode_DATA(input);
 
-    while (i < end) {
-        stack[stackptr++] = *i++;
+    while (i < isize) {
+        stack[stackptr++] = PyUnicode_READ(kind, data, i++);
         while(stackptr) {
-            Py_UNICODE code = stack[--stackptr];
+            Py_UCS4 code = stack[--stackptr];
             /* Hangul Decomposition adds three characters in
                a single step, so we need atleast that much room. */
             if (space < 3) {
-                Py_ssize_t newsize = PyUnicode_GET_SIZE(result) + 10;
+                osize += 10;
                 space += 10;
-                if (PyUnicode_Resize(&result, newsize) == -1)
+                output = PyMem_Realloc(output, osize*sizeof(Py_UCS4));
+                if (output == NULL) {
+                    PyErr_NoMemory();
                     return NULL;
-                o = PyUnicode_AS_UNICODE(result) + newsize - space;
+                }
             }
             /* Hangul Decomposition. */
             if (SBase <= code && code < (SBase+SCount)) {
@@ -531,11 +539,11 @@
                 int L = LBase + SIndex / NCount;
                 int V = VBase + (SIndex % NCount) / TCount;
                 int T = TBase + SIndex % TCount;
-                *o++ = L;
-                *o++ = V;
+                output[o++] = L;
+                output[o++] = V;
                 space -= 2;
                 if (T != TBase) {
-                    *o++ = T;
+                    output[o++] = T;
                     space --;
                 }
                 continue;
@@ -555,7 +563,7 @@
             /* Copy character if it is not decomposable, or has a
                compatibility decomposition, but we do NFD. */
             if (!count || (prefix && !k)) {
-                *o++ = code;
+                output[o++] = code;
                 space--;
                 continue;
             }
@@ -568,15 +576,20 @@
         }
     }
 
-    /* Drop overallocation. Cannot fail. */
-    PyUnicode_Resize(&result, PyUnicode_GET_SIZE(result) - space);
+    result = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                       output, o);
+    PyMem_Free(output);
+    if (!result)
+        return NULL;
+    /* result is guaranteed to be ready, as it is compact. */
+    kind = PyUnicode_KIND(result);
+    data = PyUnicode_DATA(result);
 
     /* Sort canonically. */
-    i = PyUnicode_AS_UNICODE(result);
-    prev = _getrecord_ex(*i)->combining;
-    end = i + PyUnicode_GET_SIZE(result);
-    for (i++; i < end; i++) {
-        cur = _getrecord_ex(*i)->combining;
+    i = 0;
+    prev = _getrecord_ex(PyUnicode_READ(kind, data, i))->combining;
+    for (i++; i < PyUnicode_GET_LENGTH(result); i++) {
+        cur = _getrecord_ex(PyUnicode_READ(kind, data, i))->combining;
         if (prev == 0 || cur == 0 || prev <= cur) {
             prev = cur;
             continue;
@@ -584,23 +597,24 @@
         /* Non-canonical order. Need to switch *i with previous. */
         o = i - 1;
         while (1) {
-            Py_UNICODE tmp = o[1];
-            o[1] = o[0];
-            o[0] = tmp;
+            Py_UCS4 tmp = PyUnicode_READ(kind, data, o+1);
+            PyUnicode_WRITE(kind, data, o+1,
+                            PyUnicode_READ(kind, data, o));
+            PyUnicode_WRITE(kind, data, o, tmp);
             o--;
-            if (o < PyUnicode_AS_UNICODE(result))
+            if (o < 0)
                 break;
-            prev = _getrecord_ex(*o)->combining;
+            prev = _getrecord_ex(PyUnicode_READ(kind, data, o))->combining;
             if (prev == 0 || prev <= cur)
                 break;
         }
-        prev = _getrecord_ex(*i)->combining;
+        prev = _getrecord_ex(PyUnicode_READ(kind, data, i))->combining;
     }
     return result;
 }
 
 static int
-find_nfc_index(PyObject *self, struct reindex* nfc, Py_UNICODE code)
+find_nfc_index(PyObject *self, struct reindex* nfc, Py_UCS4 code)
 {
     int index;
     for (index = 0; nfc[index].start; index++) {
@@ -619,27 +633,36 @@
 nfc_nfkc(PyObject *self, PyObject *input, int k)
 {
     PyObject *result;
-    Py_UNICODE *i, *i1, *o, *end;
+    int kind;
+    void *data;
+    Py_UCS4 *output;
+    Py_ssize_t i, i1, o, len;
     int f,l,index,index1,comb;
-    Py_UNICODE code;
-    Py_UNICODE *skipped[20];
+    Py_UCS4 code;
+    Py_ssize_t skipped[20];
     int cskipped = 0;
 
     result = nfd_nfkd(self, input, k);
     if (!result)
         return NULL;
+    /* result will be "ready". */
+    kind = PyUnicode_KIND(result);
+    data = PyUnicode_DATA(result);
+    len = PyUnicode_GET_LENGTH(result);
 
-    /* We are going to modify result in-place.
-       If nfd_nfkd is changed to sometimes return the input,
-       this code needs to be reviewed. */
-    assert(result != input);
-
-    i = PyUnicode_AS_UNICODE(result);
-    end = i + PyUnicode_GET_SIZE(result);
-    o = PyUnicode_AS_UNICODE(result);
+    /* We allocate a buffer for the output.
+       If we find that we made no changes, we still return
+       the NFD result. */
+    output = PyMem_Malloc(len * sizeof(Py_UCS4));
+    if (!output) {
+        PyErr_NoMemory();
+        Py_DECREF(result);
+        return 0;
+    }
+    i = o = 0;
 
   again:
-    while (i < end) {
+    while (i < len) {
       for (index = 0; index < cskipped; index++) {
           if (skipped[index] == i) {
               /* *i character is skipped.
@@ -652,33 +675,41 @@
       }
       /* Hangul Composition. We don't need to check for <LV,T>
          pairs, since we always have decomposed data. */
-      if (LBase <= *i && *i < (LBase+LCount) &&
-          i + 1 < end &&
-          VBase <= i[1] && i[1] <= (VBase+VCount)) {
+      code = PyUnicode_READ(kind, data, i);
+      if (LBase <= code && code < (LBase+LCount) &&
+          i + 1 < len &&
+          VBase <= PyUnicode_READ(kind, data, i+1) &&
+          PyUnicode_READ(kind, data, i+1) <= (VBase+VCount)) {
           int LIndex, VIndex;
-          LIndex = i[0] - LBase;
-          VIndex = i[1] - VBase;
+          LIndex = code - LBase;
+          VIndex = PyUnicode_READ(kind, data, i+1) - VBase;
           code = SBase + (LIndex*VCount+VIndex)*TCount;
           i+=2;
-          if (i < end &&
-              TBase <= *i && *i <= (TBase+TCount)) {
-              code += *i-TBase;
+          if (i < len &&
+              TBase <= PyUnicode_READ(kind, data, i) &&
+              PyUnicode_READ(kind, data, i) <= (TBase+TCount)) {
+              code += PyUnicode_READ(kind, data, i)-TBase;
               i++;
           }
-          *o++ = code;
+          output[o++] = code;
           continue;
       }
 
-      f = find_nfc_index(self, nfc_first, *i);
+      /* code is still input[i] here */
+      f = find_nfc_index(self, nfc_first, code);
       if (f == -1) {
-          *o++ = *i++;
+          output[o++] = code;
+          i++;
           continue;
       }
       /* Find next unblocked character. */
       i1 = i+1;
       comb = 0;
-      while (i1 < end) {
-          int comb1 = _getrecord_ex(*i1)->combining;
+      /* output base character for now; might be updated later. */
+      output[o] = PyUnicode_READ(kind, data, i);
+      while (i1 < len) {
+          Py_UCS4 code1 = PyUnicode_READ(kind, data, i1);
+          int comb1 = _getrecord_ex(code1)->combining;
           if (comb) {
               if (comb1 == 0)
                   break;
@@ -688,8 +719,8 @@
                   continue;
               }
           }
-          l = find_nfc_index(self, nfc_last, *i1);
-          /* *i1 cannot be combined with *i. If *i1
+          l = find_nfc_index(self, nfc_last, code1);
+          /* i1 cannot be combined with i. If i1
              is a starter, we don't need to look further.
              Otherwise, record the combining class. */
           if (l == -1) {
@@ -708,19 +739,28 @@
               goto not_combinable;
 
           /* Replace the original character. */
-          *i = code;
+          output[o] = code;
           /* Mark the second character unused. */
           assert(cskipped < 20);
           skipped[cskipped++] = i1;
           i1++;
-          f = find_nfc_index(self, nfc_first, *i);
+          f = find_nfc_index(self, nfc_first, output[o]);
           if (f == -1)
               break;
       }
-      *o++ = *i++;
+      /* Output character was already written.
+         Just advance the indices. */
+      o++; i++;
     }
-    if (o != end)
-        PyUnicode_Resize(&result, o - PyUnicode_AS_UNICODE(result));
+    if (o == len) {
+        /* No changes. Return original string. */
+        PyMem_Free(output);
+        return result;
+    }
+    Py_DECREF(result);
+    result = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                       output, o);
+    PyMem_Free(output);
     return result;
 }
 
@@ -728,7 +768,9 @@
 static int
 is_normalized(PyObject *self, PyObject *input, int nfc, int k)
 {
-    Py_UNICODE *i, *end;
+    Py_ssize_t i, len;
+    int kind;
+    void *data;
     unsigned char prev_combining = 0, quickcheck_mask;
 
     /* An older version of the database is requested, quickchecks must be
@@ -740,10 +782,13 @@
        as described in http://unicode.org/reports/tr15/#Annex8. */
     quickcheck_mask = 3 << ((nfc ? 4 : 0) + (k ? 2 : 0));
 
-    i = PyUnicode_AS_UNICODE(input);
-    end = i + PyUnicode_GET_SIZE(input);
-    while (i < end) {
-        const _PyUnicode_DatabaseRecord *record = _getrecord_ex(*i++);
+    i = 0;
+    kind = PyUnicode_KIND(input);
+    data = PyUnicode_DATA(input);
+    len = PyUnicode_GET_LENGTH(input);
+    while (i < len) {
+        Py_UCS4 ch = PyUnicode_READ(kind, data, i++);
+        const _PyUnicode_DatabaseRecord *record = _getrecord_ex(ch);
         unsigned char combining = record->combining;
         unsigned char quickcheck = record->normalization_quick_check;
 
@@ -772,7 +817,10 @@
                          &form, &PyUnicode_Type, &input))
         return NULL;
 
-    if (PyUnicode_GetSize(input) == 0) {
+    if (PyUnicode_READY(input) == -1)
+        return NULL;
+
+    if (PyUnicode_GET_LENGTH(input) == 0) {
         /* Special case empty input strings, since resizing
            them  later would cause internal errors. */
         Py_INCREF(input);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 13:52:06 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Thu, 29 Sep 2011 13:52:06 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Re-enable_test=2E?=
Message-ID: <E1R9F9q-00052h-Um@dinsdale.python.org>

http://hg.python.org/cpython/rev/50f008019213
changeset:   72526:50f008019213
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Thu Sep 29 13:49:10 2011 +0200
summary:
  Re-enable test.

files:
  Lib/test/test_pep3131.py |  5 +----
  1 files changed, 1 insertions(+), 4 deletions(-)


diff --git a/Lib/test/test_pep3131.py b/Lib/test/test_pep3131.py
--- a/Lib/test/test_pep3131.py
+++ b/Lib/test/test_pep3131.py
@@ -17,10 +17,7 @@
 
     def test_non_bmp_normalized(self):
         ??????? = 1
-        try:
-            self.assertIn("Unicode", dir())
-        except AssertionError:
-            raise unittest._ExpectedFailure("doesn't work yet")
+        self.assertIn("Unicode", dir())
 
     def test_invalid(self):
         try:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 14:15:13 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 14:15:13 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_a_note_in_PyUnicode=5FC?=
 =?utf8?q?opyCharacters=28=29_doc=3A_it_doesn=27t_write_null_character?=
Message-ID: <E1R9FWD-0008Bl-Qi@dinsdale.python.org>

http://hg.python.org/cpython/rev/a11970b158f2
changeset:   72527:a11970b158f2
user:        Victor Stinner <vstinner at wyplay.com>
date:        Thu Sep 29 14:14:38 2011 +0200
summary:
  Add a note in PyUnicode_CopyCharacters() doc: it doesn't write null character

Cleanup also the code (avoid the goto).

files:
  Include/unicodeobject.h |   8 +-
  Objects/unicodeobject.c |  87 ++++++++++++++--------------
  2 files changed, 49 insertions(+), 46 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -521,9 +521,9 @@
 /* Copy character from one unicode object into another, this function performs
    character conversion when necessary and falls back to memcpy if possible.
 
-   Fail if 'to' is smaller than how_many or smaller than len(from)-from_start,
-   or if kind(from[from_start:from_start+how_many]) > kind(to), or if to has
-   more than 1 reference.
+   Fail if to is too small (smaller than how_many or smaller than
+   len(from)-from_start), or if kind(from[from_start:from_start+how_many]) >
+   kind(to), or if to has more than 1 reference.
 
    Return the number of written character, or return -1 and raise an exception
    on error.
@@ -533,6 +533,8 @@
        how_many = min(how_many, len(from) - from_start)
        to[to_start:to_start+how_many] = from[from_start:from_start+how_many]
        return how_many
+
+   Note: The function doesn't write a terminating null character.
    */
 #ifndef Py_LIMITED_API
 PyAPI_FUNC(Py_ssize_t) PyUnicode_CopyCharacters(
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -615,8 +615,8 @@
                          PyObject *from, Py_ssize_t from_start,
                          Py_ssize_t how_many)
 {
-    unsigned int from_kind;
-    unsigned int to_kind;
+    unsigned int from_kind, to_kind;
+    void *from_data, *to_data;
 
     assert(PyUnicode_Check(from));
     assert(PyUnicode_Check(to));
@@ -645,44 +645,20 @@
     _PyUnicode_DIRTY(to);
 
     from_kind = PyUnicode_KIND(from);
+    from_data = PyUnicode_DATA(from);
     to_kind = PyUnicode_KIND(to);
+    to_data = PyUnicode_DATA(to);
 
     if (from_kind == to_kind) {
         /* fast path */
-        Py_MEMCPY((char*)PyUnicode_DATA(to)
+        Py_MEMCPY((char*)to_data
                       + PyUnicode_KIND_SIZE(to_kind, to_start),
-                  (char*)PyUnicode_DATA(from)
+                  (char*)from_data
                       + PyUnicode_KIND_SIZE(from_kind, from_start),
                   PyUnicode_KIND_SIZE(to_kind, how_many));
-        return how_many;
-    }
-
-    if (from_kind > to_kind) {
-        /* slow path to check for character overflow */
-        const Py_UCS4 to_maxchar = PyUnicode_MAX_CHAR_VALUE(to);
-        void *from_data = PyUnicode_DATA(from);
-        void *to_data = PyUnicode_DATA(to);
-        Py_UCS4 ch, maxchar;
-        Py_ssize_t i;
-        int overflow;
-
-        maxchar = 0;
-        overflow = 0;
-        for (i=0; i < how_many; i++) {
-            ch = PyUnicode_READ(from_kind, from_data, from_start + i);
-            if (ch > maxchar) {
-                maxchar = ch;
-                if (maxchar > to_maxchar) {
-                    overflow = 1;
-                    break;
-                }
-            }
-            PyUnicode_WRITE(to_kind, to_data, to_start + i, ch);
-        }
-        if (!overflow)
-            return how_many;
-    }
-    else if (from_kind == PyUnicode_1BYTE_KIND && to_kind == PyUnicode_2BYTE_KIND)
+    }
+    else if (from_kind == PyUnicode_1BYTE_KIND
+             && to_kind == PyUnicode_2BYTE_KIND)
     {
         _PyUnicode_CONVERT_BYTES(
             Py_UCS1, Py_UCS2,
@@ -690,7 +666,6 @@
             PyUnicode_1BYTE_DATA(from) + from_start + how_many,
             PyUnicode_2BYTE_DATA(to) + to_start
             );
-        return how_many;
     }
     else if (from_kind == PyUnicode_1BYTE_KIND
              && to_kind == PyUnicode_4BYTE_KIND)
@@ -701,7 +676,6 @@
             PyUnicode_1BYTE_DATA(from) + from_start + how_many,
             PyUnicode_4BYTE_DATA(to) + to_start
             );
-        return how_many;
     }
     else if (from_kind == PyUnicode_2BYTE_KIND
              && to_kind == PyUnicode_4BYTE_KIND)
@@ -712,14 +686,41 @@
             PyUnicode_2BYTE_DATA(from) + from_start + how_many,
             PyUnicode_4BYTE_DATA(to) + to_start
             );
-        return how_many;
-    }
-    PyErr_Format(PyExc_ValueError,
-                 "Cannot copy UCS%u characters "
-                 "into a string of UCS%u characters",
-                 1 << (from_kind - 1),
-                 1 << (to_kind -1));
-    return -1;
+    }
+    else {
+        int invalid_kinds;
+        if (from_kind > to_kind) {
+            /* slow path to check for character overflow */
+            const Py_UCS4 to_maxchar = PyUnicode_MAX_CHAR_VALUE(to);
+            Py_UCS4 ch, maxchar;
+            Py_ssize_t i;
+
+            maxchar = 0;
+            invalid_kinds = 0;
+            for (i=0; i < how_many; i++) {
+                ch = PyUnicode_READ(from_kind, from_data, from_start + i);
+                if (ch > maxchar) {
+                    maxchar = ch;
+                    if (maxchar > to_maxchar) {
+                        invalid_kinds = 1;
+                        break;
+                    }
+                }
+                PyUnicode_WRITE(to_kind, to_data, to_start + i, ch);
+            }
+        }
+        else
+            invalid_kinds = 1;
+        if (invalid_kinds) {
+            PyErr_Format(PyExc_ValueError,
+                         "Cannot copy UCS%u characters "
+                         "into a string of UCS%u characters",
+                         1 << (from_kind - 1),
+                         1 << (to_kind -1));
+            return -1;
+        }
+    }
+    return how_many;
 }
 
 /* Find the maximum code point and count the number of surrogate pairs so a

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 19:46:11 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 19:46:11 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Move_=5FPyUnicode=5FUTF8=28?=
 =?utf8?q?=29_and_=5FPyUnicode=5FUTF8=5FLENGTH=28=29_outside_unicodeobject?=
 =?utf8?q?=2Eh?=
Message-ID: <E1R9KgV-00011F-Kp@dinsdale.python.org>

http://hg.python.org/cpython/rev/8cebe7310db7
changeset:   72528:8cebe7310db7
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 19:31:34 2011 +0200
summary:
  Move _PyUnicode_UTF8() and _PyUnicode_UTF8_LENGTH() outside unicodeobject.h

Move these macros to unicodeobject.c

files:
  Include/unicodeobject.h |  10 ----------
  Objects/unicodeobject.c |   8 ++++++++
  2 files changed, 8 insertions(+), 10 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -377,16 +377,6 @@
      PyUnicode_IS_COMPACT(op) ? _PyUnicode_COMPACT_DATA(op) :   \
      _PyUnicode_NONCOMPACT_DATA(op))
 
-#define _PyUnicode_UTF8(op)                     \
-    (PyUnicode_IS_COMPACT_ASCII(op) ?           \
-     ((char*)((PyASCIIObject*)(op) + 1)) :      \
-     ((PyCompactUnicodeObject*)(op))->utf8)
-
-#define _PyUnicode_UTF8_LENGTH(op)                      \
-    (PyUnicode_IS_COMPACT_ASCII(op) ?                   \
-     ((PyASCIIObject*)(op))->length :                   \
-     ((PyCompactUnicodeObject*)(op))->utf8_length)
-
 /* Compute (index * char_size) where char_size is 2 ** (kind - 1).
 
    The index is a character index, the result is a size in bytes. */
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -105,6 +105,14 @@
         } \
     } while (0)
 
+#define _PyUnicode_UTF8(op)                     \
+    (PyUnicode_IS_COMPACT_ASCII(op) ?           \
+     ((char*)((PyASCIIObject*)(op) + 1)) :      \
+     ((PyCompactUnicodeObject*)(op))->utf8)
+#define _PyUnicode_UTF8_LENGTH(op)                      \
+    (PyUnicode_IS_COMPACT_ASCII(op) ?                   \
+     ((PyASCIIObject*)(op))->length :                   \
+     ((PyCompactUnicodeObject*)(op))->utf8_length)
 #define _PyUnicode_WSTR(op) (((PyASCIIObject*)(op))->wstr)
 #define _PyUnicode_WSTR_LENGTH(op) (((PyCompactUnicodeObject*)(op))->wstr_length)
 #define _PyUnicode_LENGTH(op) (((PyASCIIObject *)(op))->length)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 19:46:13 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 19:46:13 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_=5FPyUnicode=5FReady=28=29_?=
 =?utf8?q?cannot_be_used_on_ready_strings_anymore?=
Message-ID: <E1R9KgX-00011P-PL@dinsdale.python.org>

http://hg.python.org/cpython/rev/2fc483009f6d
changeset:   72529:2fc483009f6d
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 19:43:17 2011 +0200
summary:
  _PyUnicode_Ready() cannot be used on ready strings anymore

 * Change its prototype: PyObject* instead of PyUnicodeoObject*.
 * Remove an old assertion, the result of PyUnicode_READY (_PyUnicode_Ready)
   must be checked instead

files:
  Include/unicodeobject.h |  16 ++++++-----
  Objects/unicodeobject.c |  40 +++++++++++-----------------
  2 files changed, 25 insertions(+), 31 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -456,7 +456,7 @@
 #define PyUnicode_READY(op)                        \
     (assert(PyUnicode_Check(op)),                       \
      (PyUnicode_IS_READY(op) ?                          \
-      0 : _PyUnicode_Ready((PyUnicodeObject *)(op))))
+      0 : _PyUnicode_Ready((PyObject *)(op))))
 
 /* Return a maximum character value which is suitable for creating another
    string based on op.  This is always an approximation but more efficient
@@ -497,14 +497,16 @@
     );
 #endif
 
-/* Initializes the canonical string representation from a the deprected
-   wstr/Py_UNICODE representation.  This function is used to convert
-   unicode objects which were created using the old API to the new flexible
-   format introduced with PEP 393.  The PyUnicode_READY() macro can be
-   more efficient if the string is already ready. */
+/* Initializes the canonical string representation from a the deprecated
+   wstr/Py_UNICODE representation. This function is used to convert Unicode
+   objects which were created using the old API to the new flexible format
+   introduced with PEP 393.
+
+   Don't call this function directly, use the public PyUnicode_READY() macro
+   instead. */
 #ifndef Py_LIMITED_API
 PyAPI_FUNC(int) _PyUnicode_Ready(
-    PyUnicodeObject *unicode    /* Unicode object */
+    PyObject *unicode           /* Unicode object */
     );
 #endif
 
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -781,8 +781,9 @@
 #endif
 
 int
-_PyUnicode_Ready(PyUnicodeObject *unicode)
-{
+_PyUnicode_Ready(PyObject *obj)
+{
+    PyUnicodeObject *unicode = (PyUnicodeObject *)obj;
     wchar_t *end;
     Py_UCS4 maxchar = 0;
     Py_ssize_t num_surrogates;
@@ -790,25 +791,19 @@
     Py_ssize_t length_wo_surrogates;
 #endif
 
-    assert(PyUnicode_Check(unicode));
-
-    if (unicode->data.any != NULL) {
-        assert(PyUnicode_KIND(unicode) != PyUnicode_WCHAR_KIND);
-        return 0;
-    }
-
     /* _PyUnicode_Ready() is only intented for old-style API usage where
-     * strings were created using _PyObject_New() and where no canonical
-     * representation (the str field) has been set yet aka strings
-     * which are not yet ready.
-     */
+       strings were created using _PyObject_New() and where no canonical
+       representation (the str field) has been set yet aka strings
+       which are not yet ready. */
+    assert(PyUnicode_Check(obj));
+    assert(!PyUnicode_IS_READY(obj));
+    assert(!PyUnicode_IS_COMPACT(obj));
+    assert(_PyUnicode_KIND(obj) == PyUnicode_WCHAR_KIND);
     assert(_PyUnicode_WSTR(unicode) != NULL);
-    assert(_PyUnicode_KIND(unicode) == PyUnicode_WCHAR_KIND);
-    assert(!PyUnicode_IS_COMPACT(unicode));
-    assert(!PyUnicode_IS_READY(unicode));
+    assert(unicode->data.any == NULL);
+    assert(unicode->_base.utf8 == NULL);
     /* Actually, it should neither be interned nor be anything else: */
-    assert(_PyUnicode_STATE(unicode).interned == 0);
-    assert(unicode->_base.utf8 == NULL);
+    assert(_PyUnicode_STATE(unicode).interned == SSTATE_NOT_INTERNED);
 
 #ifdef Py_DEBUG
     ++unicode_ready_calls;
@@ -816,11 +811,8 @@
 
     end = _PyUnicode_WSTR(unicode) + _PyUnicode_WSTR_LENGTH(unicode);
     if (find_maxchar_surrogates(_PyUnicode_WSTR(unicode), end,
-                                                  &maxchar,
-                                                  &num_surrogates) == -1) {
-        assert(0 && "PyUnicode_FindMaxCharAndNumSurrogatePairs failed");
+                                &maxchar, &num_surrogates) == -1)
         return -1;
-    }
 
     if (maxchar < 256) {
         unicode->data.any = PyObject_MALLOC(_PyUnicode_WSTR_LENGTH(unicode) + 1);
@@ -1046,8 +1038,8 @@
 
     /* If not empty and not single character, copy the Unicode data
        into the new object */
-    if (find_maxchar_surrogates(u, u + size, &maxchar,
-                                                  &num_surrogates) == -1)
+    if (find_maxchar_surrogates(u, u + size,
+                                &maxchar, &num_surrogates) == -1)
         return NULL;
 
     unicode = (PyUnicodeObject *) PyUnicode_New(size - num_surrogates,

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 19:48:54 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Thu, 29 Sep 2011 19:48:54 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzMDU4?=
 =?utf8?q?=3A_ossaudiodev=3A_fix_a_file_descriptor_leak_on_error=2E_Patch_?=
 =?utf8?q?by_Thomas?=
Message-ID: <E1R9Kj8-0001BE-DZ@dinsdale.python.org>

http://hg.python.org/cpython/rev/dec00ae64ca8
changeset:   72530:dec00ae64ca8
branch:      2.7
parent:      72513:f612b783bc17
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Thu Sep 29 19:43:01 2011 +0200
summary:
  Issue #13058: ossaudiodev: fix a file descriptor leak on error. Patch by Thomas
Jarosch.

files:
  Misc/ACKS             |  1 +
  Misc/NEWS             |  3 +++
  Modules/ossaudiodev.c |  1 +
  3 files changed, 5 insertions(+), 0 deletions(-)


diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -402,6 +402,7 @@
 Geert Jansen
 Jack Jansen
 Bill Janssen
+Thomas Jarosch
 Drew Jenkins
 Flemming Kj?r Jensen
 Jiba
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -50,6 +50,9 @@
 Library
 -------
 
+- Issue #13058: ossaudiodev: fix a file descriptor leak on error. Patch by
+  Thomas Jarosch.
+
 - Issue #12931: xmlrpclib now encodes Unicode URI to ISO-8859-1, instead of
   failing with a UnicodeDecodeError.
 
diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c
--- a/Modules/ossaudiodev.c
+++ b/Modules/ossaudiodev.c
@@ -129,6 +129,7 @@
     }
 
     if (ioctl(fd, SNDCTL_DSP_GETFMTS, &afmts) == -1) {
+        close(fd);
         PyErr_SetFromErrnoWithFilename(PyExc_IOError, devicename);
         return NULL;
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 19:48:55 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Thu, 29 Sep 2011 19:48:55 +0200
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMDU4?=
 =?utf8?q?=3A_ossaudiodev=3A_fix_a_file_descriptor_leak_on_error=2E_Patch_?=
 =?utf8?q?by_Thomas?=
Message-ID: <E1R9Kj9-0001BK-6y@dinsdale.python.org>

http://hg.python.org/cpython/rev/160b52c9e8b3
changeset:   72531:160b52c9e8b3
branch:      3.2
parent:      72473:58a75eeb5c8e
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Thu Sep 29 19:46:37 2011 +0200
summary:
  Issue #13058: ossaudiodev: fix a file descriptor leak on error. Patch by Thomas
Jarosch.

files:
  Misc/ACKS             |  1 +
  Misc/NEWS             |  3 +++
  Modules/ossaudiodev.c |  1 +
  3 files changed, 5 insertions(+), 0 deletions(-)


diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -436,6 +436,7 @@
 Geert Jansen
 Jack Jansen
 Bill Janssen
+Thomas Jarosch
 Drew Jenkins
 Flemming Kj?r Jensen
 MunSic Jeong
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -82,6 +82,9 @@
 Extension Modules
 -----------------
 
+- Issue #13058: ossaudiodev: fix a file descriptor leak on error. Patch by
+  Thomas Jarosch.
+
 - Issue #13013: ctypes: Fix a reference leak in PyCArrayType_from_ctype.
   Thanks to Suman Saha for finding the bug and providing a patch.
 
diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c
--- a/Modules/ossaudiodev.c
+++ b/Modules/ossaudiodev.c
@@ -129,6 +129,7 @@
     }
 
     if (ioctl(fd, SNDCTL_DSP_GETFMTS, &afmts) == -1) {
+        close(fd);
         PyErr_SetFromErrnoWithFilename(PyExc_IOError, devicename);
         return NULL;
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 19:48:56 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Thu, 29 Sep 2011 19:48:56 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313058=3A_ossaudiodev=3A_fix_a_file_descriptor_leak_?=
 =?utf8?q?on_error=2E_Patch_by_Thomas?=
Message-ID: <E1R9KjA-0001BO-0g@dinsdale.python.org>

http://hg.python.org/cpython/rev/d8dc24c4e83d
changeset:   72532:d8dc24c4e83d
parent:      72527:a11970b158f2
parent:      72531:160b52c9e8b3
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Thu Sep 29 19:49:37 2011 +0200
summary:
  Issue #13058: ossaudiodev: fix a file descriptor leak on error. Patch by Thomas
Jarosch.

files:
  Misc/ACKS             |  1 +
  Misc/NEWS             |  3 +++
  Modules/ossaudiodev.c |  1 +
  3 files changed, 5 insertions(+), 0 deletions(-)


diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -463,6 +463,7 @@
 Geert Jansen
 Jack Jansen
 Bill Janssen
+Thomas Jarosch
 Julien Jehannet
 Drew Jenkins
 Flemming Kj?r Jensen
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1298,6 +1298,9 @@
 Extension Modules
 -----------------
 
+- Issue #13058: ossaudiodev: fix a file descriptor leak on error. Patch by
+  Thomas Jarosch.
+
 - Issue #13013: ctypes: Fix a reference leak in PyCArrayType_from_ctype.
   Thanks to Suman Saha for finding the bug and providing a patch.
 
diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c
--- a/Modules/ossaudiodev.c
+++ b/Modules/ossaudiodev.c
@@ -129,6 +129,7 @@
     }
 
     if (ioctl(fd, SNDCTL_DSP_GETFMTS, &afmts) == -1) {
+        close(fd);
         PyErr_SetFromErrnoWithFilename(PyExc_IOError, devicename);
         return NULL;
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 19:48:56 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Thu, 29 Sep 2011 19:48:56 +0200
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?b?KTogTWVyZ2Uu?=
Message-ID: <E1R9KjA-0001BT-QW@dinsdale.python.org>

http://hg.python.org/cpython/rev/f7de67283c94
changeset:   72533:f7de67283c94
parent:      72532:d8dc24c4e83d
parent:      72529:2fc483009f6d
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Thu Sep 29 19:51:46 2011 +0200
summary:
  Merge.

files:
  Include/unicodeobject.h |  26 +++++----------
  Objects/unicodeobject.c |  48 ++++++++++++++--------------
  2 files changed, 33 insertions(+), 41 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -377,16 +377,6 @@
      PyUnicode_IS_COMPACT(op) ? _PyUnicode_COMPACT_DATA(op) :   \
      _PyUnicode_NONCOMPACT_DATA(op))
 
-#define _PyUnicode_UTF8(op)                     \
-    (PyUnicode_IS_COMPACT_ASCII(op) ?           \
-     ((char*)((PyASCIIObject*)(op) + 1)) :      \
-     ((PyCompactUnicodeObject*)(op))->utf8)
-
-#define _PyUnicode_UTF8_LENGTH(op)                      \
-    (PyUnicode_IS_COMPACT_ASCII(op) ?                   \
-     ((PyASCIIObject*)(op))->length :                   \
-     ((PyCompactUnicodeObject*)(op))->utf8_length)
-
 /* Compute (index * char_size) where char_size is 2 ** (kind - 1).
 
    The index is a character index, the result is a size in bytes. */
@@ -466,7 +456,7 @@
 #define PyUnicode_READY(op)                        \
     (assert(PyUnicode_Check(op)),                       \
      (PyUnicode_IS_READY(op) ?                          \
-      0 : _PyUnicode_Ready((PyUnicodeObject *)(op))))
+      0 : _PyUnicode_Ready((PyObject *)(op))))
 
 /* Return a maximum character value which is suitable for creating another
    string based on op.  This is always an approximation but more efficient
@@ -507,14 +497,16 @@
     );
 #endif
 
-/* Initializes the canonical string representation from a the deprected
-   wstr/Py_UNICODE representation.  This function is used to convert
-   unicode objects which were created using the old API to the new flexible
-   format introduced with PEP 393.  The PyUnicode_READY() macro can be
-   more efficient if the string is already ready. */
+/* Initializes the canonical string representation from a the deprecated
+   wstr/Py_UNICODE representation. This function is used to convert Unicode
+   objects which were created using the old API to the new flexible format
+   introduced with PEP 393.
+
+   Don't call this function directly, use the public PyUnicode_READY() macro
+   instead. */
 #ifndef Py_LIMITED_API
 PyAPI_FUNC(int) _PyUnicode_Ready(
-    PyUnicodeObject *unicode    /* Unicode object */
+    PyObject *unicode           /* Unicode object */
     );
 #endif
 
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -105,6 +105,14 @@
         } \
     } while (0)
 
+#define _PyUnicode_UTF8(op)                     \
+    (PyUnicode_IS_COMPACT_ASCII(op) ?           \
+     ((char*)((PyASCIIObject*)(op) + 1)) :      \
+     ((PyCompactUnicodeObject*)(op))->utf8)
+#define _PyUnicode_UTF8_LENGTH(op)                      \
+    (PyUnicode_IS_COMPACT_ASCII(op) ?                   \
+     ((PyASCIIObject*)(op))->length :                   \
+     ((PyCompactUnicodeObject*)(op))->utf8_length)
 #define _PyUnicode_WSTR(op) (((PyASCIIObject*)(op))->wstr)
 #define _PyUnicode_WSTR_LENGTH(op) (((PyCompactUnicodeObject*)(op))->wstr_length)
 #define _PyUnicode_LENGTH(op) (((PyASCIIObject *)(op))->length)
@@ -773,8 +781,9 @@
 #endif
 
 int
-_PyUnicode_Ready(PyUnicodeObject *unicode)
-{
+_PyUnicode_Ready(PyObject *obj)
+{
+    PyUnicodeObject *unicode = (PyUnicodeObject *)obj;
     wchar_t *end;
     Py_UCS4 maxchar = 0;
     Py_ssize_t num_surrogates;
@@ -782,25 +791,19 @@
     Py_ssize_t length_wo_surrogates;
 #endif
 
-    assert(PyUnicode_Check(unicode));
-
-    if (unicode->data.any != NULL) {
-        assert(PyUnicode_KIND(unicode) != PyUnicode_WCHAR_KIND);
-        return 0;
-    }
-
     /* _PyUnicode_Ready() is only intented for old-style API usage where
-     * strings were created using _PyObject_New() and where no canonical
-     * representation (the str field) has been set yet aka strings
-     * which are not yet ready.
-     */
+       strings were created using _PyObject_New() and where no canonical
+       representation (the str field) has been set yet aka strings
+       which are not yet ready. */
+    assert(PyUnicode_Check(obj));
+    assert(!PyUnicode_IS_READY(obj));
+    assert(!PyUnicode_IS_COMPACT(obj));
+    assert(_PyUnicode_KIND(obj) == PyUnicode_WCHAR_KIND);
     assert(_PyUnicode_WSTR(unicode) != NULL);
-    assert(_PyUnicode_KIND(unicode) == PyUnicode_WCHAR_KIND);
-    assert(!PyUnicode_IS_COMPACT(unicode));
-    assert(!PyUnicode_IS_READY(unicode));
+    assert(unicode->data.any == NULL);
+    assert(unicode->_base.utf8 == NULL);
     /* Actually, it should neither be interned nor be anything else: */
-    assert(_PyUnicode_STATE(unicode).interned == 0);
-    assert(unicode->_base.utf8 == NULL);
+    assert(_PyUnicode_STATE(unicode).interned == SSTATE_NOT_INTERNED);
 
 #ifdef Py_DEBUG
     ++unicode_ready_calls;
@@ -808,11 +811,8 @@
 
     end = _PyUnicode_WSTR(unicode) + _PyUnicode_WSTR_LENGTH(unicode);
     if (find_maxchar_surrogates(_PyUnicode_WSTR(unicode), end,
-                                                  &maxchar,
-                                                  &num_surrogates) == -1) {
-        assert(0 && "PyUnicode_FindMaxCharAndNumSurrogatePairs failed");
+                                &maxchar, &num_surrogates) == -1)
         return -1;
-    }
 
     if (maxchar < 256) {
         unicode->data.any = PyObject_MALLOC(_PyUnicode_WSTR_LENGTH(unicode) + 1);
@@ -1038,8 +1038,8 @@
 
     /* If not empty and not single character, copy the Unicode data
        into the new object */
-    if (find_maxchar_surrogates(u, u + size, &maxchar,
-                                                  &num_surrogates) == -1)
+    if (find_maxchar_surrogates(u, u + size,
+                                &maxchar, &num_surrogates) == -1)
         return NULL;
 
     unicode = (PyUnicodeObject *) PyUnicode_New(size - num_surrogates,

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 19:54:40 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 19:54:40 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_test=5Fcodecs_for_Windo?=
 =?utf8?q?ws=3A_check_size_of_wchar=5Ft=2C_not_sys=2Emaxunicode?=
Message-ID: <E1R9Koi-0001jb-5q@dinsdale.python.org>

http://hg.python.org/cpython/rev/85d11cf67aa8
changeset:   72534:85d11cf67aa8
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 19:53:55 2011 +0200
summary:
  Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode

files:
  Lib/test/test_codecs.py |  83 +++++++++++++++-------------
  1 files changed, 43 insertions(+), 40 deletions(-)


diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py
--- a/Lib/test/test_codecs.py
+++ b/Lib/test/test_codecs.py
@@ -3,6 +3,9 @@
 import codecs
 import locale
 import sys, _testcapi, io
+import ctypes
+
+SIZEOF_WCHAR_T = ctypes.sizeof(ctypes.c_wchar)
 
 class Queue(object):
     """
@@ -888,53 +891,53 @@
             self.assertEqual(uni, puny.decode("punycode"))
 
 class UnicodeInternalTest(unittest.TestCase):
+    @unittest.skipUnless(SIZEOF_WCHAR_T == 4, 'specific to 32-bit wchar_t')
     def test_bug1251300(self):
         # Decoding with unicode_internal used to not correctly handle "code
         # points" above 0x10ffff on UCS-4 builds.
-        if sys.maxunicode > 0xffff:
-            ok = [
-                (b"\x00\x10\xff\xff", "\U0010ffff"),
-                (b"\x00\x00\x01\x01", "\U00000101"),
-                (b"", ""),
-            ]
-            not_ok = [
-                b"\x7f\xff\xff\xff",
-                b"\x80\x00\x00\x00",
-                b"\x81\x00\x00\x00",
-                b"\x00",
-                b"\x00\x00\x00\x00\x00",
-            ]
-            for internal, uni in ok:
-                if sys.byteorder == "little":
-                    internal = bytes(reversed(internal))
-                self.assertEqual(uni, internal.decode("unicode_internal"))
-            for internal in not_ok:
-                if sys.byteorder == "little":
-                    internal = bytes(reversed(internal))
-                self.assertRaises(UnicodeDecodeError, internal.decode,
-                    "unicode_internal")
+        ok = [
+            (b"\x00\x10\xff\xff", "\U0010ffff"),
+            (b"\x00\x00\x01\x01", "\U00000101"),
+            (b"", ""),
+        ]
+        not_ok = [
+            b"\x7f\xff\xff\xff",
+            b"\x80\x00\x00\x00",
+            b"\x81\x00\x00\x00",
+            b"\x00",
+            b"\x00\x00\x00\x00\x00",
+        ]
+        for internal, uni in ok:
+            if sys.byteorder == "little":
+                internal = bytes(reversed(internal))
+            self.assertEqual(uni, internal.decode("unicode_internal"))
+        for internal in not_ok:
+            if sys.byteorder == "little":
+                internal = bytes(reversed(internal))
+            self.assertRaises(UnicodeDecodeError, internal.decode,
+                "unicode_internal")
 
+    @unittest.skipUnless(SIZEOF_WCHAR_T == 4, 'specific to 32-bit wchar_t')
     def test_decode_error_attributes(self):
-        if sys.maxunicode > 0xffff:
-            try:
-                b"\x00\x00\x00\x00\x00\x11\x11\x00".decode("unicode_internal")
-            except UnicodeDecodeError as ex:
-                self.assertEqual("unicode_internal", ex.encoding)
-                self.assertEqual(b"\x00\x00\x00\x00\x00\x11\x11\x00", ex.object)
-                self.assertEqual(4, ex.start)
-                self.assertEqual(8, ex.end)
-            else:
-                self.fail()
+        try:
+            b"\x00\x00\x00\x00\x00\x11\x11\x00".decode("unicode_internal")
+        except UnicodeDecodeError as ex:
+            self.assertEqual("unicode_internal", ex.encoding)
+            self.assertEqual(b"\x00\x00\x00\x00\x00\x11\x11\x00", ex.object)
+            self.assertEqual(4, ex.start)
+            self.assertEqual(8, ex.end)
+        else:
+            self.fail()
 
+    @unittest.skipUnless(SIZEOF_WCHAR_T == 4, 'specific to 32-bit wchar_t')
     def test_decode_callback(self):
-        if sys.maxunicode > 0xffff:
-            codecs.register_error("UnicodeInternalTest", codecs.ignore_errors)
-            decoder = codecs.getdecoder("unicode_internal")
-            ab = "ab".encode("unicode_internal").decode()
-            ignored = decoder(bytes("%s\x22\x22\x22\x22%s" % (ab[:4], ab[4:]),
-                                    "ascii"),
-                              "UnicodeInternalTest")
-            self.assertEqual(("ab", 12), ignored)
+        codecs.register_error("UnicodeInternalTest", codecs.ignore_errors)
+        decoder = codecs.getdecoder("unicode_internal")
+        ab = "ab".encode("unicode_internal").decode()
+        ignored = decoder(bytes("%s\x22\x22\x22\x22%s" % (ab[:4], ab[4:]),
+                                "ascii"),
+                          "UnicodeInternalTest")
+        self.assertEqual(("ab", 12), ignored)
 
     def test_encode_length(self):
         # Issue 3739

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 20:02:44 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 20:02:44 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_test=5Fcodeccallbacks_f?=
 =?utf8?q?or_Windows=3A_check_size_of_wchar=5Ft=2C_not_sys=2Emaxunicode?=
Message-ID: <E1R9KwW-0002V5-PT@dinsdale.python.org>

http://hg.python.org/cpython/rev/7a50e549bd11
changeset:   72535:7a50e549bd11
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 20:01:55 2011 +0200
summary:
  Fix test_codeccallbacks for Windows: check size of wchar_t, not sys.maxunicode

files:
  Lib/test/test_codeccallbacks.py |  10 ++++++----
  1 files changed, 6 insertions(+), 4 deletions(-)


diff --git a/Lib/test/test_codeccallbacks.py b/Lib/test/test_codeccallbacks.py
--- a/Lib/test/test_codeccallbacks.py
+++ b/Lib/test/test_codeccallbacks.py
@@ -2,6 +2,8 @@
 import sys, codecs, html.entities, unicodedata
 import ctypes
 
+SIZEOF_WCHAR_T = ctypes.sizeof(ctypes.c_wchar)
+
 class PosReturn:
     # this can be used for configurable callbacks
 
@@ -206,7 +208,7 @@
             b"\x00\x00\x00\x00\x00".decode,
             "unicode-internal",
         )
-        if sys.maxunicode > 0xffff:
+        if SIZEOF_WCHAR_T == 4:
             def handler_unicodeinternal(exc):
                 if not isinstance(exc, UnicodeDecodeError):
                     raise TypeError("don't know how to handle %r" % exc)
@@ -356,7 +358,7 @@
             ["ascii", "\uffffx", 0, 1, "ouch"],
             "'ascii' codec can't encode character '\\uffff' in position 0: ouch"
         )
-        if sys.maxunicode > 0xffff:
+        if SIZEOF_WCHAR_T == 4:
             self.check_exceptionobjectargs(
                 UnicodeEncodeError,
                 ["ascii", "\U00010000x", 0, 1, "ouch"],
@@ -391,7 +393,7 @@
             ["g\uffffrk", 1, 2, "ouch"],
             "can't translate character '\\uffff' in position 1: ouch"
         )
-        if sys.maxunicode > 0xffff:
+        if SIZEOF_WCHAR_T == 4:
             self.check_exceptionobjectargs(
                 UnicodeTranslateError,
                 ["g\U00010000rk", 1, 2, "ouch"],
@@ -682,7 +684,7 @@
         # Python/codecs.c::PyCodec_XMLCharRefReplaceErrors()
         # and inline implementations
         v = (1, 5, 10, 50, 100, 500, 1000, 5000, 10000, 50000)
-        if sys.maxunicode>=100000:
+        if SIZEOF_WCHAR_T == 4:
             v += (100000, 500000, 1000000)
         s = "".join([chr(x) for x in v])
         codecs.register_error("test.xmlcharrefreplace", codecs.xmlcharrefreplace_errors)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 23:18:54 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 23:18:54 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_fileio=5Finit=28=29_checks_?=
 =?utf8?q?for_failure_on_conversion_to_Py=5FUNICODE*?=
Message-ID: <E1R9O0M-0004Pm-9i@dinsdale.python.org>

http://hg.python.org/cpython/rev/06d9652dbd67
changeset:   72536:06d9652dbd67
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 23:19:04 2011 +0200
summary:
  fileio_init() checks for failure on conversion to Py_UNICODE*

files:
  Modules/_io/fileio.c |  10 ++++++----
  1 files changed, 6 insertions(+), 4 deletions(-)


diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c
--- a/Modules/_io/fileio.c
+++ b/Modules/_io/fileio.c
@@ -259,9 +259,11 @@
     }
 
 #ifdef MS_WINDOWS
-    if (PyUnicode_Check(nameobj))
-        widename = PyUnicode_AS_UNICODE(nameobj);
-    if (widename == NULL)
+    if (PyUnicode_Check(nameobj)) {
+        widename = PyUnicode_AsUnicode(nameobj);
+        if (widename == NULL)
+            return -1;
+    } else
 #endif
     if (fd < 0)
     {
@@ -378,7 +380,7 @@
         if (self->fd < 0) {
 #ifdef MS_WINDOWS
             if (widename != NULL)
-                PyErr_SetFromErrnoWithUnicodeFilename(PyExc_IOError, widename);
+                PyErr_SetFromErrnoWithFilenameObject(PyExc_IOError, nameobj);
             else
 #endif
                 PyErr_SetFromErrnoWithFilename(PyExc_IOError, name);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 23:51:16 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 23:51:16 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_PyLocale=5Fstrxfrm=28=29_us?=
	=?utf8?q?es_the_new_Unicode_API?=
Message-ID: <E1R9OVg-0007AI-HX@dinsdale.python.org>

http://hg.python.org/cpython/rev/771064b4466f
changeset:   72537:771064b4466f
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 23:32:06 2011 +0200
summary:
  PyLocale_strxfrm() uses the new Unicode API

files:
  Modules/_localemodule.c |  39 +++++++++++-----------------
  1 files changed, 15 insertions(+), 24 deletions(-)


diff --git a/Modules/_localemodule.c b/Modules/_localemodule.c
--- a/Modules/_localemodule.c
+++ b/Modules/_localemodule.c
@@ -271,31 +271,22 @@
 static PyObject*
 PyLocale_strxfrm(PyObject* self, PyObject* args)
 {
-    Py_UNICODE *s0;
-    Py_ssize_t n0;
-    wchar_t *s, *buf = NULL;
-    size_t n1, n2;
+    PyObject *str;
+    Py_ssize_t n1;
+    wchar_t *s = NULL, *buf = NULL;
+    size_t n2;
     PyObject *result = NULL;
-#if Py_UNICODE_SIZE != SIZEOF_WCHAR_T
-    Py_ssize_t i;
-#endif
 
-    if (!PyArg_ParseTuple(args, "u#:strxfrm", &s0, &n0))
+    if (!PyArg_ParseTuple(args, "U:strxfrm", &str))
         return NULL;
 
-#if Py_UNICODE_SIZE == SIZEOF_WCHAR_T
-    s = (wchar_t *) s0;
-#else
-    s = PyMem_Malloc((n0+1)*sizeof(wchar_t));
-    if (!s)
-        return PyErr_NoMemory();
-    for (i=0; i<=n0; i++)
-        s[i] = s0[i];
-#endif
+    s = PyUnicode_AsWideCharString(str, &n1);
+    if (s == NULL)
+        goto exit;
 
     /* assume no change in size, first */
-    n1 = wcslen(s) + 1;
-    buf = PyMem_Malloc(n1*sizeof(wchar_t));
+    n1 = n1 + 1;
+    buf = PyMem_Malloc(n1 * sizeof(wchar_t));
     if (!buf) {
         PyErr_NoMemory();
         goto exit;
@@ -311,11 +302,11 @@
         n2 = wcsxfrm(buf, s, n2+1);
     }
     result = PyUnicode_FromWideChar(buf, n2);
- exit:
-    if (buf) PyMem_Free(buf);
-#if Py_UNICODE_SIZE != SIZEOF_WCHAR_T
-    PyMem_Free(s);
-#endif
+exit:
+    if (buf)
+        PyMem_Free(buf);
+    if (s)
+        PyMem_Free(s);
     return result;
 }
 #endif

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 23:51:17 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 23:51:17 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Strip_trailing_spaces_of_?=
	=?utf8?b?X3BpY2tsZS5j?=
Message-ID: <E1R9OVh-0007Au-Ax@dinsdale.python.org>

http://hg.python.org/cpython/rev/a4a9ba2c31ef
changeset:   72538:a4a9ba2c31ef
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 23:40:53 2011 +0200
summary:
  Strip trailing spaces of _pickle.c

files:
  Modules/_pickle.c |  34 +++++++++++++++++-----------------
  1 files changed, 17 insertions(+), 17 deletions(-)


diff --git a/Modules/_pickle.c b/Modules/_pickle.c
--- a/Modules/_pickle.c
+++ b/Modules/_pickle.c
@@ -605,7 +605,7 @@
 /*************************************************************************/
 
 /* Helpers for creating the argument tuple passed to functions. This has the
-   performance advantage of calling PyTuple_New() only once. 
+   performance advantage of calling PyTuple_New() only once.
 
    XXX(avassalotti): Inline directly in _Pickler_FastCall() and
    _Unpickler_FastCall(). */
@@ -813,7 +813,7 @@
     fix_imports = PyObject_IsTrue(fix_imports_obj);
     if (fix_imports == -1)
         return -1;
-    
+
     self->proto = proto;
     self->bin = proto > 0;
     self->fix_imports = fix_imports && proto < 3;
@@ -909,7 +909,7 @@
     Py_ssize_t read_size, prefetched_size = 0;
 
     assert(self->read != NULL);
-    
+
     if (_Unpickler_SkipConsumed(self) < 0)
         return -1;
 
@@ -1037,7 +1037,7 @@
         self->next_read_idx = num_read;
         return _Unpickler_CopyLine(self, self->input_buffer, num_read, result);
     }
- 
+
     /* If we get here, we've run off the end of the input string. Return the
        remaining string and let the caller figure it out. */
     *result = self->input_buffer + self->next_read_idx;
@@ -1601,7 +1601,7 @@
          * bits.
          */
         if (sign < 0 &&
-            nbytes > 1 && 
+            nbytes > 1 &&
             pdata[nbytes - 1] == 0xff &&
             (pdata[nbytes - 2] & 0x80) != 0) {
             nbytes--;
@@ -1667,7 +1667,7 @@
             return -1;
         if (_Pickler_Write(self, pdata, 9) < 0)
             return -1;
-   } 
+   }
     else {
         int result = -1;
         char *buf = NULL;
@@ -1782,10 +1782,10 @@
 #else
     const Py_ssize_t expandsize = 6;
 #endif
-    
+
     if (size > PY_SSIZE_T_MAX / expandsize)
         return PyErr_NoMemory();
-    
+
     repr = PyByteArray_FromStringAndSize(NULL, expandsize * size);
     if (repr == NULL)
         return NULL;
@@ -3055,7 +3055,7 @@
         return -1;
 
     if (state) {
-        if (save(self, state, 0) < 0 || 
+        if (save(self, state, 0) < 0 ||
             _Pickler_Write(self, &build_op, 1) < 0)
             return -1;
     }
@@ -3309,7 +3309,7 @@
        Developers often forget to call __init__() in their subclasses, which
        would trigger a segfault without this check. */
     if (self->write == NULL) {
-        PyErr_Format(PicklingError, 
+        PyErr_Format(PicklingError,
                      "Pickler.__init__() was not called by %s.__init__()",
                      Py_TYPE(self)->tp_name);
         return NULL;
@@ -3789,7 +3789,7 @@
     0,                                  /*tp_is_gc*/
 };
 
-/* Temporary helper for calling self.find_class(). 
+/* Temporary helper for calling self.find_class().
 
    XXX: It would be nice to able to avoid Python function call overhead, by
    using directly the C version of find_class(), when find_class() is not
@@ -3842,7 +3842,7 @@
         return bad_readline();
 
     errno = 0;
-    /* XXX: Should the base argument of strtol() be explicitly set to 10? 
+    /* XXX: Should the base argument of strtol() be explicitly set to 10?
        XXX(avassalotti): Should this uses PyOS_strtol()? */
     x = strtol(s, &endptr, 0);
 
@@ -4207,7 +4207,7 @@
 
     x = calc_binint(s, 4);
     if (x < 0) {
-        PyErr_SetString(UnpicklingError, 
+        PyErr_SetString(UnpicklingError,
                         "BINSTRING pickle has negative byte count");
         return -1;
     }
@@ -5001,7 +5001,7 @@
         return stack_underflow();
     if (len == x)  /* nothing to do */
         return 0;
-    if ((len - x) % 2 != 0) { 
+    if ((len - x) % 2 != 0) {
         /* Currupt or hostile pickle -- we never write one like this. */
         PyErr_SetString(UnpicklingError, "odd number of items for SETITEMS");
         return -1;
@@ -5358,7 +5358,7 @@
        not call Unpickler.__init__(). Here, we simply ensure that self->read
        is not NULL. */
     if (self->read == NULL) {
-        PyErr_Format(UnpicklingError, 
+        PyErr_Format(UnpicklingError,
                      "Unpickler.__init__() was not called by %s.__init__()",
                      Py_TYPE(self)->tp_name);
         return NULL;
@@ -5460,7 +5460,7 @@
         global = PyObject_GetAttr(module, global_name);
         Py_DECREF(module);
     }
-    else { 
+    else {
         global = PyObject_GetAttr(module, global_name);
     }
     return global;
@@ -5637,7 +5637,7 @@
  * intentional, as these should be treated as black-box implementation details.
  *
  * We do, however, have to implement pickling/unpickling support because of
- * real-world code like cvs2svn. 
+ * real-world code like cvs2svn.
  */
 
 typedef struct {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Sep 29 23:51:18 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 29 Sep 2011 23:51:18 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_raw=5Funicode=5Fescape=28?=
	=?utf8?q?=29_uses_the_new_Unicode_API?=
Message-ID: <E1R9OVi-0007Ay-3H@dinsdale.python.org>

http://hg.python.org/cpython/rev/b9558df8cc58
changeset:   72539:b9558df8cc58
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Sep 29 23:50:23 2011 +0200
summary:
  raw_unicode_escape() uses the new Unicode API

files:
  Modules/_pickle.c |  75 +++++++++++-----------------------
  1 files changed, 25 insertions(+), 50 deletions(-)


diff --git a/Modules/_pickle.c b/Modules/_pickle.c
--- a/Modules/_pickle.c
+++ b/Modules/_pickle.c
@@ -1769,33 +1769,37 @@
 /* A copy of PyUnicode_EncodeRawUnicodeEscape() that also translates
    backslash and newline characters to \uXXXX escapes. */
 static PyObject *
-raw_unicode_escape(const Py_UNICODE *s, Py_ssize_t size)
-{
+raw_unicode_escape(PyObject *obj)
+{
+    static const char *hexdigits = "0123456789abcdef";
     PyObject *repr, *result;
     char *p;
-    char *q;
-
-    static const char *hexdigits = "0123456789abcdef";
-
-#ifdef Py_UNICODE_WIDE
-    const Py_ssize_t expandsize = 10;
-#else
-    const Py_ssize_t expandsize = 6;
-#endif
+    Py_ssize_t i, size, expandsize;
+    void *data;
+    unsigned int kind;
+
+    if (PyUnicode_READY(obj))
+        return NULL;
+
+    size = PyUnicode_GET_LENGTH(obj);
+    data = PyUnicode_DATA(obj);
+    kind = PyUnicode_KIND(obj);
+    if (kind == PyUnicode_4BYTE_KIND)
+        expandsize = 10;
+    else
+        expandsize = 6;
 
     if (size > PY_SSIZE_T_MAX / expandsize)
         return PyErr_NoMemory();
-
     repr = PyByteArray_FromStringAndSize(NULL, expandsize * size);
     if (repr == NULL)
         return NULL;
     if (size == 0)
         goto done;
 
-    p = q = PyByteArray_AS_STRING(repr);
-    while (size-- > 0) {
-        Py_UNICODE ch = *s++;
-#ifdef Py_UNICODE_WIDE
+    p = PyByteArray_AS_STRING(repr);
+    for (i=0; i < size; i++) {
+        Py_UCS4 ch = PyUnicode_READ(kind, data, i);
         /* Map 32-bit characters to '\Uxxxxxxxx' */
         if (ch >= 0x10000) {
             *p++ = '\\';
@@ -1809,36 +1813,8 @@
             *p++ = hexdigits[(ch >> 4) & 0xf];
             *p++ = hexdigits[ch & 15];
         }
-        else
-#else
-            /* Map UTF-16 surrogate pairs to '\U00xxxxxx' */
-            if (ch >= 0xD800 && ch < 0xDC00) {
-                Py_UNICODE ch2;
-                Py_UCS4 ucs;
-
-                ch2 = *s++;
-                size--;
-                if (ch2 >= 0xDC00 && ch2 <= 0xDFFF) {
-                    ucs = (((ch & 0x03FF) << 10) | (ch2 & 0x03FF)) + 0x00010000;
-                    *p++ = '\\';
-                    *p++ = 'U';
-                    *p++ = hexdigits[(ucs >> 28) & 0xf];
-                    *p++ = hexdigits[(ucs >> 24) & 0xf];
-                    *p++ = hexdigits[(ucs >> 20) & 0xf];
-                    *p++ = hexdigits[(ucs >> 16) & 0xf];
-                    *p++ = hexdigits[(ucs >> 12) & 0xf];
-                    *p++ = hexdigits[(ucs >> 8) & 0xf];
-                    *p++ = hexdigits[(ucs >> 4) & 0xf];
-                    *p++ = hexdigits[ucs & 0xf];
-                    continue;
-                }
-                /* Fall through: isolated surrogates are copied as-is */
-                s--;
-                size++;
-            }
-#endif
         /* Map 16-bit characters to '\uxxxx' */
-        if (ch >= 256 || ch == '\\' || ch == '\n') {
+        else if (ch >= 256 || ch == '\\' || ch == '\n') {
             *p++ = '\\';
             *p++ = 'u';
             *p++ = hexdigits[(ch >> 12) & 0xf];
@@ -1850,9 +1826,9 @@
         else
             *p++ = (char) ch;
     }
-    size = p - q;
-
-  done:
+    size = p - PyByteArray_AS_STRING(repr);
+
+done:
     result = PyBytes_FromStringAndSize(PyByteArray_AS_STRING(repr), size);
     Py_DECREF(repr);
     return result;
@@ -1893,8 +1869,7 @@
     else {
         const char unicode_op = UNICODE;
 
-        encoded = raw_unicode_escape(PyUnicode_AS_UNICODE(obj),
-                                     PyUnicode_GET_SIZE(obj));
+        encoded = raw_unicode_escape(obj);
         if (encoded == NULL)
             goto error;
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep 30 00:50:51 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 30 Sep 2011 00:50:51 +0200 (CEST)
Subject: [Python-checkins] r88903 - tracker/instances/python-dev/schema.py
Message-ID: <3S8rj70vwyzPGT@mail.python.org>

Author: ezio.melotti
Date: Fri Sep 30 00:50:50 2011
New Revision: 88903

Log:
#421: allow users to create issues with hgrepo links.

Modified:
   tracker/instances/python-dev/schema.py

Modified: tracker/instances/python-dev/schema.py
==============================================================================
--- tracker/instances/python-dev/schema.py	(original)
+++ tracker/instances/python-dev/schema.py	Fri Sep 30 00:50:50 2011
@@ -267,7 +267,7 @@
                               properties=('title', 'type',
                                           'components', 'versions',
                                           'severity',
-                                          'messages', 'files', 'nosy'),
+                                          'messages', 'files', 'nosy', 'hgrepos'),
                               description='User can report and discuss issues')
 db.security.addPermissionToRole('User', p)
 

From python-checkins at python.org  Fri Sep 30 00:54:00 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 30 Sep 2011 00:54:00 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_array_module_stores_the_typ?=
 =?utf8?q?ecode_in_a_char=2C_instead_of_Py=5FUNICODE?=
Message-ID: <E1R9PUO-0003uy-2r@dinsdale.python.org>

http://hg.python.org/cpython/rev/7042a83f37e6
changeset:   72540:7042a83f37e6
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 30 00:03:59 2011 +0200
summary:
  array module stores the typecode in a char, instead of Py_UNICODE

files:
  Modules/arraymodule.c |  35 ++++++++++++++----------------
  1 files changed, 16 insertions(+), 19 deletions(-)


diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c
--- a/Modules/arraymodule.c
+++ b/Modules/arraymodule.c
@@ -22,7 +22,7 @@
  * functions aren't visible yet.
  */
 struct arraydescr {
-    Py_UNICODE typecode;
+    char typecode;
     int itemsize;
     PyObject * (*getitem)(struct arrayobject *, Py_ssize_t);
     int (*setitem)(struct arrayobject *, Py_ssize_t, PyObject *);
@@ -1510,7 +1510,7 @@
 {
     Py_UNICODE *ustr;
     Py_ssize_t n;
-    Py_UNICODE typecode;
+    char typecode;
 
     if (!PyArg_ParseTuple(args, "u#:fromunicode", &ustr, &n))
         return NULL;
@@ -1545,7 +1545,7 @@
 static PyObject *
 array_tounicode(arrayobject *self, PyObject *unused)
 {
-    Py_UNICODE typecode;
+    char typecode;
     typecode = self->ob_descr->typecode;
     if ((typecode != 'u')) {
         PyErr_SetString(PyExc_ValueError,
@@ -1642,7 +1642,7 @@
  * be found.
  */
 static enum machine_format_code
-typecode_to_mformat_code(int typecode)
+typecode_to_mformat_code(char typecode)
 {
 #ifdef WORDS_BIGENDIAN
     const int is_big_endian = 1;
@@ -1721,7 +1721,7 @@
         intsize = sizeof(PY_LONG_LONG);
         is_signed = 0;
         break;
-#endif 
+#endif
     default:
         return UNKNOWN_FORMAT;
     }
@@ -1752,7 +1752,7 @@
  * NULL is returned to indicate a failure.
  */
 static PyObject *
-make_array(PyTypeObject *arraytype, Py_UNICODE typecode, PyObject *items)
+make_array(PyTypeObject *arraytype, char typecode, PyObject *items)
 {
     PyObject *new_args;
     PyObject *array_obj;
@@ -1761,7 +1761,7 @@
     assert(arraytype != NULL);
     assert(items != NULL);
 
-    typecode_obj = PyUnicode_FromUnicode(&typecode, 1);
+    typecode_obj = PyUnicode_FromOrdinal(typecode);
     if (typecode_obj == NULL)
         return NULL;
 
@@ -1791,17 +1791,14 @@
     PyObject *items;
     PyObject *converted_items;
     PyObject *result;
-    int typecode_int;
-    Py_UNICODE typecode;
+    int typecode;
     enum machine_format_code mformat_code;
     struct arraydescr *descr;
 
     if (!PyArg_ParseTuple(args, "OCiO:array._array_reconstructor",
-                    &arraytype, &typecode_int, &mformat_code, &items))
+                    &arraytype, &typecode, &mformat_code, &items))
         return NULL;
 
-    typecode = (Py_UNICODE)typecode_int;
-
     if (!PyType_Check(arraytype)) {
         PyErr_Format(PyExc_TypeError,
             "first argument must a type object, not %.200s",
@@ -1815,7 +1812,7 @@
         return NULL;
     }
     for (descr = descriptors; descr->typecode != '\0'; descr++) {
-        if (descr->typecode == typecode)
+        if ((int)descr->typecode == typecode)
             break;
     }
     if (descr->typecode == '\0') {
@@ -1837,9 +1834,9 @@
     }
 
     /* Fast path: No decoding has to be done. */
-    if (mformat_code == typecode_to_mformat_code(typecode) ||
+    if (mformat_code == typecode_to_mformat_code((char)typecode) ||
         mformat_code == UNKNOWN_FORMAT) {
-        return make_array(arraytype, typecode, items);
+        return make_array(arraytype, (char)typecode, items);
     }
 
     /* Slow path: Decode the byte string according to the given machine
@@ -1985,7 +1982,7 @@
         return NULL;
     }
 
-    result = make_array(arraytype, typecode, converted_items);
+    result = make_array(arraytype, (char)typecode, converted_items);
     Py_DECREF(converted_items);
     return result;
 }
@@ -2074,8 +2071,8 @@
 static PyObject *
 array_get_typecode(arrayobject *a, void *closure)
 {
-    Py_UNICODE tc = a->ob_descr->typecode;
-    return PyUnicode_FromUnicode(&tc, 1);
+    char typecode = a->ob_descr->typecode;
+    return PyUnicode_FromOrdinal(typecode);
 }
 
 static PyObject *
@@ -2147,7 +2144,7 @@
 static PyObject *
 array_repr(arrayobject *a)
 {
-    Py_UNICODE typecode;
+    char typecode;
     PyObject *s, *v = NULL;
     Py_ssize_t len;
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep 30 00:54:00 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 30 Sep 2011 00:54:00 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_array_module_uses_the_new_U?=
	=?utf8?q?nicode_API?=
Message-ID: <E1R9PUO-0003v8-Rn@dinsdale.python.org>

http://hg.python.org/cpython/rev/7fa098f6dc6a
changeset:   72541:7fa098f6dc6a
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 30 00:51:10 2011 +0200
summary:
  array module uses the new Unicode API

 * Use Py_UCS4* buffer instead of Py_UNICODE*
 * Use "I" or "L" format, instead of "u" format

files:
  Lib/test/test_array.py |   8 ++-
  Modules/arraymodule.c  |  63 ++++++++++++++---------------
  2 files changed, 36 insertions(+), 35 deletions(-)


diff --git a/Lib/test/test_array.py b/Lib/test/test_array.py
--- a/Lib/test/test_array.py
+++ b/Lib/test/test_array.py
@@ -218,10 +218,14 @@
         self.assertEqual(bi[1], len(a))
 
     def test_byteswap(self):
-        a = array.array(self.typecode, self.example)
+        if self.typecode == 'u':
+            example = '\U00100100'
+        else:
+            example = self.example
+        a = array.array(self.typecode, example)
         self.assertRaises(TypeError, a.byteswap, 42)
         if a.itemsize in (1, 2, 4, 8):
-            b = array.array(self.typecode, self.example)
+            b = array.array(self.typecode, example)
             b.byteswap()
             if a.itemsize==1:
                 self.assertEqual(a, b)
diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c
--- a/Modules/arraymodule.c
+++ b/Modules/arraymodule.c
@@ -174,24 +174,25 @@
 static PyObject *
 u_getitem(arrayobject *ap, Py_ssize_t i)
 {
-    return PyUnicode_FromUnicode(&((Py_UNICODE *) ap->ob_item)[i], 1);
+    return PyUnicode_FromOrdinal(((Py_UCS4 *) ap->ob_item)[i]);
 }
 
 static int
 u_setitem(arrayobject *ap, Py_ssize_t i, PyObject *v)
 {
-    Py_UNICODE *p;
-    Py_ssize_t len;
+    PyObject *p;
 
-    if (!PyArg_Parse(v, "u#;array item must be unicode character", &p, &len))
+    if (!PyArg_Parse(v, "U;array item must be unicode character", &p))
         return -1;
-    if (len != 1) {
+    if (PyUnicode_READY(p))
+        return -1;
+    if (PyUnicode_GET_LENGTH(p) != 1) {
         PyErr_SetString(PyExc_TypeError,
                         "array item must be unicode character");
         return -1;
     }
     if (i >= 0)
-        ((Py_UNICODE *)ap->ob_item)[i] = p[0];
+        ((Py_UCS4 *)ap->ob_item)[i] = PyUnicode_READ_CHAR(p, 0);
     return 0;
 }
 
@@ -443,6 +444,13 @@
     return 0;
 }
 
+#if SIZEOF_INT == 4
+#  define STRUCT_LONG_FORMAT "I"
+#elif SIZEOF_LONG == 4
+#  define STRUCT_LONG_FORMAT "L"
+#else
+#  error "Unable to get struct format for Py_UCS4"
+#endif
 
 /* Description of types.
  *
@@ -452,7 +460,7 @@
 static struct arraydescr descriptors[] = {
     {'b', 1, b_getitem, b_setitem, "b", 1, 1},
     {'B', 1, BB_getitem, BB_setitem, "B", 1, 0},
-    {'u', sizeof(Py_UNICODE), u_getitem, u_setitem, "u", 0, 0},
+    {'u', sizeof(Py_UCS4), u_getitem, u_setitem, STRUCT_LONG_FORMAT, 0, 0},
     {'h', sizeof(short), h_getitem, h_setitem, "h", 1, 1},
     {'H', sizeof(short), HH_getitem, HH_setitem, "H", 1, 0},
     {'i', sizeof(int), i_getitem, i_setitem, "i", 1, 1},
@@ -1508,25 +1516,26 @@
 static PyObject *
 array_fromunicode(arrayobject *self, PyObject *args)
 {
-    Py_UNICODE *ustr;
+    PyObject *ustr;
     Py_ssize_t n;
-    char typecode;
 
-    if (!PyArg_ParseTuple(args, "u#:fromunicode", &ustr, &n))
+    if (!PyArg_ParseTuple(args, "U:fromunicode", &ustr))
         return NULL;
-    typecode = self->ob_descr->typecode;
-    if ((typecode != 'u')) {
+    if (self->ob_descr->typecode != 'u') {
         PyErr_SetString(PyExc_ValueError,
             "fromunicode() may only be called on "
             "unicode type arrays");
         return NULL;
     }
+    if (PyUnicode_READY(ustr))
+        return NULL;
+    n = PyUnicode_GET_LENGTH(ustr);
     if (n > 0) {
         Py_ssize_t old_size = Py_SIZE(self);
         if (array_resize(self, old_size + n) == -1)
             return NULL;
-        memcpy(self->ob_item + old_size * sizeof(Py_UNICODE),
-               ustr, n * sizeof(Py_UNICODE));
+        if (!PyUnicode_AsUCS4(ustr, (Py_UCS4 *)self->ob_item + old_size, n, 0))
+            return NULL;
     }
 
     Py_INCREF(Py_None);
@@ -1545,14 +1554,14 @@
 static PyObject *
 array_tounicode(arrayobject *self, PyObject *unused)
 {
-    char typecode;
-    typecode = self->ob_descr->typecode;
-    if ((typecode != 'u')) {
+    if (self->ob_descr->typecode != 'u') {
         PyErr_SetString(PyExc_ValueError,
              "tounicode() may only be called on unicode type arrays");
         return NULL;
     }
-    return PyUnicode_FromUnicode((Py_UNICODE *) self->ob_item, Py_SIZE(self));
+    return PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND,
+                                     (Py_UCS4 *) self->ob_item,
+                                     Py_SIZE(self));
 }
 
 PyDoc_STRVAR(tounicode_doc,
@@ -1659,13 +1668,7 @@
         return UNSIGNED_INT8;
 
     case 'u':
-        if (sizeof(Py_UNICODE) == 2) {
-            return UTF16_LE + is_big_endian;
-        }
-        if (sizeof(Py_UNICODE) == 4) {
-            return UTF32_LE + is_big_endian;
-        }
-        return UNKNOWN_FORMAT;
+        return UTF32_LE + is_big_endian;
 
     case 'f':
         if (sizeof(float) == 4) {
@@ -2411,14 +2414,8 @@
         view->strides = &(view->itemsize);
     view->format = NULL;
     view->internal = NULL;
-    if ((flags & PyBUF_FORMAT) == PyBUF_FORMAT) {
+    if ((flags & PyBUF_FORMAT) == PyBUF_FORMAT)
         view->format = self->ob_descr->formats;
-#ifdef Py_UNICODE_WIDE
-        if (self->ob_descr->typecode == 'u') {
-            view->format = "w";
-        }
-#endif
-    }
 
  finish:
     self->ob_exports++;
@@ -2543,7 +2540,7 @@
                         return NULL;
                     }
                     self->ob_item = item;
-                    Py_SIZE(self) = n / sizeof(Py_UNICODE);
+                    Py_SIZE(self) = n / sizeof(Py_UCS4);
                     memcpy(item, PyUnicode_AS_DATA(initial), n);
                     self->allocated = Py_SIZE(self);
                 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep 30 01:45:22 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 30 Sep 2011 01:45:22 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_posix_module_catches_PyUnic?=
 =?utf8?q?ode=5FAsUnicode=28=29_failure?=
Message-ID: <E1R9QI6-00087i-TA@dinsdale.python.org>

http://hg.python.org/cpython/rev/f265b20ac9c9
changeset:   72542:f265b20ac9c9
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 30 01:44:27 2011 +0200
summary:
  posix module catches PyUnicode_AsUnicode() failure

 * Replace PyUnicode_AS_UNICODE by PyUnicode_AsUnicode, PyUnicode_AS_UNICODE is
   no more a real macro
 * Replace Py_UNICODE by wchar_t in code specific to Windows

files:
  Modules/posixmodule.c |  279 ++++++++++++++++++-----------
  1 files changed, 175 insertions(+), 104 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -697,7 +697,7 @@
 }
 
 static PyObject *
-win32_error_unicode(char* function, Py_UNICODE* filename)
+win32_error_unicode(char* function, wchar_t* filename)
 {
     /* XXX - see win32_error for comments on 'function' */
     errno = GetLastError();
@@ -707,6 +707,20 @@
         return PyErr_SetFromWindowsErr(errno);
 }
 
+static PyObject *
+win32_error_object(char* function, PyObject* filename)
+{
+    /* XXX - see win32_error for comments on 'function' */
+    errno = GetLastError();
+    if (filename)
+        return PyErr_SetExcFromWindowsErrWithFilenameObject(
+                    PyExc_WindowsError,
+                    errno,
+                    filename);
+    else
+        return PyErr_SetFromWindowsErr(errno);
+}
+
 static int
 convert_to_unicode(PyObject **param)
 {
@@ -881,17 +895,21 @@
     char *ansi;
     BOOL result;
 
-    if (!PyArg_ParseTuple(args, wformat, &uni))
-        PyErr_Clear();
-    else {
+    if (PyArg_ParseTuple(args, wformat, &uni))
+    {
+        wchar_t *wstr = PyUnicode_AsUnicode(uni);
+        if (wstr == NULL)
+            return NULL;
         Py_BEGIN_ALLOW_THREADS
-        result = funcW(PyUnicode_AsUnicode(uni));
+        result = funcW(wstr);
         Py_END_ALLOW_THREADS
         if (!result)
-            return win32_error_unicode(func, PyUnicode_AsUnicode(uni));
+            return win32_error_object(func, uni);
         Py_INCREF(Py_None);
         return Py_None;
     }
+    PyErr_Clear();
+
     if (!PyArg_ParseTuple(args, format, &ansi))
         return NULL;
     Py_BEGIN_ALLOW_THREADS
@@ -1801,7 +1819,7 @@
               int (*statfunc)(const char *, STRUCT_STAT *),
 #endif
               char *wformat,
-              int (*wstatfunc)(const Py_UNICODE *, STRUCT_STAT *))
+              int (*wstatfunc)(const wchar_t *, STRUCT_STAT *))
 {
     STRUCT_STAT st;
     PyObject *opath;
@@ -1810,18 +1828,18 @@
     PyObject *result;
 
 #ifdef MS_WINDOWS
-    PyUnicodeObject *po;
+    PyObject *po;
     if (PyArg_ParseTuple(args, wformat, &po)) {
-        Py_UNICODE *wpath = PyUnicode_AS_UNICODE(po);
+        wchar_t *wpath = PyUnicode_AsUnicode(po);
+        if (wpath == NULL)
+            return NULL;
 
         Py_BEGIN_ALLOW_THREADS
-            /* PyUnicode_AS_UNICODE result OK without
-               thread lock as it is a simple dereference. */
         res = wstatfunc(wpath, &st);
         Py_END_ALLOW_THREADS
 
         if (res != 0)
-            return win32_error_unicode("stat", wpath);
+            return win32_error_object("stat", po);
         return _pystat_fromstructstat(&st);
     }
     /* Drop the argument parsing error as narrow strings
@@ -1870,12 +1888,13 @@
 
 #ifdef MS_WINDOWS
     DWORD attr;
-    PyUnicodeObject *po;
+    PyObject *po;
     if (PyArg_ParseTuple(args, "Ui:access", &po, &mode)) {
+        wchar_t* wpath = PyUnicode_AsUnicode(po);
+        if (wpath == NULL)
+            return NULL;
         Py_BEGIN_ALLOW_THREADS
-        /* PyUnicode_AS_UNICODE OK without thread lock as
-           it is a simple dereference. */
-        attr = GetFileAttributesW(PyUnicode_AS_UNICODE(po));
+        attr = GetFileAttributesW(wpath);
         Py_END_ALLOW_THREADS
         goto finish;
     }
@@ -2025,23 +2044,25 @@
     int res;
 #ifdef MS_WINDOWS
     DWORD attr;
-    PyUnicodeObject *po;
+    PyObject *po;
     if (PyArg_ParseTuple(args, "Ui|:chmod", &po, &i)) {
+        wchar_t *wpath = PyUnicode_AsUnicode(po);
+        if (wpath == NULL)
+            return NULL;
         Py_BEGIN_ALLOW_THREADS
-        attr = GetFileAttributesW(PyUnicode_AS_UNICODE(po));
+        attr = GetFileAttributesW(wpath);
         if (attr != 0xFFFFFFFF) {
             if (i & _S_IWRITE)
                 attr &= ~FILE_ATTRIBUTE_READONLY;
             else
                 attr |= FILE_ATTRIBUTE_READONLY;
-            res = SetFileAttributesW(PyUnicode_AS_UNICODE(po), attr);
+            res = SetFileAttributesW(wpath, attr);
         }
         else
             res = 0;
         Py_END_ALLOW_THREADS
         if (!res)
-            return win32_error_unicode("chmod",
-                                       PyUnicode_AS_UNICODE(po));
+            return win32_error_object("chmod", po);
         Py_INCREF(Py_None);
         return Py_None;
     }
@@ -2429,12 +2450,20 @@
     PyObject *osrc, *odst;
     char *src, *dst;
     BOOL rslt;
-
-    PyUnicodeObject *usrc, *udst;
-    if (PyArg_ParseTuple(args, "UU:link", &usrc, &udst)) {
+    PyObject *usrc, *udst;
+
+    if (PyArg_ParseTuple(args, "UU:link", &usrc, &udst))
+    {
+        wchar_t *wsrc, *wdst;
+        wsrc = PyUnicode_AsUnicode(usrc);
+        if (wsrc == NULL)
+            return NULL;
+        wdst = PyUnicode_AsUnicode(udst);
+        if (wdst == NULL)
+            return NULL;
+
         Py_BEGIN_ALLOW_THREADS
-        rslt = CreateHardLinkW(PyUnicode_AS_UNICODE(udst),
-                               PyUnicode_AS_UNICODE(usrc), NULL);
+        rslt = CreateHardLinkW(wdst, wsrc, NULL);
         Py_END_ALLOW_THREADS
 
         if (rslt == 0)
@@ -2495,13 +2524,15 @@
     PyObject *po = NULL;
     if (PyArg_ParseTuple(args, "|U:listdir", &po)) {
         WIN32_FIND_DATAW wFileData;
-        Py_UNICODE *wnamebuf, *po_wchars;
+        wchar_t *wnamebuf, *po_wchars;
 
         if (po == NULL) { /* Default arg: "." */
             po_wchars = L".";
             len = 1;
         } else {
-            po_wchars = PyUnicode_AS_UNICODE(po);
+            po_wchars = PyUnicode_AsUnicode(po);
+            if (po_wchars == NULL)
+                return NULL;
             len = PyUnicode_GET_SIZE(po);
         }
         /* Overallocate for \\*.*\0 */
@@ -2512,7 +2543,7 @@
         }
         wcscpy(wnamebuf, po_wchars);
         if (len > 0) {
-            Py_UNICODE wch = wnamebuf[len-1];
+            wchar_t wch = wnamebuf[len-1];
             if (wch != L'/' && wch != L'\\' && wch != L':')
                 wnamebuf[len++] = L'\\';
             wcscpy(wnamebuf + len, L"*.*");
@@ -2887,18 +2918,24 @@
     char outbuf[MAX_PATH*2];
     char *temp;
 #ifdef MS_WINDOWS
-    PyUnicodeObject *po;
-    if (PyArg_ParseTuple(args, "U|:_getfullpathname", &po)) {
-        Py_UNICODE *wpath = PyUnicode_AS_UNICODE(po);
-        Py_UNICODE woutbuf[MAX_PATH*2], *woutbufp = woutbuf;
-        Py_UNICODE *wtemp;
+    PyObject *po;
+
+    if (PyArg_ParseTuple(args, "U|:_getfullpathname", &po))
+    {
+        wchar_t *wpath;
+        wchar_t woutbuf[MAX_PATH*2], *woutbufp = woutbuf;
+        wchar_t *wtemp;
         DWORD result;
         PyObject *v;
+
+        wpath = PyUnicode_AsUnicode(po);
+        if (wpath == NULL)
+            return NULL;
         result = GetFullPathNameW(wpath,
                                   Py_ARRAY_LENGTH(woutbuf),
                                   woutbuf, &wtemp);
         if (result > Py_ARRAY_LENGTH(woutbuf)) {
-            woutbufp = malloc(result * sizeof(Py_UNICODE));
+            woutbufp = malloc(result * sizeof(wchar_t));
             if (!woutbufp)
                 return PyErr_NoMemory();
             result = GetFullPathNameW(wpath, result, woutbufp, &wtemp);
@@ -2906,7 +2943,7 @@
         if (result)
             v = PyUnicode_FromUnicode(woutbufp, wcslen(woutbufp));
         else
-            v = win32_error_unicode("GetFullPathNameW", wpath);
+            v = win32_error_object("GetFullPathNameW", po);
         if (woutbufp != woutbuf)
             free(woutbufp);
         return v;
@@ -2914,8 +2951,8 @@
     /* Drop the argument parsing error as narrow strings
        are also valid. */
     PyErr_Clear();
-
-#endif
+#endif
+
     if (!PyArg_ParseTuple (args, "O&:_getfullpathname",
                            PyUnicode_FSConverter, &opath))
         return NULL;
@@ -2944,12 +2981,14 @@
     int buf_size;
     wchar_t *target_path;
     int result_length;
-    PyObject *result;
+    PyObject *po, *result;
     wchar_t *path;
 
-    if (!PyArg_ParseTuple(args, "u|:_getfinalpathname", &path)) {
-        return NULL;
-    }
+    if (!PyArg_ParseTuple(args, "U|:_getfinalpathname", &po))
+        return NULL;
+    path = PyUnicode_AsUnicode(po);
+    if (path == NULL)
+        return NULL;
 
     if(!check_GetFinalPathNameByHandle()) {
         /* If the OS doesn't have GetFinalPathNameByHandle, return a
@@ -2968,18 +3007,15 @@
         FILE_FLAG_BACKUP_SEMANTICS,
         NULL);
 
-    if(hFile == INVALID_HANDLE_VALUE) {
-        return win32_error_unicode("GetFinalPathNamyByHandle", path);
-        return PyErr_Format(PyExc_RuntimeError,
-                            "Could not get a handle to file.");
-    }
+    if(hFile == INVALID_HANDLE_VALUE)
+        return win32_error_object("CreateFileW", po);
 
     /* We have a good handle to the target, use it to determine the
        target path name. */
     buf_size = Py_GetFinalPathNameByHandleW(hFile, 0, 0, VOLUME_NAME_NT);
 
     if(!buf_size)
-        return win32_error_unicode("GetFinalPathNameByHandle", path);
+        return win32_error_object("GetFinalPathNameByHandle", po);
 
     target_path = (wchar_t *)malloc((buf_size+1)*sizeof(wchar_t));
     if(!target_path)
@@ -2988,10 +3024,10 @@
     result_length = Py_GetFinalPathNameByHandleW(hFile, target_path,
                                                  buf_size, VOLUME_NAME_DOS);
     if(!result_length)
-        return win32_error_unicode("GetFinalPathNamyByHandle", path);
+        return win32_error_object("GetFinalPathNamyByHandle", po);
 
     if(!CloseHandle(hFile))
-        return win32_error_unicode("GetFinalPathNameByHandle", path);
+        return win32_error_object("CloseHandle", po);
 
     target_path[result_length] = 0;
     result = PyUnicode_FromUnicode(target_path, result_length);
@@ -3033,11 +3069,13 @@
 {
     PyObject *opath;
     char *path;
-    PyUnicodeObject *po;
+    PyObject *po;
     DWORD attributes;
 
     if (PyArg_ParseTuple(args, "U|:_isdir", &po)) {
-        Py_UNICODE *wpath = PyUnicode_AS_UNICODE(po);
+        wchar_t *wpath = PyUnicode_AsUnicode(po);
+        if (wpath == NULL)
+            return NULL;
 
         attributes = GetFileAttributesW(wpath);
         if (attributes == INVALID_FILE_ATTRIBUTES)
@@ -3078,15 +3116,18 @@
     int mode = 0777;
 
 #ifdef MS_WINDOWS
-    PyUnicodeObject *po;
-    if (PyArg_ParseTuple(args, "U|i:mkdir", &po, &mode)) {
+    PyObject *po;
+    if (PyArg_ParseTuple(args, "U|i:mkdir", &po, &mode))
+    {
+        wchar_t *wpath = PyUnicode_AsUnicode(po);
+        if (wpath == NULL)
+            return NULL;
+
         Py_BEGIN_ALLOW_THREADS
-        /* PyUnicode_AS_UNICODE OK without thread lock as
-           it is a simple dereference. */
-        res = CreateDirectoryW(PyUnicode_AS_UNICODE(po), NULL);
+        res = CreateDirectoryW(wpath, NULL);
         Py_END_ALLOW_THREADS
         if (!res)
-            return win32_error_unicode("mkdir", PyUnicode_AS_UNICODE(po));
+            return win32_error_object("mkdir", po);
         Py_INCREF(Py_None);
         return Py_None;
     }
@@ -3098,8 +3139,6 @@
         return NULL;
     path = PyBytes_AsString(opath);
     Py_BEGIN_ALLOW_THREADS
-    /* PyUnicode_AS_UNICODE OK without thread lock as
-       it is a simple dereference. */
     res = CreateDirectoryA(path, NULL);
     Py_END_ALLOW_THREADS
     if (!res) {
@@ -3225,6 +3264,7 @@
 {
 #ifdef MS_WINDOWS
     PyObject *o1, *o2;
+    wchar_t *w1, *w2;
     char *p1, *p2;
     BOOL result;
     if (!PyArg_ParseTuple(args, "OO:rename", &o1, &o2))
@@ -3235,9 +3275,14 @@
         Py_DECREF(o1);
         goto error;
     }
-    Py_BEGIN_ALLOW_THREADS
-    result = MoveFileW(PyUnicode_AsUnicode(o1),
-                       PyUnicode_AsUnicode(o2));
+    w1 = PyUnicode_AsUnicode(o1);
+    if (w1 == NULL)
+        goto error;
+    w2 = PyUnicode_AsUnicode(o2);
+    if (w2 == NULL)
+        goto error;
+    Py_BEGIN_ALLOW_THREADS
+    result = MoveFileW(w1, w2);
     Py_END_ALLOW_THREADS
     Py_DECREF(o1);
     Py_DECREF(o2);
@@ -3497,7 +3542,7 @@
 {
 #ifdef MS_WINDOWS
     PyObject *arg;
-    PyUnicodeObject *obwpath;
+    PyObject *obwpath;
     wchar_t *wpath = NULL;
     PyObject *oapath;
     char *apath;
@@ -3508,23 +3553,26 @@
     PyObject *result = NULL;
 
     if (PyArg_ParseTuple(args, "UO|:utime", &obwpath, &arg)) {
-        wpath = PyUnicode_AS_UNICODE(obwpath);
+        wpath = PyUnicode_AsUnicode(obwpath);
+        if (wpath == NULL)
+            return NULL;
         Py_BEGIN_ALLOW_THREADS
         hFile = CreateFileW(wpath, FILE_WRITE_ATTRIBUTES, 0,
                             NULL, OPEN_EXISTING,
                             FILE_FLAG_BACKUP_SEMANTICS, NULL);
         Py_END_ALLOW_THREADS
         if (hFile == INVALID_HANDLE_VALUE)
-            return win32_error_unicode("utime", wpath);
-    } else
+            return win32_error_object("utime", obwpath);
+    }
+    else {
         /* Drop the argument parsing error as narrow strings
            are also valid. */
         PyErr_Clear();
 
-    if (!wpath) {
         if (!PyArg_ParseTuple(args, "O&O:utime",
                         PyUnicode_FSConverter, &oapath, &arg))
             return NULL;
+
         apath = PyBytes_AsString(oapath);
         Py_BEGIN_ALLOW_THREADS
         hFile = CreateFileA(apath, FILE_WRITE_ATTRIBUTES, 0,
@@ -6372,7 +6420,7 @@
     wchar_t *path;
     DWORD n_bytes_returned;
     DWORD io_result;
-    PyObject *result;
+    PyObject *po, *result;
     HANDLE reparse_point_handle;
 
     char target_buffer[MAXIMUM_REPARSE_DATA_BUFFER_SIZE];
@@ -6380,8 +6428,11 @@
     wchar_t *print_name;
 
     if (!PyArg_ParseTuple(args,
-                  "u:readlink",
-                  &path))
+                  "U:readlink",
+                  &po))
+        return NULL;
+    path = PyUnicode_AsUnicode(po);
+    if (path == NULL)
         return NULL;
 
     /* First get a handle to the reparse point */
@@ -6397,9 +6448,7 @@
     Py_END_ALLOW_THREADS
 
     if (reparse_point_handle==INVALID_HANDLE_VALUE)
-    {
-        return win32_error_unicode("readlink", path);
-    }
+        return win32_error_object("readlink", po);
 
     Py_BEGIN_ALLOW_THREADS
     /* New call DeviceIoControl to read the reparse point */
@@ -6415,9 +6464,7 @@
     Py_END_ALLOW_THREADS
 
     if (io_result==0)
-    {
-        return win32_error_unicode("readlink", path);
-    }
+        return win32_error_object("readlink", po);
 
     if (rdb->ReparseTag != IO_REPARSE_TAG_SYMLINK)
     {
@@ -6468,6 +6515,7 @@
 {
     static char *kwlist[] = {"src", "dest", "target_is_directory", NULL};
     PyObject *src, *dest;
+    wchar_t *wsrc, *wdest;
     int target_is_directory = 0;
     DWORD res;
     WIN32_FILE_ATTRIBUTE_DATA src_info;
@@ -6485,16 +6533,24 @@
     if (win32_can_symlink == 0)
         return PyErr_Format(PyExc_OSError, "symbolic link privilege not held");
 
-    if (!convert_to_unicode(&src)) { return NULL; }
+    if (!convert_to_unicode(&src))
+        return NULL;
     if (!convert_to_unicode(&dest)) {
         Py_DECREF(src);
         return NULL;
     }
 
+    wsrc = PyUnicode_AsUnicode(src);
+    if (wsrc == NULL)
+        goto error;
+    wdest = PyUnicode_AsUnicode(dest);
+    if (wsrc == NULL)
+        goto error;
+
     /* if src is a directory, ensure target_is_directory==1 */
     if(
         GetFileAttributesExW(
-            PyUnicode_AsUnicode(src), GetFileExInfoStandard, &src_info
+            wsrc, GetFileExInfoStandard, &src_info
         ))
     {
         target_is_directory = target_is_directory ||
@@ -6502,20 +6558,21 @@
     }
 
     Py_BEGIN_ALLOW_THREADS
-    res = Py_CreateSymbolicLinkW(
-        PyUnicode_AsUnicode(dest),
-        PyUnicode_AsUnicode(src),
-        target_is_directory);
-    Py_END_ALLOW_THREADS
+    res = Py_CreateSymbolicLinkW(wdest, wsrc, target_is_directory);
+    Py_END_ALLOW_THREADS
+
     Py_DECREF(src);
     Py_DECREF(dest);
     if (!res)
-    {
-        return win32_error_unicode("symlink", PyUnicode_AsUnicode(src));
-    }
+        return win32_error_object("symlink", src);
 
     Py_INCREF(Py_None);
     return Py_None;
+
+error:
+    Py_DECREF(src);
+    Py_DECREF(dest);
+    return NULL;
 }
 #endif /* defined(HAVE_SYMLINK) && defined(MS_WINDOWS) */
 
@@ -6710,12 +6767,14 @@
     int fd;
 
 #ifdef MS_WINDOWS
-    PyUnicodeObject *po;
+    PyObject *po;
     if (PyArg_ParseTuple(args, "Ui|i:open", &po, &flag, &mode)) {
+        wchar_t *wpath = PyUnicode_AsUnicode(po);
+        if (wpath == NULL)
+            return NULL;
+
         Py_BEGIN_ALLOW_THREADS
-        /* PyUnicode_AS_UNICODE OK without thread
-           lock as it is a simple dereference. */
-        fd = _wopen(PyUnicode_AS_UNICODE(po), flag, mode);
+        fd = _wopen(wpath, flag, mode);
         Py_END_ALLOW_THREADS
         if (fd < 0)
             return posix_error();
@@ -7717,6 +7776,8 @@
     }
 #ifdef MS_WINDOWS
     newenv = PyUnicode_AsUnicode(newstr);
+    if (newenv == NULL)
+        goto error;
     _snwprintf(newenv, len, L"%s=%s", s1, s2);
     if (_wputenv(newenv)) {
         posix_error();
@@ -9178,9 +9239,10 @@
     PyObject *ofilepath;
     char *filepath;
     char *operation = NULL;
+    wchar_t *wpath, *woperation;
     HINSTANCE rc;
 
-    PyObject *unipath, *woperation = NULL;
+    PyObject *unipath, *uoperation = NULL;
     if (!PyArg_ParseTuple(args, "U|s:startfile",
                           &unipath, &operation)) {
         PyErr_Clear();
@@ -9188,26 +9250,35 @@
     }
 
     if (operation) {
-        woperation = PyUnicode_DecodeASCII(operation,
+        uoperation = PyUnicode_DecodeASCII(operation,
                                            strlen(operation), NULL);
-        if (!woperation) {
+        if (!uoperation) {
             PyErr_Clear();
             operation = NULL;
             goto normal;
         }
     }
 
-    Py_BEGIN_ALLOW_THREADS
-    rc = ShellExecuteW((HWND)0, woperation ? PyUnicode_AS_UNICODE(woperation) : 0,
-        PyUnicode_AS_UNICODE(unipath),
-        NULL, NULL, SW_SHOWNORMAL);
-    Py_END_ALLOW_THREADS
-
-    Py_XDECREF(woperation);
+    wpath = PyUnicode_AsUnicode(unipath);
+    if (wpath == NULL)
+        goto normal;
+    if (uoperation) {
+        woperation = PyUnicode_AsUnicode(uoperation);
+        if (woperation == NULL)
+            goto normal;
+    }
+    else
+        woperation = NULL;
+
+    Py_BEGIN_ALLOW_THREADS
+    rc = ShellExecuteW((HWND)0, woperation, wpath,
+                       NULL, NULL, SW_SHOWNORMAL);
+    Py_END_ALLOW_THREADS
+
+    Py_XDECREF(uoperation);
     if (rc <= (HINSTANCE)32) {
-        PyObject *errval = win32_error_unicode("startfile",
-                                               PyUnicode_AS_UNICODE(unipath));
-        return errval;
+        win32_error_object("startfile", unipath);
+        return NULL;
     }
     Py_INCREF(Py_None);
     return Py_None;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep 30 01:53:50 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 30 Sep 2011 01:53:50 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_array=2Earray=28=27u=27?=
	=?utf8?q?=29_constructor?=
Message-ID: <E1R9QQI-0000O2-Bd@dinsdale.python.org>

http://hg.python.org/cpython/rev/8b4828bcd67a
changeset:   72543:8b4828bcd67a
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 30 01:54:04 2011 +0200
summary:
  Fix array.array('u') constructor

files:
  Modules/arraymodule.c |  18 ++++++++++++------
  1 files changed, 12 insertions(+), 6 deletions(-)


diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c
--- a/Modules/arraymodule.c
+++ b/Modules/arraymodule.c
@@ -2529,19 +2529,25 @@
                 Py_DECREF(v);
             }
             else if (initial != NULL && PyUnicode_Check(initial))  {
-                Py_ssize_t n = PyUnicode_GET_DATA_SIZE(initial);
+                Py_ssize_t n;
+                if (PyUnicode_READY(initial)) {
+                    Py_DECREF(a);
+                    return NULL;
+                }
+                n = PyUnicode_GET_LENGTH(initial);
                 if (n > 0) {
                     arrayobject *self = (arrayobject *)a;
-                    char *item = self->ob_item;
-                    item = (char *)PyMem_Realloc(item, n);
+                    Py_UCS4 *item = (Py_UCS4 *)self->ob_item;
+                    item = (char *)PyMem_Realloc(item, n * sizeof(Py_UCS4));
                     if (item == NULL) {
                         PyErr_NoMemory();
                         Py_DECREF(a);
                         return NULL;
                     }
-                    self->ob_item = item;
-                    Py_SIZE(self) = n / sizeof(Py_UCS4);
-                    memcpy(item, PyUnicode_AS_DATA(initial), n);
+                    self->ob_item = (char*)item;
+                    Py_SIZE(self) = n;
+                    if (!PyUnicode_AsUCS4(initial, item, n, 0))
+                        return NULL;
                     self->allocated = Py_SIZE(self);
                 }
             }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep 30 01:55:35 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 30 Sep 2011 01:55:35 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_a_compiler_warning?=
Message-ID: <E1R9QRz-0000fX-8M@dinsdale.python.org>

http://hg.python.org/cpython/rev/e88fc377f961
changeset:   72544:e88fc377f961
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 30 01:55:49 2011 +0200
summary:
  Fix a compiler warning

files:
  Modules/arraymodule.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c
--- a/Modules/arraymodule.c
+++ b/Modules/arraymodule.c
@@ -2538,7 +2538,7 @@
                 if (n > 0) {
                     arrayobject *self = (arrayobject *)a;
                     Py_UCS4 *item = (Py_UCS4 *)self->ob_item;
-                    item = (char *)PyMem_Realloc(item, n * sizeof(Py_UCS4));
+                    item = (Py_UCS4 *)PyMem_Realloc(item, n * sizeof(Py_UCS4));
                     if (item == NULL) {
                         PyErr_NoMemory();
                         Py_DECREF(a);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep 30 02:26:40 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 30 Sep 2011 02:26:40 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_PyUnicode=5FCopyCharacters?=
 =?utf8?q?=28=29_uses_exceptions_instead_of_assertions?=
Message-ID: <E1R9Qw4-000359-AT@dinsdale.python.org>

http://hg.python.org/cpython/rev/5d49d11f912d
changeset:   72545:5d49d11f912d
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 30 02:26:10 2011 +0200
summary:
  PyUnicode_CopyCharacters() uses exceptions instead of assertions

Call PyErr_BadInternalCall() if inputs are not unicode strings.

files:
  Objects/unicodeobject.c |  6 ++++--
  1 files changed, 4 insertions(+), 2 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -626,8 +626,10 @@
     unsigned int from_kind, to_kind;
     void *from_data, *to_data;
 
-    assert(PyUnicode_Check(from));
-    assert(PyUnicode_Check(to));
+    if (!PyUnicode_Check(from) || !PyUnicode_Check(to)) {
+        PyErr_BadInternalCall();
+        return -1;
+    }
 
     if (PyUnicode_READY(from))
         return -1;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep 30 02:26:41 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 30 Sep 2011 02:26:41 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_PyUnicode=5FCopy=28=29_?=
 =?utf8?q?function=2C_include_it_to_the_public_API?=
Message-ID: <E1R9Qw5-00035J-6x@dinsdale.python.org>

http://hg.python.org/cpython/rev/c75b04996f54
changeset:   72546:c75b04996f54
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Sep 30 02:26:44 2011 +0200
summary:
  Add PyUnicode_Copy() function, include it to the public API

files:
  Include/unicodeobject.h |   5 +++
  Modules/posixmodule.c   |   3 +-
  Objects/unicodeobject.c |  42 +++++++++++++++-------------
  3 files changed, 28 insertions(+), 22 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -510,6 +510,11 @@
     );
 #endif
 
+/* Get a copy of a Unicode string. */
+PyAPI_FUNC(PyObject*) PyUnicode_Copy(
+    PyObject *unicode
+    );
+
 /* Copy character from one unicode object into another, this function performs
    character conversion when necessary and falls back to memcpy if possible.
 
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -729,8 +729,7 @@
     else if (PyUnicode_Check(*param))
         /* For a Unicode subtype that's not a Unicode object,
            return a true Unicode object with the same data. */
-        *param = PyUnicode_FromUnicode(PyUnicode_AS_UNICODE(*param),
-                                       PyUnicode_GET_SIZE(*param));
+        *param = PyUnicode_Copy(*param);
     else
         *param = PyUnicode_FromEncodedObject(*param,
                                              Py_FileSystemDefaultEncoding,
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -1209,6 +1209,20 @@
     return NULL;
 }
 
+PyObject*
+PyUnicode_Copy(PyObject *unicode)
+{
+    if (!PyUnicode_Check(unicode)) {
+        PyErr_BadInternalCall();
+        return NULL;
+    }
+    if (PyUnicode_READY(unicode))
+        return NULL;
+    return PyUnicode_FromKindAndData(PyUnicode_KIND(unicode),
+                                     PyUnicode_DATA(unicode),
+                                     PyUnicode_GET_LENGTH(unicode));
+}
+
 
 /* Widen Unicode objects to larger buffers.
    Return NULL if the string is too wide already. */
@@ -9061,9 +9075,7 @@
         Py_INCREF(self);
         return (PyObject *) self;
     }
-    return PyUnicode_FromKindAndData(PyUnicode_KIND(self),
-                                     PyUnicode_DATA(self),
-                                     PyUnicode_GET_LENGTH(self));
+    return PyUnicode_Copy(self);
   error:
     if (srelease && sbuf)
         PyMem_FREE(sbuf);
@@ -10477,7 +10489,8 @@
         return NULL;
     kind = PyUnicode_KIND(self);
     data = PyUnicode_1BYTE_DATA(self);
-    return PyUnicode_FromKindAndData(kind, data + PyUnicode_KIND_SIZE(kind, start),
+    return PyUnicode_FromKindAndData(kind,
+                                     data + PyUnicode_KIND_SIZE(kind, start),
                                      end-start);
 }
 
@@ -11267,8 +11280,7 @@
         return self;
     } else
         /* Subtype -- return genuine unicode string with the same value. */
-        return PyUnicode_FromUnicode(PyUnicode_AS_UNICODE(self),
-                                     PyUnicode_GET_SIZE(self));
+        return PyUnicode_Copy(self);
 }
 
 PyDoc_STRVAR(swapcase__doc__,
@@ -11453,10 +11465,7 @@
             return (PyObject*) self;
         }
         else
-            return PyUnicode_FromUnicode(
-                PyUnicode_AS_UNICODE(self),
-                PyUnicode_GET_SIZE(self)
-                );
+            return PyUnicode_Copy(self);
     }
 
     fill = width - _PyUnicode_LENGTH(self);
@@ -11652,16 +11661,9 @@
              "S.__sizeof__() -> size of S in memory, in bytes");
 
 static PyObject *
-unicode_getnewargs(PyUnicodeObject *v)
-{
-    PyObject *copy;
-    unsigned char *data;
-    int kind;
-    if (PyUnicode_READY(v) == -1)
-        return NULL;
-    kind = PyUnicode_KIND(v);
-    data = PyUnicode_1BYTE_DATA(v);
-    copy = PyUnicode_FromKindAndData(kind, data, PyUnicode_GET_LENGTH(v));
+unicode_getnewargs(PyObject *v)
+{
+    PyObject *copy = PyUnicode_Copy(v);
     if (!copy)
         return NULL;
     return Py_BuildValue("(N)", copy);

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Fri Sep 30 05:26:25 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Fri, 30 Sep 2011 05:26:25 +0200
Subject: [Python-checkins] Daily reference leaks (c75b04996f54): sum=0
Message-ID: <E1R9Tk1-0000ac-QS@ap.vmr.nerim.net>

results for c75b04996f54 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogy4gYpQ', '-x']

From python-checkins at python.org  Fri Sep 30 07:46:35 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 30 Sep 2011 07:46:35 +0200
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_ResourceWarnings_in_mak?=
	=?utf8?q?eunicodedata=2Epy=2E?=
Message-ID: <E1R9Vvf-0003T4-SU@dinsdale.python.org>

http://hg.python.org/cpython/rev/b0ef6a5a6dcc
changeset:   72547:b0ef6a5a6dcc
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Fri Sep 30 08:46:25 2011 +0300
summary:
  Fix ResourceWarnings in makeunicodedata.py.

files:
  Tools/unicode/makeunicodedata.py |  167 +++++++++---------
  1 files changed, 87 insertions(+), 80 deletions(-)


diff --git a/Tools/unicode/makeunicodedata.py b/Tools/unicode/makeunicodedata.py
--- a/Tools/unicode/makeunicodedata.py
+++ b/Tools/unicode/makeunicodedata.py
@@ -816,15 +816,15 @@
                  expand=1,
                  cjk_check=True):
         self.changed = []
-        file = open_data(UNICODE_DATA, version)
         table = [None] * 0x110000
-        while 1:
-            s = file.readline()
-            if not s:
-                break
-            s = s.strip().split(";")
-            char = int(s[0], 16)
-            table[char] = s
+        with open_data(UNICODE_DATA, version) as file:
+            while 1:
+                s = file.readline()
+                if not s:
+                    break
+                s = s.strip().split(";")
+                char = int(s[0], 16)
+                table[char] = s
 
         cjk_ranges_found = []
 
@@ -855,32 +855,34 @@
         self.table = table
         self.chars = list(range(0x110000)) # unicode 3.2
 
-        file = open_data(COMPOSITION_EXCLUSIONS, version)
         self.exclusions = {}
-        for s in file:
-            s = s.strip()
-            if not s:
-                continue
-            if s[0] == '#':
-                continue
-            char = int(s.split()[0],16)
-            self.exclusions[char] = 1
+        with open_data(COMPOSITION_EXCLUSIONS, version) as file:
+            for s in file:
+                s = s.strip()
+                if not s:
+                    continue
+                if s[0] == '#':
+                    continue
+                char = int(s.split()[0],16)
+                self.exclusions[char] = 1
 
         widths = [None] * 0x110000
-        for s in open_data(EASTASIAN_WIDTH, version):
-            s = s.strip()
-            if not s:
-                continue
-            if s[0] == '#':
-                continue
-            s = s.split()[0].split(';')
-            if '..' in s[0]:
-                first, last = [int(c, 16) for c in s[0].split('..')]
-                chars = list(range(first, last+1))
-            else:
-                chars = [int(s[0], 16)]
-            for char in chars:
-                widths[char] = s[1]
+        with open_data(EASTASIAN_WIDTH, version) as file:
+            for s in file:
+                s = s.strip()
+                if not s:
+                    continue
+                if s[0] == '#':
+                    continue
+                s = s.split()[0].split(';')
+                if '..' in s[0]:
+                    first, last = [int(c, 16) for c in s[0].split('..')]
+                    chars = list(range(first, last+1))
+                else:
+                    chars = [int(s[0], 16)]
+                for char in chars:
+                    widths[char] = s[1]
+
         for i in range(0, 0x110000):
             if table[i] is not None:
                 table[i].append(widths[i])
@@ -888,36 +890,39 @@
         for i in range(0, 0x110000):
             if table[i] is not None:
                 table[i].append(set())
-        for s in open_data(DERIVED_CORE_PROPERTIES, version):
-            s = s.split('#', 1)[0].strip()
-            if not s:
-                continue
 
-            r, p = s.split(";")
-            r = r.strip()
-            p = p.strip()
-            if ".." in r:
-                first, last = [int(c, 16) for c in r.split('..')]
-                chars = list(range(first, last+1))
-            else:
-                chars = [int(r, 16)]
-            for char in chars:
-                if table[char]:
-                    # Some properties (e.g. Default_Ignorable_Code_Point)
-                    # apply to unassigned code points; ignore them
-                    table[char][-1].add(p)
+        with open_data(DERIVED_CORE_PROPERTIES, version) as file:
+            for s in file:
+                s = s.split('#', 1)[0].strip()
+                if not s:
+                    continue
 
-        for s in open_data(LINE_BREAK, version):
-            s = s.partition('#')[0]
-            s = [i.strip() for i in s.split(';')]
-            if len(s) < 2 or s[1] not in MANDATORY_LINE_BREAKS:
-                continue
-            if '..' not in s[0]:
-                first = last = int(s[0], 16)
-            else:
-                first, last = [int(c, 16) for c in s[0].split('..')]
-            for char in range(first, last+1):
-                table[char][-1].add('Line_Break')
+                r, p = s.split(";")
+                r = r.strip()
+                p = p.strip()
+                if ".." in r:
+                    first, last = [int(c, 16) for c in r.split('..')]
+                    chars = list(range(first, last+1))
+                else:
+                    chars = [int(r, 16)]
+                for char in chars:
+                    if table[char]:
+                        # Some properties (e.g. Default_Ignorable_Code_Point)
+                        # apply to unassigned code points; ignore them
+                        table[char][-1].add(p)
+
+        with open_data(LINE_BREAK, version) as file:
+            for s in file:
+                s = s.partition('#')[0]
+                s = [i.strip() for i in s.split(';')]
+                if len(s) < 2 or s[1] not in MANDATORY_LINE_BREAKS:
+                    continue
+                if '..' not in s[0]:
+                    first = last = int(s[0], 16)
+                else:
+                    first, last = [int(c, 16) for c in s[0].split('..')]
+                for char in range(first, last+1):
+                    table[char][-1].add('Line_Break')
 
         # We only want the quickcheck properties
         # Format: NF?_QC; Y(es)/N(o)/M(aybe)
@@ -928,31 +933,33 @@
         # for older versions, and no delta records will be created.
         quickchecks = [0] * 0x110000
         qc_order = 'NFD_QC NFKD_QC NFC_QC NFKC_QC'.split()
-        for s in open_data(DERIVEDNORMALIZATION_PROPS, version):
-            if '#' in s:
-                s = s[:s.index('#')]
-            s = [i.strip() for i in s.split(';')]
-            if len(s) < 2 or s[1] not in qc_order:
-                continue
-            quickcheck = 'MN'.index(s[2]) + 1 # Maybe or No
-            quickcheck_shift = qc_order.index(s[1])*2
-            quickcheck <<= quickcheck_shift
-            if '..' not in s[0]:
-                first = last = int(s[0], 16)
-            else:
-                first, last = [int(c, 16) for c in s[0].split('..')]
-            for char in range(first, last+1):
-                assert not (quickchecks[char]>>quickcheck_shift)&3
-                quickchecks[char] |= quickcheck
+        with open_data(DERIVEDNORMALIZATION_PROPS, version) as file:
+            for s in file:
+                if '#' in s:
+                    s = s[:s.index('#')]
+                s = [i.strip() for i in s.split(';')]
+                if len(s) < 2 or s[1] not in qc_order:
+                    continue
+                quickcheck = 'MN'.index(s[2]) + 1 # Maybe or No
+                quickcheck_shift = qc_order.index(s[1])*2
+                quickcheck <<= quickcheck_shift
+                if '..' not in s[0]:
+                    first = last = int(s[0], 16)
+                else:
+                    first, last = [int(c, 16) for c in s[0].split('..')]
+                for char in range(first, last+1):
+                    assert not (quickchecks[char]>>quickcheck_shift)&3
+                    quickchecks[char] |= quickcheck
         for i in range(0, 0x110000):
             if table[i] is not None:
                 table[i].append(quickchecks[i])
 
-        zip = zipfile.ZipFile(open_data(UNIHAN, version))
-        if version == '3.2.0':
-            data = zip.open('Unihan-3.2.0.txt').read()
-        else:
-            data = zip.open('Unihan_NumericValues.txt').read()
+        with open_data(UNIHAN, version) as file:
+            zip = zipfile.ZipFile(file)
+            if version == '3.2.0':
+                data = zip.open('Unihan-3.2.0.txt').read()
+            else:
+                data = zip.open('Unihan_NumericValues.txt').read()
         for line in data.decode("utf-8").splitlines():
             if not line.startswith('U+'):
                 continue

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Sep 30 22:35:10 2011
From: python-checkins at python.org (eric.smith)
Date: Fri, 30 Sep 2011 22:35:10 +0200
Subject: [Python-checkins] =?utf8?q?peps=3A_Error_in_PEP_example=2E_Closes?=
	=?utf8?q?_=2313079=2E?=
Message-ID: <E1R9jna-00042a-Hg@dinsdale.python.org>

http://hg.python.org/peps/rev/9a9bd05b9fca
changeset:   3952:9a9bd05b9fca
user:        Eric V. Smith <eric at trueblade.com>
date:        Fri Sep 30 16:35:03 2011 -0400
summary:
  Error in PEP example. Closes #13079.

files:
  pep-3101.txt |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/pep-3101.txt b/pep-3101.txt
--- a/pep-3101.txt
+++ b/pep-3101.txt
@@ -339,7 +339,7 @@
     whose format specifiers might look something like the
     arguments to the strftime() function:
 
-        "Today is: {0:a b d H:M:S Y}".format(datetime.now())
+        "Today is: {0:%a %b %d %H:%M:%S %Y}".format(datetime.now())
 
     For all built-in types, an empty format specification will produce
     the equivalent of str(value).  It is recommended that objects

-- 
Repository URL: http://hg.python.org/peps